Category Archives: IT Notes

elasticsearch won’t start and leaves no logs

Posted on by

I was installing elasticsearch via logstash & grafana setup. However, upon going to the url for grafana, I had Upgrade Required Your version of Elasticsearch is too old. Kibana requires Elasticsearch 0.90.9 or above. at the top of the screen.

I goto the box where I’m installing and find the process is not running.

                                                                                                           [ OK ] 
root logstash:~# ps -ef | grep lastic
root      2201  1588  0 18:44 pts/0    00:00:00 grep --color=auto lastic

So I look in init script & find the log dir is /var/log/elasticsearch.

root logstash:~# grep LOG_DIR /etc/init.d/elasticsearch 
LOG_DIR=/var/log/$NAME
...

I look in the log dir and there is nothing.

root logstash:~# ls -latr /var/log/elasticsearch/
total 24
drwxrwxr-x 12 root          syslog         4096 Feb 13 18:09 ..
drwxr-xr-x  2 elasticsearch elasticsearch  4096 Feb 13 18:39 .

WTF!? How do I debug this?!?!

Then I found this.

So I edit my init script to display my startup command.

root logstash:~# vi /etc/init.d/elasticsearch

I add the log_daemon_msg as below:

# Start Daemon
log_daemon_msg "sudo -u $ES_USER $DAEMON $DAEMON_OPTS"
start-stop-daemon --start -b --user "$ES_USER" -c "$ES_USER" --pidfile "$PID_FILE" --exec $DAEMON -- $DAEMON_OPTS
log_end_msg $?

Now when I start elasticsearch I see the exact command being run to kick it off. I will use this to run elasticsearch EXACTLY as the init script does so I can figure out what is wrong.

root logstash:~# service elasticsearch start
 * Starting Elasticsearch Server                                                                                                                       * sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch                                                              [ OK ]

OK Great. Now that I have the command, I can run it.

root logstash:~# sudo -u elasticsearch  /usr/bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* org.elasticsearch.bootstrap.Elasticsearch
log4j:WARN No appenders could be found for logger (node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
{1.1.2}: Initialization Failed ...
- ElasticsearchIllegalStateException[Failed to obtain node lock, is the following location writable?: [/usr/share/elasticsearch/data/elasticsearch]]
	IOException[failed to obtain lock on /usr/share/elasticsearch/data/elasticsearch/nodes/49]
		IOException[Cannot create directory: /usr/share/elasticsearch/data/elasticsearch/nodes/49]

AHHA!!! I can’t create a dir under nodes. The process is running as the elasticsearch user. Who owns the parent dir?

root logstash:~# ls -latr /usr/share/elasticsearch
total 36
-rw-r--r--   1 root root 8093 May 22  2014 README.textile
-rw-r--r--   1 root root  150 May 22  2014 NOTICE.txt
-rw-r--r--   1 root root 2141 May 22  2014 core-signatures.txt
drwxr-xr-x 114 root root 4096 Feb 13 16:50 ..
drwxr-xr-x   3 root root 4096 Feb 13 17:28 data
drwxr-xr-x   3 root root 4096 Feb 13 17:47 lib
drwxr-xr-x   2 root root 4096 Feb 13 17:47 bin
drwxr-xr-x   5 root root 4096 Feb 13 17:47 .

root owns it, but the process is being run as the elasticsearch user. Therefore, let’s chown the dir so elasticsearch user can write to it:

root logstash:~# chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data

And start it again:

root logstash:~# service elasticsearch start
 * Starting Elasticsearch Server                                                                                                                       * sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch                                                              [ OK ] 
root logstash:~# sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch
root logstash:~# {1.1.2}: Setup Failed ...
- SettingsException[Failed to load settings from [file:/etc/elasticsearch/elasticsearch.yml]]
	ScannerException[while scanning a simple key; could not found expected ':';  in 'reader', line 380, column 1:

A simple problem to fix. I’m missing a space between the : and value.

root logstash:~# vi /etc/elasticsearch/elasticsearch.yml

replace:

script.disable_dynamic:true

with:

script.disable_dynamic: true

And it’s now running!

                                                                                           [ OK ] 
root logstash:~# service elasticsearch start
 * Starting Elasticsearch Server                                                                                                                       * sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch                                                              [ OK ] 
root logstash:~# ps -ef | grep elasticsearch
elastic+  7125     1 96 19:02 ?        00:00:09 /usr/lib/jvm/java-7-oracle/bin/java -Xms2g -Xmx2g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.2.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
root      7141  1446  0 19:02 pts/0    00:00:00 grep --color=auto elasticsearch

CentOS – Install Puppet Agent

Posted on by
  1. Enable base repo for yum 
    [root xenserver ~]# vi /etc/yum.repos.d/CentOS-Base.repo
...
[base]
...
enabled=1

    [root xenserver ~]# rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
  2. Install Puppet 
    [root xenserver ~]#  yum install -y puppet
  3. Enable Service 
    [root xenserver ~]# chkconfig puppet on
  4. If your master is not called puppet, set it in your conf file 
    [root xenserver ~]# vi /etc/puppet/puppet.conf 
[agent]
server = puppet-master.h8n.lan
...
  5. Start your puppet service. 
    [root xenserver ~]# /etc/init.d/puppet start
Starting puppet agent:                                     [  OK  ]
  6. Now goto your puppet master so you can accept the cert of the new client. 
    root puppet-master:~# puppet cert list
  "xenserver" (SHA256) C6:68:BD:94:6D:1A:19:AB:38:3E:AD:EC:33:3D:B4:E0:5D:02:B6:C9:76:16:BE:C3:81:A3:9F:6D:A0:51:BD:DC

    root puppet-master:~# puppet cert sign xenserver
Notice: Signed certificate request for xenserver
Notice: Removing file Puppet::SSL::CertificateRequest xenserver at '/var/lib/puppet/ssl/ca/requests/xenserver.pem'

Ubuntu – Install Puppet Agent

Posted on by
  1. Download the Puppet Labs package 
    user puppet-agent:# cd ~; wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb
  2. Install the Puppet Labs package 
    user puppet-agent:# sudo dpkg -i puppetlabs-release-trusty.deb
  3. Update apt’s available packages 
    user puppet-agent:# sudo apt-get update
  4. Install the Puppet Agent package 
    user puppet-agent:# sudo apt-get install puppet
  5. Modify puppet default file. 
    user puppet-agent:# sudo vi /etc/default/puppet
    1. Enable the Puppet Agent by changing START from “no” to “yes” 
      START=yes
  6. Modify puppet.conf 
    user puppet-agent:# sudo vi /etc/puppet/puppet.conf
    1. Delete the templatedir line and the [master] section from puppet.conf 
      sudo vi /etc/puppet/puppet.conf
    2. Tell your agent where it’s master is. This step is not required if your master is called puppet, but this will only happen wiht micro networks. 
      [agent]
server = puppet-master.my.lan
  7. Start the puppet agent 
    user puppet-agent:# sudo service puppet start
  8. Now goto your puppet master so you can accept the cert of the new client. 
    root puppet-master:# puppet cert list
"puppet-agent" (SHA256) XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX

    root puppet-master:# puppet cert sign puppet-agent
Notice: Signed certificate request for puppet-agent
Notice: Removing file Puppet::SSL::CertificateRequest puppet-agent at '/var/lib/puppet/ssl/ca/requests/puppet-agent.pem'

Source

Ubuntu Install Zabbix Client

Posted on by
  1. Update your apt-get sources to include recent zabbix 
    sudo vi /etc/apt/sources.list

    # Zabbix Application PPA
deb http://ppa.launchpad.net/tbfr/zabbix/ubuntu precise main
deb-src http://ppa.launchpad.net/tbfr/zabbix/ubuntu precise main
  2. Add the PPA’s key so that apt-get trusts the source: 
    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C407E17D5F76A32B
  3. Update your repo & install the agent. 
    sudo apt-get update
sudo apt-get install zabbix-agent
  4. Configure to point to your Zabbix server & know the hostname of client. 
    vi /etc/zabbix/zabbix_agentd.conf
...
Server=10.0.0.9
...
Hostname=docker.foo.lan
...
  5. Now bounce the agent. 
    Source

named.conf permission denied

Posted on by

I’m trying to bounce named and continue getting the same error.


[user networkadmin ~]# sudo service named restart ;tail -30 /var/log/messages
Stopping named:                                            [  OK  ]
Starting named:                                            [FAILED]
...
Jan 29 14:18:12 networkadmin named[9865]: ----------------------------------------------------
Jan 29 14:18:12 networkadmin named[9865]: adjusted limit on open files from 4096 to 1048576
Jan 29 14:18:12 networkadmin named[9865]: found 1 CPU, using 1 worker thread
Jan 29 14:18:12 networkadmin named[9865]: using up to 4096 sockets
Jan 29 14:18:12 networkadmin named[9865]: loading configuration from '/etc\/x'

Linux Port Querying

Posted on by 
[root ip-10-249-66-147 bin]# lsof -i tcp:8009
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    1368 root   40u  IPv6 185235      0t0  TCP *:8009 (LISTEN)

[root ip-10-249-66-147 bin]# netstat -talnp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1132/sshd           
...
LISTEN      1368/java          
tcp        0      0 :::8009                     :::*                        LISTEN      1368/java           
tcp        0      0 :::8080                     :::*                        LISTEN      1368/java           
tcp        0      0 ::ffff:127.0.0.1:5201       :::*                        LISTEN      25725/java

Start Xenserver VM from cmd line

Posted on by
  1. Use vm-list to determine the name of the vm you want to start 
    [root my-xen-server-box ~]# xe vm-list
uuid ( RO)           : 60c1607b-1e51-bfe7-088e-a4a868b80f7f
     name-label ( RW): docker
    power-state ( RO): running


uuid ( RO)           : 96b5c2e3-5c76-a354-74ef-c324069a7c91
     name-label ( RW): jenkins
    power-state ( RO): running


uuid ( RO)           : e0bac120-5676-50fc-7989-c11f5cf1537d
     name-label ( RW): puppet_host1
    power-state ( RO): running


uuid ( RO)           : cc502056-ee19-1c97-a90c-a6dfe766db67
     name-label ( RW): devops
    power-state ( RO): halted


uuid ( RO)           : c4555386-c498-4434-8926-3a42e607aabc
     name-label ( RW): networkadmin
    power-state ( RO): running


uuid ( RO)           : f312e45d-506e-a626-5b28-ba9a179f5524
     name-label ( RW): webserver
    power-state ( RO): halted


uuid ( RO)           : dc50cbba-ec9f-4fba-8135-d104a4dc0c61
     name-label ( RW): Control domain on host: xenserver
    power-state ( RO): running


uuid ( RO)           : e4177291-c3e8-8f6c-93ad-a127fe4e9b62
     name-label ( RW): puppet_host2
    power-state ( RO): running


uuid ( RO)           : dbe265c4-a19d-8211-eb57-161359509a2c
     name-label ( RW): puppet_master
    power-state ( RO): running
  2. Now use the name-label to start the VM 
    [root xenserver ~]# xe vm-start vm=webserver
[root xenserver ~]#

Xenserver list vms

Posted on by 
[root my-xen-server-box ~]# xe vm-list
uuid ( RO)           : 60c1607b-1e51-bfe7-088e-a4a868b80f7f
     name-label ( RW): docker
    power-state ( RO): running


uuid ( RO)           : 96b5c2e3-5c76-a354-74ef-c324069a7c91
     name-label ( RW): jenkins
    power-state ( RO): running

...