elasticsearch won’t start and leaves no logs

I was installing elasticsearch via logstash & grafana setup. However, upon going to the url for grafana, I had Upgrade Required Your version of Elasticsearch is too old. Kibana requires Elasticsearch 0.90.9 or above. at the top of the screen.

I goto the box where I’m installing and find the process is not running.

                                                                                                           [ OK ] 
root logstash:~# ps -ef | grep lastic
root      2201  1588  0 18:44 pts/0    00:00:00 grep --color=auto lastic

So I look in init script & find the log dir is /var/log/elasticsearch.

root logstash:~# grep LOG_DIR /etc/init.d/elasticsearch 
LOG_DIR=/var/log/$NAME
...

I look in the log dir and there is nothing.

root logstash:~# ls -latr /var/log/elasticsearch/
total 24
drwxrwxr-x 12 root          syslog         4096 Feb 13 18:09 ..
drwxr-xr-x  2 elasticsearch elasticsearch  4096 Feb 13 18:39 .

WTF!? How do I debug this?!?!

Then I found this.

So I edit my init script to display my startup command.

root logstash:~# vi /etc/init.d/elasticsearch 

I add the log_daemon_msg as below:

# Start Daemon
log_daemon_msg "sudo -u $ES_USER $DAEMON $DAEMON_OPTS"
start-stop-daemon --start -b --user "$ES_USER" -c "$ES_USER" --pidfile "$PID_FILE" --exec $DAEMON -- $DAEMON_OPTS
log_end_msg $?

Now when I start elasticsearch I see the exact command being run to kick it off. I will use this to run elasticsearch EXACTLY as the init script does so I can figure out what is wrong.

root logstash:~# service elasticsearch start
 * Starting Elasticsearch Server                                                                                                                       * sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch                                                              [ OK ] 

OK Great. Now that I have the command, I can run it.

root logstash:~# sudo -u elasticsearch  /usr/bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* org.elasticsearch.bootstrap.Elasticsearch
log4j:WARN No appenders could be found for logger (node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
{1.1.2}: Initialization Failed ...
- ElasticsearchIllegalStateException[Failed to obtain node lock, is the following location writable?: [/usr/share/elasticsearch/data/elasticsearch]]
	IOException[failed to obtain lock on /usr/share/elasticsearch/data/elasticsearch/nodes/49]
		IOException[Cannot create directory: /usr/share/elasticsearch/data/elasticsearch/nodes/49] 

AHHA!!! I can’t create a dir under nodes. The process is running as the elasticsearch user. Who owns the parent dir?

root logstash:~# ls -latr /usr/share/elasticsearch
total 36
-rw-r--r--   1 root root 8093 May 22  2014 README.textile
-rw-r--r--   1 root root  150 May 22  2014 NOTICE.txt
-rw-r--r--   1 root root 2141 May 22  2014 core-signatures.txt
drwxr-xr-x 114 root root 4096 Feb 13 16:50 ..
drwxr-xr-x   3 root root 4096 Feb 13 17:28 data
drwxr-xr-x   3 root root 4096 Feb 13 17:47 lib
drwxr-xr-x   2 root root 4096 Feb 13 17:47 bin
drwxr-xr-x   5 root root 4096 Feb 13 17:47 .

root owns it, but the process is being run as the elasticsearch user. Therefore, let’s chown the dir so elasticsearch user can write to it:

root logstash:~# chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data

And start it again:

root logstash:~# service elasticsearch start
 * Starting Elasticsearch Server                                                                                                                       * sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch                                                              [ OK ] 
root logstash:~# sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch
root logstash:~# {1.1.2}: Setup Failed ...
- SettingsException[Failed to load settings from [file:/etc/elasticsearch/elasticsearch.yml]]
	ScannerException[while scanning a simple key; could not found expected ':';  in 'reader', line 380, column 1:

A simple problem to fix. I’m missing a space between the : and value.

root logstash:~# vi /etc/elasticsearch/elasticsearch.yml 

replace:

script.disable_dynamic:true

with:

script.disable_dynamic: true

And it’s now running!

                                                                                           [ OK ] 
root logstash:~# service elasticsearch start
 * Starting Elasticsearch Server                                                                                                                       * sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch                                                              [ OK ] 
root logstash:~# ps -ef | grep elasticsearch
elastic+  7125     1 96 19:02 ?        00:00:09 /usr/lib/jvm/java-7-oracle/bin/java -Xms2g -Xmx2g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.2.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
root      7141  1446  0 19:02 pts/0    00:00:00 grep --color=auto elasticsearch