I was installing elasticsearch via logstash & grafana setup. However, upon going to the url for grafana, I had Upgrade Required Your version of Elasticsearch is too old. Kibana requires Elasticsearch 0.90.9 or above. at the top of the screen.
I goto the box where I’m installing and find the process is not running.
[ OK ]
root logstash:~# ps -ef | grep lastic
root 2201 1588 0 18:44 pts/0 00:00:00 grep --color=auto lastic
So I look in init script & find the log dir is /var/log/elasticsearch.
root logstash:~# grep LOG_DIR /etc/init.d/elasticsearch
LOG_DIR=/var/log/$NAME
...
I look in the log dir and there is nothing.
root logstash:~# ls -latr /var/log/elasticsearch/
total 24
drwxrwxr-x 12 root syslog 4096 Feb 13 18:09 ..
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Feb 13 18:39 .
WTF!? How do I debug this?!?!
Then I found this.
So I edit my init script to display my startup command.
root logstash:~# vi /etc/init.d/elasticsearch
I add the log_daemon_msg as below:
# Start Daemon
log_daemon_msg "sudo -u $ES_USER $DAEMON $DAEMON_OPTS"
start-stop-daemon --start -b --user "$ES_USER" -c "$ES_USER" --pidfile "$PID_FILE" --exec $DAEMON -- $DAEMON_OPTS
log_end_msg $?
Now when I start elasticsearch I see the exact command being run to kick it off. I will use this to run elasticsearch EXACTLY as the init script does so I can figure out what is wrong.
root logstash:~# service elasticsearch start
* Starting Elasticsearch Server * sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch [ OK ]
OK Great. Now that I have the command, I can run it.
root logstash:~# sudo -u elasticsearch /usr/bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* org.elasticsearch.bootstrap.Elasticsearch
log4j:WARN No appenders could be found for logger (node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
{1.1.2}: Initialization Failed ...
- ElasticsearchIllegalStateException[Failed to obtain node lock, is the following location writable?: [/usr/share/elasticsearch/data/elasticsearch]]
IOException[failed to obtain lock on /usr/share/elasticsearch/data/elasticsearch/nodes/49]
IOException[Cannot create directory: /usr/share/elasticsearch/data/elasticsearch/nodes/49]
AHHA!!! I can’t create a dir under nodes. The process is running as the elasticsearch user. Who owns the parent dir?
root logstash:~# ls -latr /usr/share/elasticsearch
total 36
-rw-r--r-- 1 root root 8093 May 22 2014 README.textile
-rw-r--r-- 1 root root 150 May 22 2014 NOTICE.txt
-rw-r--r-- 1 root root 2141 May 22 2014 core-signatures.txt
drwxr-xr-x 114 root root 4096 Feb 13 16:50 ..
drwxr-xr-x 3 root root 4096 Feb 13 17:28 data
drwxr-xr-x 3 root root 4096 Feb 13 17:47 lib
drwxr-xr-x 2 root root 4096 Feb 13 17:47 bin
drwxr-xr-x 5 root root 4096 Feb 13 17:47 .
root owns it, but the process is being run as the elasticsearch user. Therefore, let’s chown the dir so elasticsearch user can write to it:
root logstash:~# chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data
And start it again:
root logstash:~# service elasticsearch start
* Starting Elasticsearch Server * sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch [ OK ]
root logstash:~# sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch
root logstash:~# {1.1.2}: Setup Failed ...
- SettingsException[Failed to load settings from [file:/etc/elasticsearch/elasticsearch.yml]]
ScannerException[while scanning a simple key; could not found expected ':'; in 'reader', line 380, column 1:
A simple problem to fix. I’m missing a space between the : and value.
root logstash:~# vi /etc/elasticsearch/elasticsearch.yml
replace:
script.disable_dynamic:true
with:
script.disable_dynamic: true
And it’s now running!
[ OK ]
root logstash:~# service elasticsearch start
* Starting Elasticsearch Server * sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch [ OK ]
root logstash:~# ps -ef | grep elasticsearch
elastic+ 7125 1 96 19:02 ? 00:00:09 /usr/lib/jvm/java-7-oracle/bin/java -Xms2g -Xmx2g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.2.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
root 7141 1446 0 19:02 pts/0 00:00:00 grep --color=auto elasticsearch
