So I was working on logstash and didn’t like the huge / worthless messages.
Reference on what patterns already exist:
Grock Patterns Reference
An amazing tool for figuring out your pattern:
http://grokdebug.herokuapp.com/
I modified
root logstash:/etc/logstash/conf.d# vi 10-syslog.conf
to look like
filter { if [type] == "syslog" { if [host] == "10.0.2.3" { grok { remove_tag => "_grokparsefailure" add_tag => "networkadmin" } } else if [host] == "10.0.2.1" { grok { match => { "message" => "%{IPTABLES}"} patterns_dir => ["/var/lib/logstash/etc/grok"] remove_tag => ["_grokparsefailure"] add_tag => ["ddwrt"] } if [src_ip] { geoip { source => "src_ip" target => "geoip" add_field => [ "[geoip][src][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][src][coordinates]", "%{[geoip][latitude]}" ] } mutate { convert => [ "[geoip][coordinates]", "float" ] } } if [dst_ip] { geoip { source => "dst_ip" target => "geoip" add_field => [ "[geoip][dst][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][dst][coordinates]", "%{[geoip][latitude]}" ] } mutate { convert => [ "[geoip][coordinates]", "float" ] } } # http://www.networkassassin.com/elk-for-network-operations/ #Geolocate logs that have SourceAddress and if that SourceAddress is a non-RFC1918 address or APIPA address if [src_ip] and [src_ip] !~ "(^127\.0\.0\.1)|(^10\.)|(^192\.168\.)" { geoip { database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" source => "src_ip" target => "SourceGeo" } #Delete 0,0 in SourceGeo.location if equal to 0,0 if ([SourceGeo.location] and [SourceGeo.location] =~ "0,0") { mutate { replace => [ "SourceGeo.location", "" ] } } } #Geolocate logs that have DestinationAddress and if that DestinationAddress is a non-RFC1918 address or APIPA address if [dst_ip] and [dst_ip] !~ "(^127\.0\.0\.1)|(^10\.)|(^192\.168\.)" { geoip { database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" source => "dst_ip" target => "DestinationGeo" } #Delete 0,0 in DestinationGeo.location if equal to 0,0 if ([DestinationGeo.location] and [DestinationGeo.location] =~ "0,0") { mutate { replace => [ "dst_ip.location", "" ] } } } } else { grok { remove_tag => "_grokparsefailure" add_tag => "syslog from what IP???????" } } } else { grok { match => ["message", "%{GREEDYDATA:syslog_message}"] overwrite => ["message"] add_tag => "not syslog" #add_field => [ "received_at", "%{timestamp}" ] #add_field => [ "received_from", "%{host}" ] } } }
Then bounce the service
root logstash:/etc/logstash/conf.d# service logstash restart; tail -f /var/log/logstash/logstash.log logstash stop/waiting logstash start/running, process 5248 {:timestamp=>"2015-02-17T18:15:17.043000-0800", :message=>"Using milestone 1 input plugin 'lumberjack'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones", :level=>:warn} {:timestamp=>"2015-02-17T18:15:17.174000-0800", :message=>"Using milestone 1 filter plugin 'syslog_pri'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones", :level=>:warn} {:timestamp=>"2015-02-17T18:15:17.973000-0800", :message=>"Using milestone 1 input plugin 'syslog'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones", :level=>:warn} {:timestamp=>"2015-02-17T18:16:03.604000-0800", :message=>"Using milestone 1 input plugin 'lumberjack'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones", :level=>:warn} {:timestamp=>"2015-02-17T18:16:03.732000-0800", :message=>"Using milestone 1 filter plugin 'syslog_pri'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones", :level=>:warn} {:timestamp=>"2015-02-17T18:16:04.527000-0800", :message=>"Using milestone 1 input plugin 'syslog'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.4.2-modified/plugin-milestones", :level=>:warn}
Now I can filter my traffic & map it in Kibana