hasecuritysolutions/logstash

Contains Logstash related content including tons of Logstash configurations

logstash's Issues

USB Drive Logstash Configuration error

I'm trying to use your USB drive logstash configuration that you have shown in elastic webinar
I'm getting this error, Can you please give me the configuration that you used in the webinar.

PS F:\ELK\logstash-7.1.0> .\bin\logstash -f logstash1.conf --config.reload.automatic
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.runtime.encoding.EncodingService (file:/F:/ELK/logstash-7.1.0/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.Console.cs
WARNING: Please consider reporting this to the maintainers of org.jruby.runtime.encoding.EncodingService
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to F:/ELK/logstash-7.1.0/logs which is now configured via log4j2.properties
[2019-06-03T11:18:11,781][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-06-03T11:18:11,791][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.0"}
[2019-06-03T11:18:14,256][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-06-03T11:18:14,397][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-06-03T11:18:14,438][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-06-03T11:18:14,441][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2019-06-03T11:18:14,459][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-06-03T11:18:14,472][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-06-03T11:18:14,554][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@Version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
warning: thread "[main]-pipeline-manager" terminated with exception (report_on_exception is true):
SyntaxError: (ruby filter code):3: syntax error, unexpected keyword_end

                 eval at org/jruby/RubyKernel.java:1061
             register at F:/ELK/logstash-7.1.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby.rb:59
             register at org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56
     register_plugins at F:/ELK/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:191
                 each at org/jruby/RubyArray.java:1792
     register_plugins at F:/ELK/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:190

maybe_setup_out_plugins at F:/ELK/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:446
start_workers at F:/ELK/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:203
run at F:/ELK/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:145
start at F:/ELK/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:104
[2019-06-03T11:18:14,631][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create

, action_result: false", :backtrace=>nil}
[2019-06-03T11:18:14,791][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SyntaxError) (ruby filter code):3: syntax error, unexpected keyword_end

warning: thread "Api Webserver" terminated with exception (report_on_exception is true):

Changes for Logstash 5.3

Building out a system after taking SEC555 in San Diego last week (excellent class Seth). There are a number of changes to the configurations as part of Logstash 5.0+ (with event['variable'] being deprecated and replaced with event.get('variable') and event.set('variable') ). I can provide the diffs, files or (if you give me access) setup a Logstash 5 fork... let me know how I can help.

Geo IP Database [ERROR 404]

Can you give me alternate links to download Geo IP databases these links are not working
sudo wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
sudo wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
sudo wget http://geolite.maxmind.com/download/geoip/database/GeoIPv6.dat.gz
sudo wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz
sudo wget http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz