Kibana with es5 failed

[immunochomik]

Kibana with es5 failed

[immunochomik]

Hello @MatthD it probably is a connection error,
did you set up the cors correctly ?
have a look at https://gist.github.com/rmoff/379e6ce46eb128110f38
it is about old ES but this thing did not change, well there was one broken release of ES 2.3 or something in witch that did not work at all.

[immunochomik]

@MatthD, basically you have something miss configured and It is hard to say what from your comment,
I have got this docker image that has in it kibana3 and ES 5.3 configured correctly, If you are interested enough you could pull it start it and have a look at the

• apache config for kibana
• kibana config
• ES config

https://hub.docker.com/r/immunoglobul/quick-log/

One other thing to keep in mind is that kibana3 is in browser only app where as kibana4-5 has its server-proxy thing and that is impotent because in case of kibana 3 the browser that is running kibana needs to be able to talk directly to the ES server.

[alexolivan]

Hi people... I'm on the same board.
I found this fork from Kibana3 just... well, no words, amazing if it would work... I don't like kibana 5, is a giant step back in my opinion.

I have a working kibana 3 -> elasticsearch 1.4.5 in production.
Now I'm testing ELK 5.4 .... working with asociated packaged kibana in repository.

I created an VHost with this kibana3 fork and it claims "connection failed".

The problem is that I'm using plain elasticsearch: http://192.168.1.1:9200 at config.js and it does not work, and I can put that simple URL on the browser and get reply from the EL server.
I suspect this may be related to the need of some proxy, since in our current deploy we use an intermediate nginx server... maybe I need some apache proxying.

I'm investigating...

[alexolivan]

It works! ... incredible work!

The problem was... I don't know, but I guess it is related to browser cache.

Thank you for this incredible work!

[MatthD]

I did not get it to work, so I just changed the template of my kibana5 to keep new functionalities & got a better design.

If someone is interested I can explain how I got this look:

[immunochomik]

it looks cool :)

[immunochomik]

@alexolivan thanks :)

[alexolivan]

I have been playing a little bit more with this and I'm finding problems:
I only get data as long as index is set with 'timestamping' -> none, so, when enabling timestamping (and then you can choose indices) I'm unable to select a certain Index.

Also, terms, does not work to me...although I can get histograms and tables to work (so I get data and defined pinned queries) terms do appear blank, no pie, no table, no donut... nothing... just a button 'Make Queries' that seems to do nothing.
Same happens with stats... they do not work, although on the histogram I see my queries pinned being drawed...

...but man... it is sooooooo close... and I'm still happy to see that the problem with kibana 4/5 (crap compared with kibana3) has been identified :-D

Thank you!!!

[immunochomik]

Hi @alexolivan it sounds like index mapping problem, could you post the mapping of the index? and on what fields are you trying to do the terms, they will work only on keyword type.

[alexolivan]

Hey hi again!

Ok... I'm not very confortable when manually querying EL with CURL....
After a little bit googling to be able to understand your question (never heard before about the mapping concept for indices, that's my actual EL knowledge level :-P) I managed to get the following:

By doing CURL for /logstash-apache2-2017.07.18/_mapping

I got the following reply:

{"logstash-apache2-2017.07.18":{"mappings":{"apache2-log":{"_all":{"enabled":true,"norms":false},"dynamic_templates":[{"message_field":{"path_match":"message","match_mapping_type":"string","mapping":{"norms":false,"type":"text"}}},{"string_fields":{"match":"*","match_mapping_type":"string","mapping":{"fields":{"keyword":{"type":"keyword"}},"norms":false,"type":"text"}}}],"properties":{"@timestamp":{"type":"date","include_in_all":false},"@version":{"type":"keyword","include_in_all":false},"agent":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"auth":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"bytes":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"clientip":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"geoip":{"dynamic":"true","properties":{"city_name":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"continent_code":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"coordinates":{"type":"float"},"country_code2":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"country_code3":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"country_name":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"dma_code":{"type":"long"},"ip":{"type":"ip"},"latitude":{"type":"half_float"},"location":{"type":"geo_point"},"longitude":{"type":"half_float"},"postal_code":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"region_code":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"region_name":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"timezone":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}}}},"host":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"httpversion":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"ident":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"path":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"rawrequest":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"referrer":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"request":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"request_domain":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"response":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"server_domain":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"server_name":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"server_service":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"server_type":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"service":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"tags":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"timestamp":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"type":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"verb":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}}}},"_default_":{"_all":{"enabled":true,"norms":false},"dynamic_templates":[{"message_field":{"path_match":"message","match_mapping_type":"string","mapping":{"norms":false,"type":"text"}}},{"string_fields":{"match":"*","match_mapping_type":"string","mapping":{"fields":{"keyword":{"type":"keyword"}},"norms":false,"type":"text"}}}],"properties":{"@timestamp":{"type":"date","include_in_all":false},"@version":{"type":"keyword","include_in_all":false},"geoip":{"dynamic":"true","properties":{"ip":{"type":"ip"},"latitude":{"type":"half_float"},"location":{"type":"geo_point"},"longitude":{"type":"half_float"}}}}}}}}

As you see, it is a plain Apache2 GET event.
I'm using the logstash included, plain grok %{COMBINEDAPACHELOG} toghether with Geo-IP for getting geolocation information in logstash.
Also, in logstash (which is 95% of my actual ELK knowledge), I have an elasticsearch output that puts apache2 tagged events/loglines into logstash-apache2-YYYY.MM.DD indices.

This is exactly what I learned to do and works in Kibana3 and Logstash 1.5. (in fact, this works with the ugly kibana5, or at lest that seems to... I have to admit I don't get kibana5 philosophy, but it seems to work.)

If there is some better CURL URL trick I should query the EL instance that prints better output / more useful let me know...
...So That's all the info I'm able to provide for the moment just for my EL knowledge...

Whit respect on "which fields I'm trying to do the terms" well.... Im trying to do on 'response' field and the "request" field, which I guess they're kind of test/string....

Thank you very much on your interest.

[immunochomik]

Cool @alexolivan, the request and response are of type "text", and term aggregation can be done only on type "keyword" have a look at the EL documentation about it.

'@timestamp' is of type date so it should work in the time histogram.

[alexolivan]

Hi!
Yes you're right somehow... the problem is that I 've tried to summarize two separate problems in one shot, leading to confussion.
With respect to my Apache2 logs, now it is clear why terms do not work... I guess the old included logstash grok at 1.5 made the fields work by default, since I didn't remember tweeking anything... plug and play...

With regards to my Indices/timestamp trouble, the problem is it is not refered to Apache2 logs, but to 100% 'custom' logstash filter, and grok pattern, to parse CISCO IOS ACL log lines...
By repeating the Index-mapping opeation, this is what I get:

{"logstash-apache2-2017.07.18":{"mappings":{"apache2-log":{"_all":{"enabled":true,"norms":false},"dynamic_templates":[{"message_field":{"path_match":"message","match_mapping_type":"string","mapping":{"norms":false,"type":"text"}}},{"string_fields":{"match":"","match_mapping_type":"string","mapping":{"fields":{"keyword":{"type":"keyword"}},"norms":false,"type":"text"}}}],"properties":{"@timestamp":{"type":"date","include_in_all":false},"@Version":{"type":"keyword","include_in_all":false},"agent":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"auth":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"bytes":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"clientip":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"geoip":{"dynamic":"true","properties":{"city_name":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"continent_code":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"coordinates":{"type":"float"},"country_code2":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"country_code3":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"country_name":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"dma_code":{"type":"long"},"ip":{"type":"ip"},"latitude":{"type":"half_float"},"location":{"type":"geo_point"},"longitude":{"type":"half_float"},"postal_code":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"region_code":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"region_name":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"timezone":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}}}},"host":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"httpversion":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"ident":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"path":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"rawrequest":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"referrer":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"request":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"request_domain":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"response":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"server_domain":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"server_name":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"server_service":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"server_type":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"service":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"tags":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"timestamp":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"type":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}},"verb":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword"}}}}},"default":{"_all":{"enabled":true,"norms":false},"dynamic_templates":[{"message_field":{"path_match":"message","match_mapping_type":"string","mapping":{"norms":false,"type":"text"}}},{"string_fields":{"match":"","match_mapping_type":"string","mapping":{"fields":{"keyword":{"type":"keyword"}},"norms":false,"type":"text"}}}],"properties":{"@timestamp":{"type":"date","include_in_all":false},"@Version":{"type":"keyword","include_in_all":false},"geoip":{"dynamic":"true","properties":{"ip":{"type":"ip"},"latitude":{"type":"half_float"},"location":{"type":"geo_point"},"longitude":{"type":"half_float"}}}}}}}}

Being self-made, I guess I did something wrong from the very beginning, although it works for me on Kibana 5.
Interestingly I see the @timestamp field appears (to me) identical as in Apache2, but this one does not work :-P

[immunochomik]

Hi @alexolivan,

1. I will need you to send me one example of a document from Es after that insert, question is, does it have the @timestamp field and what is in it.
2. You will need to try to debug kibana query. (One of the icons on the time the histogram panel will show you what is exact query that kibana is sending to the Es, you will need to send that query manually and show me what is the response, are there any errors?

[alexolivan]

Hi... sure...

First I'm going to paste an example as appears in by the elasticsearch-head application (this is more friendly than ES querying to me :-P)

{ "_index": "logstash-ciscoiosacl-2017.07.10", "_type": "cisco-ios-acl", "_id": "AV0pz6gZFUgG1wnKttGl", "_version": 1, "_score": 1, "_source": { "syslog_server_domain": "example.net", "type": "cisco-ios-acl", "ACL_name": "DSL-INBOUND", "dst_ip": "AAA.BBB.YYY.XXX", "packets": "20", "syslog_server_type": "Debian", "src_ip": "47.91.176.48", "path": "xxxxxxxxxx.log", "log_facility": "SEC", "syslog_timestamp": "2017-07-10T02:04:20.775223+02:00", "@version": "1", "host": "xxxxxxxx", "syslog_server_name": "xxxxxxxx", "geoip": { "timezone": "America/Los_Angeles", "ip": "47.91.176.48", "latitude": 37.526, "coordinates": [ -122.3558 , 37.526 ], "continent_code": "NA", "city_name": "San Mateo", "country_code2": "US", "country_name": "United States", "dma_code": 807, "country_code3": "US", "region_name": "California", "location": [ -122.3558 , 37.526 ], "postal_code": "94402", "longitude": -122.3558, "region_code": "CA" }, "ACL_action": "denied", "tags": [ "cisco-ios-acl" , "_dateparsefailure" , "ciscoiosacl-match" ], "src_port": "60101", "log_severity": "6", "@timestamp": "2017-07-10T00:04:21.136Z", "syslog_hostname": "---------------------", "log_mnemonic": "IPACCESSLOGP", "proto": "udp", "dst_port": "44876", "syslog_sequence_number": "265364", "acl_timestamp": "Jul 10 00:54:05.116" } }

Here is.
I have replaced some strings and IP addresses with foo strings, but 99% of the stuff is copy/pasted.
There is an @timestamp field with a very similar if not equal value than the ones found at apache2 indices...
Now I'll try to figure out how to answer question nº2!!! :-P

Thank you very much for your interest!!!!

[alexolivan]

Now for nº 2... on histogram, vie clicking the i icon (inspect)... throws this

curl -XGET 'http://kibana.example.net:9200//_search?pretty' -d ''

The cool thing is that, trying to compare with Apache2, I have realized that on Apache2 it seems to do not work either, as I had _all on the query.
By Going to dashboard Configure Index GUI settings, then, setting it to Timestamping -> day, Indexpattern -> [logstash-apache2-]YYYY.MM.DD (as I learnt and usually do), then saving, and then refresh the browser, then all data vanishes.
So ALL my Indices have the same problem.

It is like it is not getting the Index name properly from the GUI or something I guess...

[immunochomik]

Hi @alexolivan sorry for the late response, last week was mental.

if the only thing that you get from the inspect is the line that you pasted that means that Kibana is not sending the query at all. you should get something like

curl -XGET 'http://localhost:9500/persons1/_search?pretty' -d '{ "size": 0, "aggs": { "0": { "filter": { "bool": { "must": [ { "match_all": {} }, { "query_string": { "query": "*" } } ] } }, "aggs": { "dh": { "date_histogram": { "field": "time", "interval": "1d" } } } } } }'

[immunochomik]

But it looks like your problem is misconfigured dashboard. So your index is named: 'logstash-ciscoiosacl-2017.07.10' and I imagine that you have one of these per day, and the data updates daily, (I have never used logstash so I do not know),
In the dashboard settings Index pane, I would set the index patter to 'logstash-ciscoiosacl*' and Timestamping to none. I do not know what the Timestamping option suppose to do but it is likely that it does not work at all anymore. The index pattern with a wildcard character will send query to all the indexes starting with 'logstash-ciscoiosacl', and then histogram panel should send correct aggregation query based on it's setting.