Kibana dashboards

[stamusnetworks]

Kibana dashboards

[arr0n]

lol, protip remember to docker rmi ams_kibana_1 before scratching your head to why your changes aren't in you test build ;)

[pevma]

Those are Kibana 3 templates - they are not compatible with Kibana 4 - that is part of Amsterdam

[mgaulton]

ahh. Thank you. I missed that detail.

[mgaulton]

Any chance of updated versions? :)

[pevma]

Yes work in progress.
Any help/ideas/contribution is much appreciated and welcome as well.

[mgaulton]

Sadly, i'm very new to ELK and such. But i'll help if i can!

[pevma]

No worries - there are many ways you can help - testing is one of them for example.

[mgaulton]

Ideas i'm toying with
-bro integration
-honeypot integrations
-trace of an event across suricata/bro/honeypot for full picture of the event.

[NoahJaehnert]

+1 - I'm interested in using Kibana4 dashboards as well. Any updates on this?

[pevma]

Hopefully within 2 weeks we will push some new dashboards (cc @regit )

[NoahJaehnert]

Any update on the dashboard/Kibana4 front?

[pevma]

:)
I am working on the dashboards this and next week.

[NoahJaehnert]

Hooray. Thanks for the update. I'm happy to test if you need it.

[pevma]

cool - will let you know.
Thanks!

[Eagleman7]

Any updates on this? I just switched from SELKS to Amsterdam and I was missing the dashboards.

[NoahJaehnert]

+1 - Any updates?

[pevma]

Sorry for the delay.
I am working on putting all 130-140 visualizations into dashboard in the next couple of days and if everything works ok with the tests- will update git by week's end.

[pevma]

I hope it was worthed the wait - you can test those -
https://github.com/StamusNetworks/KTS

[arr0n]

so pevma, from your post it says "Kibana 4.4.x needs to be patched in order to load the templates. If you have installed Kibana in /opt/kibana for example" <-- how does this work in the Amsterdam environment? My guess is that Kibana will need to be "built" with a Dockerfile (and the patches added that way) in the docker-compose.yml file. Does that sound about right?

[pevma]

yep ...sounds right ...i think @regit is looking into it and is going to have some update on that soon.

[arr0n]

okay dokie, if i get a chance tomorrow i'll try and do it and make a pull request

[pevma]

that will be super! Thanks

[arr0n]

So i've put into the kibana dockerfile the patching process, but the load.sh script requires connection to a live elasticsearch. So how i see this working is, patch kibana in the dockerfile then either run the load.sh script on start up or do a 'docker exec -it kibana /bin/bash -c "source /tmp/KTS/load.sh"' once its up and running.

My guess is though, and you could stop me if i'm wrong but the if we run it at start up it will destroy whatever saved searches and dashboards are there, so potentially a user loses what they've got and if we do the docker exec one, it'll need to be done every time the amsterdam is started again

[arr0n]

also, is they're any harm by applying the patches but not running the load.sh to kibana too?

[regit]

There is no harm: the patches have been forgotten by Elastics team and should have been part of kibana.

[arr0n]

so what i've done is rewritten the kibana dockerfile in the src/docker folder. It pulls in git, clones the KTS repo, patches kibana, does some sed to change the load.sh script from 127.0.0.1:9200 to elasticsearch:9200, took the original docker-entrypoint,sh script from kibana, added the load.sh script
to it. This means it should load the visualisations/dashboards each run.

I'll give it a run through and see how it behaves, but with a quick glance does everything discussed above make sense, have i forgotten anything blatant?

[regit]

On load.sh you can just give it elasticsearch:9200 as first argument and it will use that instead of localhost.
The tricky part that may be missing is that you need elasticsearch to be up and really running. I did not check load.sh code but maybe a retry policy would be needed.

[arr0n]

as predicted load.sh tries to connect and fails, i'll put a delay in but i'm going to have to sign off this for today and pick it back up again tomorrow.

[pevma]

@arr0n - you can try something like this:

counter=0
while ! echo exit | nc localhost 9200; do
counter=$((counter + 1))
echo "Port 9200 is NOT listening/open !!";
echo "The script will retry to connect 6 times - once every 10 sec";
echo "and exit afterwords if not successful!"; echo;
if [ "$counter" -lt 6 ]; then
sleep 10
else
echo "Tried to connect 6 times already...exiting!"
exit 1
fi
done
(of course make sure the correct spacing is in place)
After/underneath that code block have the load.sh script execution sequence. (this is how we do it for the current SELKS dashboard loading script to check if ES is listening on 9200)

[arr0n]

hey @pevma cheers, i shall have a look at this later on today. Thanks

A

[lcg79]

Great work guys! 👏👏👏

Looking forward for those dashboards 😛

Thanks!

[regit]

Implemented in git. It will be part of next release.