Comments (2)
For anyone falling into this, after reading 300 articles about Kafka and Go and taking a peek at the code I realized flows were being "encoded" by something called Protocol Buffers, which is something I didn't know about until yesterday but in simple terms is JSON on steroids, luckily there was a plugin in Logstash for "deserializing" protocol buffers and I tried it but it didn't quite work, Logstash forums say they're a bit outdated on protobuf tech. Logstash console below.
{ "ForwardingStatus" => 64, "TimeReceived" => 1571337358, "IcmpType" => 0, "TCPFlags" => 16, "DstVlan" => 0, "FragmentId" => 0, "FlowDirection" => 0, "SrcVlan" => 0, "SequenceNum" => 82825472, "SrcAS" => 0, "SrcAddr" => "4S#:", "Packets" => 1, "DstIf" => 219, "SrcPort" => 443, "DstAddr" => "-?=+-?=-?=", "IPTTL" => 0, "Type" => "NETFLOW_V9", "NextHopAS" => 0, "SamplerAddress" => "\ne\v-?=", "Etype" => 2048, "DstPort" => 62650, "IngressVrfID" => 1610612739, "EgressVrfID" => 1610612736, "SamplingRate" => 0, "NextHop" => "\nek-?=",
I decided to create my own function in Golang to send directly to Elasticsearch, and it did work, but I'm not sure if it will have an impact in performance given that I know nothing about how Go implements concurrency, I'm hoping to receive around 15k flows/sec and hardware is not an issue so I'll keep improving as I learn Go.
from goflow.
Hi @jcdaniel14,
Yes, at the moment, it's only Protobuf. In the future, other serialization format may be enabled.
There was a derived technique to do this that another user showed me, would be to plug the logging output to logstash: #8
I usually advise to use Kafka as it provides automatic message distribution and I am more confident on the performance. Nonetheless, you can use a different "transport" to format into JSON (which is introduced here and here).
The way of creating your own function in Golang to send to ElasticSearch is one way. Kafka helps managing the flow of incoming messages by partitioning it and provides a buffer in case of delays. At 15kflows/sec you should be fine anyway.
If it's in a custom GoFlow version, the risk is to lose UDP packets due to CPU being busy pushing the current ones to ES. If it's an external program plugged on Kafka, it's fine.
from goflow.
Related Issues (20)
- How to lookup the received netflow packets HOT 2
- Can this project be used as a library? HOT 2
- Renormalise the sampled data HOT 2
- gloflow2 doesn't multiply sampling rate with bytes/packets
- Nat Event On Output(Question) HOT 2
- Protobuf messages are corrupted or don't match the provided schema HOT 4
- Port Mirror Question HOT 2
- vendor-specific field parsing HOT 2
- Support for IPFIX dataLinkFrameSection HOT 1
- SIGSEGV when port is already in use
- GoFlow segfaults with bad input HOT 3
- How can we configure goflow to enable collector for Netflow V9 format?
- Same Time* for sflow even if they are different
- [Kafka SSL] Custom truststore and keystore.
- IPFIX Template decoding does not handle Enterprise-Specific Information Elements
- Compiling FlowMessagePb proto
- Cardinality explosion in flow_traffic metrics HOT 1
- Is this project alive ? HOT 3
- Sampler Address HOT 3
- Openvswitch ovs-vsctl connection HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goflow.