johnomernik/goputnfonkafka

A PoC Attempt at working with Go to take Netflow from a Ubiquiti router, jsonify it, and put it on a kafka topic

goputnfonkafka's Issues

Clean up JSON

Right now, for the proof of concept, I sorta force everything into a string. This is inefficient as data goes, and the netflow spec is well documented, so in GO, how can we get the data into the proper JSON types prior to outputting.

A whole lot of arguments...

Given that this was just proof of concept, there are not many arguments allowed. Instead, we need a bunch of things as it pertains to configurations. Including:

• The ability to use ENV Variables for arguments
• Arguments for
• Output
  • On Screen (STDOUT)
  • File
  • Kafka Configurations
  • Do we want to have different outputs other than JSON?
• Adding a TS field based on the TS the collector gets the packet. I didn't see the TS in the Netflow data, but would love to have this field added here.
• Kafka Configuration items
  • SSL Config. (Some of that is there)
  • Understanding using the Async Producer (What I am using) vs. the Sync Producer
  • Ensuring Partitions are setup and used properly. (I like the "Random Partitioner" option in kafkacat, can we emmulate that?)
  • Compression
• Should we disucss adding fields? Name resolution options, with a cache? Geo IP lookups? etc