Failed to decode a v9/IPFIX ExportPacket

[h-haghpanah]

Failed to decode a v9/IPFIX ExportPacket

[byt3-m3]

That's actually a good sign! What is missing is an additional buffer for exports with templates, which templates have not yet been received. Have a look at the following snippet from the collector:

python-netflow-v9-softflowd/netflow/collector.py

Lines 131 to 139 in 5cdb514

	except (V9TemplateNotRecognized, IPFIXTemplateNotRecognized):
	# TODO: differentiate between v9 and IPFIX, use separate to_retry lists
	if time.time() - pkt.ts > PACKET_TIMEOUT:
	logger.warning("Dropping an old and undecodable v9/IPFIX ExportPacket")
	else:
	to_retry.append(pkt)
	logger.debug("Failed to decode a v9/IPFIX ExportPacket - will "
	"re-attempt when a new template is discovered")
	continue

Sweet I got it, it's ugly but it's working. I think I will make a high-level wrapper to make programmatic usage a little easier if that's ok.

port = 2055
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    templates = {"netflow": {}, "ipfix": {}}
    sock.bind(("0.0.0.0", port))
    print(f"Server Is Running on port: {port}")
    to_retry = []
    while True:
        time.sleep(.25)
        if len(to_retry) > 0:
            print(len(to_retry))
            pkt = to_retry.pop()
            try:
                pkt = netflow.parse_packet(pkt, templates)
                print(pkt.templates)
            except:
                continue

        payload, client = sock.recvfrom(4096)

        try:
            pkt = netflow.parse_packet(payload, templates)
            print(pkt)
        except:
            to_retry.append(payload)


log_collector_B()

[bitkeks]

@nachofest
because @byt3-m3 added the templates = {"netflow": {}, "ipfix": {}} dict and passed it into the collector.
The templates are therefore saved and used across received packets.

1. Exporter sends exports with unknown (to the collector) templates
2. Exporter sends templates, as regular job
3. Collector saves these templates, re-parses all previously unknown packet templates
4. All new exports from this exporter can now be parsed according to the saved templates

[bitkeks]

Hi @h-haghpanah,
thank you for opening this issue! Sorry for the delay, when I first saw your report I was planning on taking some time to document the CLI and link it here. Have not yet been able to do so, sadly.

In general, Failed to decode a v9/IPFIX ExportPacket - will re-attempt when a new template is discovered does not have to be a consistent error. Normally, one of the next exports delivers the templates and old packets get parsed successfully.

Guess you were using UDP? It might happen that the exporter sends templates at the beginning, while your collector instance is not yet listening. In that case the UDP packets are lost, as are the templates. But as mentioned, they should be re-sent regularly.

Could you tell me which exporter software you use? Or a router?

[byt3-m3]

I think this is the same issue I am experiencing. I have pointed out the spot where the error is occurring for me. It errors with a key error for the templates dict, I am using a Cisco CSR 1000v router and the flow exporter

def parse_packet(data: Union[str, bytes], templates=None):
    if templates is None:  # compatibility for v1 and v5
        templates = {}

    if type(data) == str:
        # hex dump as string
        data = bytes.fromhex(data)
    elif type(data) == bytes:
        # check representation based on utf-8 decoding result
        try:
            # hex dump as bytes, but not hex
            dec = data.decode()
            data = bytes.fromhex(dec)
        except UnicodeDecodeError:
            # use data as given, assuming hex-formatted bytes
            pass

    version = get_export_version(data)
    if version == 1:
        return V1ExportPacket(data)
    elif version == 5:
        return V5ExportPacket(data)
    elif version == 9:
        return V9ExportPacket(data, templates['netflow']) <--------- Problem, I tried to change it to a dict.get() style look and no go,
    elif version == 10:
        return IPFIXExportPacket(data, templates["ipfix"]) 
    raise UnknownExportVersion(data, version)

[bitkeks]

Hello @byt3-m3,
may I ask how you use the code you referenced? Are you using the netflow.collector tool? Then the templates should be created with default (empty) values inside the run() method of the server:

python-netflow-v9-softflowd/netflow/collector.py

Lines 113 to 127 in 5cdb514

	def run(self):
	# Process packets from the queue
	try:
	templates = {"netflow": {}, "ipfix": {}}
	to_retry = []
	while not self._shutdown.is_set():
	try:
	# 0.5s delay to limit CPU usage while waiting for new packets
	pkt = self.input.get(block=True, timeout=0.5) # type: RawPacket
	except queue.Empty:
	continue
	try:
	# templates is passed as reference, updated in V9ExportPacket
	export = parse_packet(pkt.data, templates)

There are other possibilities, depending on your use case.

[byt3-m3]

Sorry this is the implementation that is causing the error to throw at the area I mentioned above.

def log_collector():
    port = 2055
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

    sock.bind(("0.0.0.0", port))
    print(f"Server Is Running on port: {port}")
    while True:
        payload, client = sock.recvfrom(4096)
        p = netflow.parse_packet(payload)
        
log_collector()

[byt3-m3]

Hello @byt3-m3,
may I ask how you use the code you referenced? Are you using the netflow.collector tool? Then the templates should be created with default (empty) values inside the run() method of the server:

python-netflow-v9-softflowd/netflow/collector.py

Lines 113 to 127 in 5cdb514

	def run(self):
	# Process packets from the queue
	try:
	templates = {"netflow": {}, "ipfix": {}}
	to_retry = []
	while not self._shutdown.is_set():
	try:
	# 0.5s delay to limit CPU usage while waiting for new packets
	pkt = self.input.get(block=True, timeout=0.5) # type: RawPacket
	except queue.Empty:
	continue
	try:
	# templates is passed as reference, updated in V9ExportPacket
	export = parse_packet(pkt.data, templates)

There are other possibilities, depending on your use case.

This is the exact error

D:\code\python\netflow_parser\venv\Scripts\python.exe D:/code/python/netflow_parser/netflow_parser/core.py
Server Is Running on port: 2055
Traceback (most recent call last):
  File "D:/code/python/netflow_parser/netflow_parser/core.py", line 39, in <module>
    log_collector_B()
  File "D:/code/python/netflow_parser/netflow_parser/core.py", line 35, in log_collector_B
    p = netflow.parse_packet(payload)
  File "D:\code\python\netflow_parser\venv\lib\site-packages\netflow\utils.py", line 57, in parse_packet
    return V9ExportPacket(data, templates['netflow'])
KeyError: 'netflow'

Process finished with exit code 1

[bitkeks]

Perfect, thanks for the quick follow-up!
Try the following:

    templates = {"netflow": {}, "ipfix": {}}
    while True:
        payload, client = sock.recvfrom(4096)
        p = netflow.parse_packet(payload, templates)

[byt3-m3]

templates = {"netflow": {}, "ipfix": {}}

Stupid me before I looked at your code, I realized I may be missing the templates var smh, I was actually trying to figure out if it need to be generated or if the function will auto set it, but I could not find it for the life if me in the code, I kept seeing the reference but no instantiations smh lol, I will try now. thank you.

[byt3-m3]

still no go

Server Is Running on port: 2055
Traceback (most recent call last):
  File "D:/code/python/netflow_parser/netflow_parser/core.py", line 41, in <module>
    log_collector_B()
  File "D:/code/python/netflow_parser/netflow_parser/core.py", line 37, in log_collector_B
    p = netflow.parse_packet(payload, templates=templates)
  File "D:\code\python\netflow_parser\venv\lib\site-packages\netflow\utils.py", line 57, in parse_packet
    return V9ExportPacket(data, templates['netflow'])
  File "D:\code\python\netflow_parser\venv\lib\site-packages\netflow\v9.py", line 373, in __init__
    raise V9TemplateNotRecognized
netflow.v9.V9TemplateNotRecognized

Process finished with exit code 1

[bitkeks]

That's actually a good sign! What is missing is an additional buffer for exports with templates, which templates have not yet been received. Have a look at the following snippet from the collector:

python-netflow-v9-softflowd/netflow/collector.py

Lines 131 to 139 in 5cdb514

	except (V9TemplateNotRecognized, IPFIXTemplateNotRecognized):
	# TODO: differentiate between v9 and IPFIX, use separate to_retry lists
	if time.time() - pkt.ts > PACKET_TIMEOUT:
	logger.warning("Dropping an old and undecodable v9/IPFIX ExportPacket")
	else:
	to_retry.append(pkt)
	logger.debug("Failed to decode a v9/IPFIX ExportPacket - will "
	"re-attempt when a new template is discovered")
	continue

[bitkeks]

The README example is missing the usage of templates, as I've seen. This should also be extended.

[nachofest]

could someone explain to me why the code above by @byt3-m3 does the job? I also had the same problem but using that code snippet allowed me to read the exported flows.

[bitkeks]

This issue had no activity for a long time, feel free to reopen