cn-tu/go-flows

Flow Exporter implementation in go. CN contact: GV

go-flows's Issues

Allow non-existing ports in flowkey

It would be nice to have the source/destination port of a flow with packets that use protocols without port to be set to 0 (or some other specific value), so that they still go into the flow table and are exported.

UpConvertTypes may return invalid values

I'm testing the implementation of the get function, and it doesn't seem to work.
It fails in flows/record.go, because UpConvertTypes returns (what fmt prints as) <bad>.

This happens here:

go-flows/flows/record.go

Lines 290 to 291 in b044915

	ie.Type = UpConvertTypes(init[argumentPos[0]].ie.Type, init[argumentPos[1]].ie.Type)
	ie.Length = ipfix.DefaultSize[ie.Type]

and then there's an index out of range error, since ie.Type isn't a valid index in ipfix.DefaultSize.

Since I don't understand what this if chain is comparing (and I see some exceptions for select/map/apply in other places), it might also be that the get/slice functions require some changes here.

SIGSEGV when exporting each packet as a flow

I want a csv file with one packet per line and thought using go-flows would be faster than tshark.
However, when trying it out on the MAWI data, it crashes after some time.

Here's the command I'm using:
time go-flows offline -active 0 features /features.json export csv /shared/exported.csv input /pcap.pcap
I expected that -active 0 == 1 packet p/ flow, but I'm not completely sure.

The MAWI file I'm using is: http://mawi.wide.ad.jp/mawi/samplepoint-F/2015/201501011400.html.

The generated CSV file has around 13,299,618 lines (different values on different runs), out of 58,066,726 total packets (although there might be buffering in the csv exporter).

Traceback (go-flows compiled from zeromq branch):

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x78e4fe]

goroutine 9 [running]:
github.com/google/gopacket.gopacket.Endpoint.Raw(...)
        /home/dferreira/gocode/src/github.com/CN-TU/go-flows/packet/features.go:49
github.com/CN-TU/go-flows/packet.(*sourceTransportPort).Event(0xc508e39890, 0x89f800, 0xc4360f7080, 0x13b52b13783cd0a0, 0xd51700, 0xc508e36840, 0xc508e34e10, 0x0, 0x0)
        /home/dferreira/gocode/src/github.com/CN-TU/go-flows/packet/features.go:49 +0x9e
github.com/CN-TU/go-flows/flows.(*record).Event(0xc508e34e10, 0x89f800, 0xc4360f7080, 0x13b52b13783cd0a0, 0xd51700, 0xc508e36840, 0xc508e34e10)
        /home/dferreira/gocode/src/github.com/CN-TU/go-flows/flows/record.go:36 +0xb4
github.com/CN-TU/go-flows/flows.(*BaseFlow).Event(0xc508e36840, 0xd49900, 0xc4360f7080, 0x13b52b13783cd0a0)
        /home/dferreira/gocode/src/github.com/CN-TU/go-flows/flows/flow.go:151 +0x99
github.com/CN-TU/go-flows/flows.(*FlowTable).Event(0xc4200992c0, 0xd49900, 0xc4360f7080)
        /home/dferreira/gocode/src/github.com/CN-TU/go-flows/flows/table.go:90 +0x2c9
github.com/CN-TU/go-flows/packet.NewParallelFlowTable.func2(0xc4200104d0, 0xc4200145a0, 0xc4200992c0, 0xc420040d30)
        /home/dferreira/gocode/src/github.com/CN-TU/go-flows/packet/table.go:172 +0x86
created by github.com/CN-TU/go-flows/packet.NewParallelFlowTable
        /home/dferreira/gocode/src/github.com/CN-TU/go-flows/packet/table.go:156 +0x302

real    1m29.242s
user    1m51.264s
sys     0m15.696s

Features file:

{
  "version": "v2",
  "preprocessing": {
    "flows": [
      {
        "features": [
          "flowStartNanoseconds",
          "sourceIPAddress",
          "destinationIPAddress",
          "sourceTransportPort",
          "destinationTransportPort",
          "packetTotalCount",
          "protocolIdentifier"
        ],
        "key_features": [
        ]
      }
    ]
  }
}

The error seems to be consistent across runs, meaning the addresses don't change.
I doubt it's failing hardware again, but am still running memtester for good measure.

UPDATE: memtester 5 times in 9GB (out of 16GB) shows 0 errors.

Any thoughts?

Tests with pcap files

Tests with pcap files with multiple encapsulation layers (Ethernet, SLL) should be written, to ensure critical code doesn't break.

Implementing application layer features

I'm trying to implement some DNS features, and am struggling to understand why this isn't working.

You can see my attempt at a feature which returns the requested domain names:
https://github.com/dcferreira/go-flows/blob/c9c48e24880ac88f40feebfd58e5df720aca70b2/modules/features/custom/application.go#L145-L153
You can use this example to run it.

The result is that layer is always nil, and so nothing is ever output. I'm using a pcap with plenty of DNS requests, and that wireshark recognizes fine.

What am I doing wrong here?

Read Error (and doesn't stop)

Command executed:

time go-flows offline -active 0 features /features.json export csv /shared/exported.csv input /pcap.pcap

pcap.pcap file (contains 3 files, explanation below): merge.tar.gz

Output:

2018/02/01 12:29:36 Error: Read Error
2018/02/01 12:29:36 Error: Read Error

real    0m0.267s
user    0m0.064s
sys     0m0.036s

The provided pcap literally has only 2 packets, and still I see no reason for this error.
I'm pretty sure however that it has something to do with how the file is generated.

I start with 52 pcap files, each of which seems to function correctly by itself (I didn't test all of them, but tested more than half).
I merge these files using mergecap, and obtain these Read Errors.

If I merge only file # 1 and file # 2, the merged file still functions, even if I merge them in reverse order.
If I merge only file # 1 and file # 20, it fails.

For the files attached above, I grabbed one packet from pcap # 1 and one packet from pcap # 20.
The difference I found in the files is that, when it has this error, it has 2 (or more) interfaces (when there's no error it has only 1):

Interface #0 info:
                     Encapsulation = Linux cooked-mode capture (25/113 - linux-sll)
                     Capture length = 65535
                     Time precision = microseconds (6)
                     Time ticks per second = 1000000
                     Number of stat entries = 0
                     Number of packets = 1
Interface #1 info:
                     Name = Unknown/not available in original file format(libpcap)
                     Encapsulation = Linux cooked-mode capture (25/113 - linux-sll)
                     Capture length = 262144
                     Time precision = microseconds (6)
                     Time ticks per second = 1000000
                     Time resolution = 0x06
                     Number of stat entries = 0
                     Number of packets = 1

I can't seem to get rid of this 2nd interface. I tried using the -I flag in mergecap (that's what the filenames reference), but it whichever option I choose, when merging it always gets these 2 interfaces.

So:

• I don't know why/where this error is thrown, or why it doesn't crash the program;
• I don't understand how to fix my pcaps so that they function with go-flows.

-perpacket and flowDirection

I would like to know how the flowDirection feature is extracted if the -perpacket flag is set.

If one extracts single packets (a conversion of a pcap into a csv), I thought that we do not care much about the flow key and hence, there is no way to get the direction, but it seems that go-flows creates flow tables but then writes an entry each time a packet is fed. Is this the case?

SLL support

I have about 100GB of pcaps encapsulated with SLL (https://wiki.wireshark.org/SLL).
You can see an example of such file here: sllpcap.tar.gz

Gopacket supposedly supports this (see google/gopacket#37, and https://github.com/google/gopacket/blob/master/layers/linux_sll.go), but it seems it doesn't work with the flow exporter.
Running (pcap file is the one above)

go-flows offline features examples/complex_v1.json export csv out.csv input 1_first.pcap

returns

2018/01/30 18:35:30 File format not implemented

What am I missing here? Shouldn't go-flows support this?
If not, how can I force into a format that go-flows supports? (I don't need the information on this SLL layer)

Error when creating feature with "and" function

Hi,

I've created a configuration where a feature is designed that uses the "add" function:

{
  "active_timeout": 1800,
  "idle_timeout": 300,
  "bidirectional": true,
  "features": [
    "sourceIPAddress",
    "destinationIPAddress",
    "protocolIdentifier",
    "sourceTransportPort",
    "destinationTransportPort",
    {
      "count": [
        {
          "and": [
            {
              "select": [
                {
                  "greater": [
                    "ipTotalLength",
                    128
                  ]
                }
              ]
            },
            {
              "select": [
                {
                  "leq": [
                    "ipTotalLength",
                    256
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "key_features": [
    "sourceIPv4Address",
    "destinationIPv4Address",
    "sourceTransportPort",
    "destinationTransportPort",
    "protocolIdentifier"
  ]
}

The feature is inspired by an example found here: https://nta-meta-analysis.readthedocs.io/en/latest/features.html#example-features

However, when I run go-flows, I get an error:

./go-flows run features config.json export csv out.csv source libpcap file.pcap

panic: runtime error: index out of range [2] with length 2

goroutine 1 [running]:
github.com/CN-TU/go-flows/flows.(*astCall).build(0xc0001cf900, 0x4)
        /home/rmarkham/Adhoc/go-flows/flows/ast.go:499 +0x770
github.com/CN-TU/go-flows/flows.(*astCall).build(0xc0001cf7c0, 0x5)
        /home/rmarkham/Adhoc/go-flows/flows/ast.go:511 +0x5eb
github.com/CN-TU/go-flows/flows.(*ast).build(0xc0000bf490)
        /home/rmarkham/Adhoc/go-flows/flows/ast.go:731 +0x86
github.com/CN-TU/go-flows/flows.(*ast).compile(0xc0000bf490, 0x0)
        /home/rmarkham/Adhoc/go-flows/flows/ast.go:974 +0x33d
github.com/CN-TU/go-flows/flows.(*RecordListMaker).AppendRecord(0xc00003fac0, {0xc0001c8200?, 0x0?, 0x0?}, {0x0?, 0x3?, 0xc0000240c0?}, {0x0, 0x0, 0x0}, ...)
        /home/rmarkham/Adhoc/go-flows/flows/record.go:300 +0xbe
main.parseArguments({0x93787e, 0x3}, {0xc0000240c0, 0x8, 0x8})
        /home/rmarkham/Adhoc/go-flows/run.go:307 +0x1148
main.main()
        /home/rmarkham/Adhoc/go-flows/go-flows.go:154 +0x610

go-flows was installed with go version go1.22.1 linux/amd64

Improve perpacket export

• Set flow key to none, since it is not needed
• Improve the table work distribution (won't distribute if no flow key...)
• Enable standard gc and test if this improves things

Future work:

• Maybe implement a flow that resets after each packet instead of allocating a new one and then throwing it right away...

How to properly use the CSV input feature

I'm having some problems with labeled data using go-flows.

I have some pcaps, and a CSV file which states the existing attacks in the pcaps, along with their timestamp/IPs/ports/protocol.
I wanted to use the __label feature of go-flows to label the flows in the pcaps.

Here's what I did:

1. Run go-flows just for packets, so that I have a list of packets with enough identifiable features that I can find the correct attack in the CSV file with the attacks.
   To do this, I ran

go-flows offline -stats -perpacket -filter "tcp or udp or icmp" features /features.json export csv /shared/exported.csv input INPUT_PCAPS

which generates an exported.csv file with one packer per line.

2. Run a script which, for each packet in exported.csv, looks up if it is an attack in the original CSV file, and prints either "Normal" or its attack class (depending on whether or not it is found in the original CSV).
   Save this output to a new CSV file out.csv.
   This just has one class per line, plus a line for a header.
   It has exactly as many lines as exported.csv, meaning one line per packet

3. Run go-flows again, with the same pcaps as in the beginning, and out.csv as CSV inputs for the labels (features.json here includes some feature that uses __label):

go-flows offline -stats -filter "tcp or udp or icmp" features /features.json export csv /shared/exported_flows.csv input /out.csv INPUT_PCAPS

I expected that exported_flows.csv had flows with correct labels, but they all seem to be incorrect.

Is this behavior expectable?
Is there some randomness that I'm not accounting for, or did I misunderstand something in how the labels in CSV file work?

Change label function to decoded version

Other transport layers

At the moment, transport layers that aren't TCP/UDP simply aren't exported.

I tried the following changes so that it supports OSPF, but couldn't get it to work:

• Added a layers.OSPFv3 attribute here (chose the v3 because it provides CanDecode() unlike just layers.OSPF --- not sure what's the difference to v2 though):
  

  go-flows/packet/buffer.go

  Lines 233 to 235 in a066f89

  	udp layers.UDP
  	icmpv4 ICMPv4Flow
  	icmpv6 ICMPv6Flow

• Added another case for layers.LayerTypeOSPF here, that is the same as TCP except for the first line (and things referencing TCP changed to OSPF):
  

  go-flows/packet/buffer.go

  Lines 336 to 348 in a066f89

  	case layers.LayerTypeTCP:
  	layer.(*layers.TCP).SetInternalPortsForTesting()
  	if pb.first != layers.LayerTypeEthernet && pb.first != layers.LayerTypeIPv4 && pb.first != layers.LayerTypeIPv6 {
  	pb.first = layers.LayerTypeTCP
  	}
  	if pb.transport != nil {
  	log.Panic("Can only assign one Transport Layer")
  	}
  	pb.transport = layer.(gopacket.TransportLayer)
  	if pb.proto == 0 {
  	pb.proto = uint8(layers.IPProtocolTCP)
  	}
  	case layers.LayerTypeICMPv4:

• Added another case for layers.LayerTypeOSPF here:
  

  go-flows/packet/buffer.go

  Lines 554 to 559 in a066f89

  	case layers.LayerTypeTCP:
  	decoder = &pb.tcp
  	case layers.LayerTypeICMPv4:
  	decoder = &pb.icmpv4
  	case layers.LayerTypeICMPv6:
  	decoder = &pb.icmpv6

• Added another case for layers.LayerTypeOSPF here:
  

  go-flows/packet/key.go

  Lines 67 to 72 in a066f89

  	case layers.LayerTypeUDP:
  	protoB = byte(layers.IPProtocolUDP)
  	case layers.LayerTypeICMPv4:
  	protoB = byte(layers.IPProtocolICMPv4)
  	case layers.LayerTypeICMPv6:
  	protoB = byte(layers.IPProtocolICMPv6)

It seems this is not enough to make it work though (although this was kind of a bruteforce attempt at a fix, since I don't fully understand the codebase).

I am wondering if:

• this is the way other transport layers should be added; there seems to be a lot of boilerplate if that is the case
• it would be possible to simply not decode the layer, when features don't require it. Right now I wanted to extract flows from some pcaps which contain about 130 different protocols; if I need to add each one to the flow exporter I'll simply ignore them, but if I could export them as long as the features don't require it, it would be better :)

the distinct operation does not work as expected

When using this pcap

With the following configuration file:

{
    "features": [
        "packetTotalCount",
        {"distinct":["sourceIPAddress"]},
        "ipTotalLength"
    ],
    "bidirectional": false,
    "active_timeout": 0,
    "idle_timeout": 0,
    "key_features": [
        "__timeWindow1s"
    ]
}

The output returns (line 13 for example) 3 packets but 0 distinct sources and 0 total length. I would guess that the problem is in the distinct function and specifically how it increments the count (it does not take care of all cases?).

panic: runtime error: integer divide by zero - {"divide": ["ipTotalLength", "flowDurationMilliseconds"]}

Hello,

I'm deriving some features, and some of them that need to use the function "divide" (e.g. {"divide": ["ipTotalLength", "flowDurationMilliseconds"]}) are returning a panic. I don't know if it is expected or if it should be treated by the tool.

I got this error running go-flows on Monday CIC-IDS2017 pcap file (https://www.unb.ca/cic/datasets/ids-2017.html)

The error:

go-flows run features features.json export csv Monday-WorkingHours.csv source libpcap ../pcap-ng/Monday-WorkingHours.pcap;
panic: runtime error: integer divide by zero

goroutine 6 [running]:
github.com/CN-TU/go-flows/modules/features/operations.(*divideFlow).Stop(0xc0000dce40, 0xa0?, 0x0?)
/mnt/extractors/go-flows-master/modules/features/operations/math_generated.go:283 +0x2bc
github.com/CN-TU/go-flows/flows.(*record).stop(0xc000324370, 0xc0?, 0xc0000dc7c0, 0x0?)
/mnt/extractors/go-flows-master/flows/record.go:92 +0x130
github.com/CN-TU/go-flows/flows.(*record).Export(0xc000324370, 0x78?, 0xc0000dc7c0, 0x14cdd00b53b0de28, 0xc0000ee2a0, 0x0)
/mnt/extractors/go-flows-master/flows/record.go:169 +0x7d
github.com/CN-TU/go-flows/flows.(*BaseFlow).Export(0xc0000e0ba0, 0xc8?, 0xc0005c3800?, 0xc0000dc7c0?)
/mnt/extractors/go-flows-master/flows/flow.go:147 +0x3b
github.com/CN-TU/go-flows/packet.(*tcpFlow).Event(0xc0000e0ba0, {0x9503c8?, 0xc0005c3800}, 0xc0000dc7c0)
/mnt/extractors/go-flows-master/packet/flow.go:42 +0x126
github.com/CN-TU/go-flows/flows.(*FlowTable).Event(0xc0000ee2a0, {0x9503c8, 0xc0005c3800})
/mnt/extractors/go-flows-master/flows/table.go:120 +0x22a
github.com/CN-TU/go-flows/packet.NewFlowTable.func2()
/mnt/extractors/go-flows-master/packet/table.go:182 +0x150
created by github.com/CN-TU/go-flows/packet.NewFlowTable
/mnt/extractors/go-flows-master/packet/table.go:171 +0x348
panic: runtime error: integer divide by zero

goroutine 4 [running]:
github.com/CN-TU/go-flows/modules/features/operations.(*divideFlow).Stop(0xc00018b040, 0xe0?, 0x0?)
/mnt/extractors/go-flows-master/modules/features/operations/math_generated.go:283 +0x2bc
github.com/CN-TU/go-flows/flows.(*record).stop(0xc0002685f0, 0xc0?, 0xc0000dc4c0, 0x0?)
/mnt/extractors/go-flows-master/flows/record.go:92 +0x130
github.com/CN-TU/go-flows/flows.(*record).Export(0xc0002685f0, 0x78?, 0xc0000dc4c0, 0x14cdd00b5721e908, 0xc0000ee0e0, 0x0)
/mnt/extractors/go-flows-master/flows/record.go:169 +0x7d
github.com/CN-TU/go-flows/flows.(*BaseFlow).Export(0xc000181080, 0xc8?, 0xc0005fc000?, 0xc0000dc4c0?)
/mnt/extractors/go-flows-master/flows/flow.go:147 +0x3b
github.com/CN-TU/go-flows/packet.(*tcpFlow).Event(0xc000181080, {0x9503c8?, 0xc0005fc000}, 0xc0000dc4c0)
/mnt/extractors/go-flows-master/packet/flow.go:42 +0x126
github.com/CN-TU/go-flows/flows.(*FlowTable).Event(0xc0000ee0e0, {0x9503c8, 0xc0005fc000})
/mnt/extractors/go-flows-master/flows/table.go:120 +0x22a
github.com/CN-TU/go-flows/packet.NewFlowTable.func2()
/mnt/extractors/go-flows-master/packet/table.go:182 +0x150
created by github.com/CN-TU/go-flows/packet.NewFlowTable
/mnt/extractors/go-flows-master/packet/table.go:171 +0x348