netsniff-ng/netsniff-ng

A Swiss army knife for your daily Linux network plumbing.

netsniff-ng's Issues

New web design

I got some help to prepare a new design for the netsniff-ng project website.
The main goal is to improve content readability by having all information in a single screen and to improve code layout for easy maintenance.
Accessibility is also improved for people having disabilities.

A preview can be seen at http://test.netsniff-ng.org.

It does not need PHP and JQuery is used to organise content. Using it also opens, new perspective in terms of flexibility.

I created a new branch starting by eroullit/netsniff-ng@5861a7d03ed9872547d31bd9a01592e87376c192

Some things are still missing for now:

• FAQ needs to be changed to fit new design
• Some formatting is needed for browser with disabled Javascript
• Left panel layout

build_nacl.sh on x86 with errors

The following output can be seen on x86 build:

# ./build_nacl.sh ~/nacl
Building NaCl (this takes a while) ...
readlink: missing operand
Try `readlink --help' for more information.
dirname: missing operand
Try `dirname --help' for more information.
readlink: missing operand
Try `readlink --help' for more information.
dirname: missing operand
Try `dirname --help' for more information.
Path for linking: 
Path for including: 
Please input the path where NaCl is like the following:
././nacl_path.sh.sh <include_path> <lib_path>
Done!

$ uname -a
Linux thinkbox 3.2.11 #1 SMP Fri Mar 16 13:00:39 CET 2012 i686 GNU/Linux

The script seems to work fine on x86_64. Current stable Debian was used.

Implement packet capture for other hardware layers

Check how we could make the rx ring functionality so generic that it could also process packets from other hardware layer such as bluetooth, IRDA, token ring etc.

ubuntu compliling don't work

on ubuntu almost two main issue 👍 Wunsused.... flag is not recognized:
NACL_LIB_DIR/NACL_INC_DIR is undefined, building libnacl with curvetun!
Building netsniff-ng toolkit (0.5.8-rc0) for i686-linux-gnu:
Building netsniff-ng:
-e CC hash.c
cc1: error: unrecognized command line option "-Wunused-but-set-variable"
make: *** [netsniff-ng/hash.o] Errore 1
and:
mac80211.c:20:38: fatal error: libnl3/netlink/genl/genl.h: File o directory non esistente
compilation terminated.
make: *** [netsniff-ng/mac80211.o] Errore 1

need to compile only on redHat or Debian ? i don't think so ... wath wrong in my enviroment ?

Evaluate packet anonymization

See, if it possible to implement an anonymous mode where packets would be anonymized while capturing

ifpps, show the 't' value in ncurses

ifpps, show the 't' value (== interval) in ncurses.

netsniff-ng generating corrupt pcaps when capturing traffic between Amazon EC2 VM's

I think I've hit an issue with netsniff-ng release 0.5.6 generating corrupt pcaps. I have two Amazon EC2 virtual machines running, one with Linux and one with Windows. I was running an HTTP proxy and netsniff-ng on the Linux VM with the following command line:

$ sudo netsniff-ng --in eth0 --out dump.pcap --silent --dump-path . --filter bpf.txt

After browsing for a couple minutes from the Windows VM, I had two 3+ MB pcaps, both of which I copied to dropbox:

http://dl.dropbox.com/u/70279462/dump_00000.pcap
http://dl.dropbox.com/u/70279462/dump_00001.pcap

I initially thought it was an xplico problem since xplico did not recognize most of the sessions within the pcaps. However, when I tried to open with Wireshark, I got an error. The capinfos utility has a clear explanation of the issue:

$ capinfos dump_00000.pcap
capinfos: An error occurred after reading 135 packets from "dump_00000.pcap": File contains a record that's not valid.
(pcap: File has 5242946-byte packet, bigger than maximum of 65535)

$ capinfos dump_00001.pcap
capinfos: An error occurred after reading 48 packets from "dump_00001.pcap": File contains a record that's not valid.
(pcap: File has 5242946-byte packet, bigger than maximum of 65535)

Let me know if you are struggling to repro/test fix and I can give you access to my VM's. Thanks.

Jonathan

Flowtop

Flowtop will be a top-like tool to track netfilter connections and extract statistics out of it.

@borkmann, Could you please evaluate the feature needed to implement flowtop in the toolkit?

Find a way to run our tools without using the root account

Our tools are mostly based on PF_PACKET sockets which need elevated privileges in order to work properly.
As a workaround, you can run the tool using sudo but it gives privileges to the whole process.

We have to find a way to give the process the needed permissions just to open PF_PACKET sockets.

Evaluate libe1000 integration

Evaluate the LibE1000 user-space library to receive and transmit packets from/to Intel E1000 adapters. LibE1000 implements "OS bypass" mechanism, where fast path processing and RX/TX data transfers are done without kernel assistance (i.e. no system calls).

http://sourceforge.net/projects/libe1000/

If it's faster for e1000(e) adapters than RX/TX rings, we should include this into netsniff-ng + trafgen via compile option, so that either the libe1000 or RX/TX ring can be used.

Fix for libcli < 1.9.7 by adding sys/time.h to configure test.

The older versions lack include <sys/time.h> in libcli.h, [1] fixed the configure script to not fail in in the test [2].

[1] http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/netsniff-ng/files/netsniff-ng-0.5.8-libcli-test.patch?revision=1.1&view=markup
[2] http://b-4.xmw.de/var/log/portage/build/net-analyzer/netsniff-ng-0.5.8_rc2:20130805-055918.log

astraceroute: dns tracing, paris traceroute-like backend

We need to extend astraceroute to allow for dns tracerouting (there was a nice SIGCOMM paper in 2012), plus we need to extend the backend to enable paris-traceroute like analysis.

Be more aware of address family and hardware type

The code makes the strong assumption that we are working on Ethernet hardware with IP on it.
This code needs to be more generic at this place and detect which kind of hardware is which interface

trafgen: not sending all the packets specified by num

trafgen -v

trafgen 0.5.8-rc2+ (Ziggomatic), Git id: v0.5.8-rc2-26-gc2887b3

I'm having two separate issues.
I'm having an issue with trafgen where the only way I can get it to generate the exact number of packets specified by --num is if I do this AND by issuing it to the shell in contrast from using the same command multiple times in script:

$ trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 --cpus 1 --num 15000000
--kernel-pull 6 works as well.

I've tried various kernel pulls, CPU, and ring size combinations and found that the
above is the only one where my system will generate all 15000000 packets.

I'm reporting no tx loss on the sending host. I've been watching /proc/net/dev and ethtool and do not see any tx discards or drops.

If I launch trafgen with the same arguments and options from a shell script or atd job
the number of packets actually sent vary greatly. For my tests I'm running trafgen
every 3 minutes. The script is executed via crond on the sending host every hour.

!/bin/bash

sleep 10

for i in {0..29..3}
do

echo "trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 --cpus 1 --num 15000000" | at now +${i}min

done

The receiving and sending machine's /proc/net/dev packet column
show the following values. 1 for each of 10 runs per tool is listed.
I'm not sure why the first trafgen instance for each tool sends the full
15000000 packets and the rest slack off.

root@recv:~/64# grep ^eth2: *.stat | awk '{ print $1, $3 }'
daemonlogger.stat:eth2: 15000000
daemonlogger.stat:eth2: 13691387
daemonlogger.stat:eth2: 13441450
daemonlogger.stat:eth2: 13884259
daemonlogger.stat:eth2: 13757665
daemonlogger.stat:eth2: 13714474
daemonlogger.stat:eth2: 13581056
daemonlogger.stat:eth2: 13530621
daemonlogger.stat:eth2: 13562485
daemonlogger.stat:eth2: 13460224
dumpcap.stat:eth2: 15000000
dumpcap.stat:eth2: 13984965
dumpcap.stat:eth2: 13974693
dumpcap.stat:eth2: 13772195
dumpcap.stat:eth2: 13914422
dumpcap.stat:eth2: 14044146
dumpcap.stat:eth2: 13782769
dumpcap.stat:eth2: 13813973
dumpcap.stat:eth2: 13869773
dumpcap.stat:eth2: 13756627
...

Here's two tests to demonstrate the difference between the # packets sent on my machine and the the trafgen options used. The /proc/net/dev output below is on the sending machine from which trafgen is running.

$ trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 --cpus 1 --num 15000000

eth2: 0 0 0 0 0 0 0 0 1020000000 15000000 0 0 0 0 0 0

$ trafgen --in configs/64.cfg --out eth2 --verbose --num 15000000

eth2: 0 0 0 0 0 0 0 0 1016840720 14953540 0 0 0 0 0 0

Here's a diff between the ethtool output of each:

$ diff good.ethtool.txt bad.ethtool.txt
28c28

< tx_octets: 1020000000

 tx_octets: 1016840720

53c53

< tx_ucast_packets: 15000000

 tx_ucast_packets: 14953540

65,66c65,66
< dma_readq_full: 5006163

< dma_read_prioq_full: 0

 dma_readq_full: 104878
 dma_read_prioq_full: 235

68,72c68,72
< ring_set_send_prod_index: 15000000
< ring_status_update: 1143980
< nic_irqs: 488244
< nic_avoided_irqs: 655736

< nic_tx_threshold_hit: 1118132

 ring_set_send_prod_index: 14952642
 ring_status_update: 2150811
 nic_irqs: 367275
 nic_avoided_irqs: 1783536
 nic_tx_threshold_hit: 2124554

mausezahn, netsniff-ng: new protocol dissectors

SCTP, DCCP at least and improvements for the current ones, i.e. --less output.

Sharness inclusion into netsniff-ng

Put sharness into netsniff-ng/src/t/ with some basic tests.

Make TARGET_LINK_LIBRARIES names in CMake consistent

Some library names have small and captial letters (LibGeoIP_LIBRARY), some only captial letters (NETFILTER_CONNTRACK_LIBRARY) and so on. Please make all of those names consistent in regard of naming.

curvetun: add option to disable syslog logging

Request for an cmdline option to disable all logging for the curvetun VPN, so that it cannot be determined which user signed on on a server. This could improve anonymity.

curvetun: udp|tcp server config parsing

If neither udp or tcp is specified in a server config file, tcp defaults (say, if one were to misspell udp). I ended up digging into the source code before realizing my error.

Could parsing in ct_servmgmt.c throw warnings or errors to syslog? Or is that too polluting? I would have benefitted from such.

Capture 802.1Q headers

Hi,

When I make traffic capture via the ethernet interface, the pcap file is created and packets are received but without 802.1Q header. If I use tshark, I get all headers but netsniff-ng remove 802.1Q headers.

I compiled netsniff-ng 0.5.7 from source.
Is that normal behavior ?

Thanks,
Guillaume

0.5.8-rc2 doesn't compile

If I switch to v0.5.8.-rc1 it compiles no problem. I'm using Debian Jessie x86_64.

Any ideas?

ifpps: stats_diff error

Ifpps throws an error after a few lines when I use a 10 second interval.
ifpps --dev eth2 --csv --loop --interval 10000

ifpps: ifpps.c:539: stats_diff: Assertion `!((new->rx_bytes - old->rx_bytes) > (new->rx_bytes))' failed.

mausezahn integration

Over mid till long term, we need to migrate mausezahn code from staging into the normal tree.

Evaluate Cocinelle

Cocinelle is a tool to perform automatic code review.

It would allow us to check the syntax of newly introduced code and catch bugs early on.
It is already used successfully on the Linux kernel.

More info on http://coccinelle.lip6.fr/

Cleanups / TODO file

Clean up code issue & TODO file things ... here should be discussed what needs to be done regarding cleaning code!

after `make install` I can't find ifpps command

I'm using Ubuntu 11.10. following the instructions in INSTALL:

$ cd netsniff-ng/src/
$ mkdir build
$ cd build
$ cmake ..
$ make

make install

and after that I got ifpps: command not found

Evaluate gcov

See if it is possible to use gcov to spot where the code is most used and where it is almost used none for testing purpose

netsniff-ng: Incorrect packet count

I send the same # of packets each time (15000000) and tcpdump reports the correct amount while netsniff-ng reports an extra 2 or 4 packets.

rx: 972.748 MiB 15000000 pkts 0 drops 0 errors
tx: 0.000 MiB 0 pkts 0 drops 0 errors

$ tcpdump -nni eth2 -w /dev/null
^C15000000 packets captured
15000000 packets received by filter
0 packets dropped by kernel

$ netsniff-ng --in eth2 --ring-size 1GiB --out /dev/null --silent --verbose
RX: 1024.00 MiB, 16384 Frames, each 65536 Byte allocated
Running! Hang up with ^C!

15000002  packets incoming (0 unread on exit)
15000002  packets passed filter
       0  packets failed filter (out of space)
  0.0000% packet droprate
     113  sec, 98840 usec in total

$ netsniff-ng --in eth2 --ring-size 512MiB --out /dev/null --silent --verbose
RX: 512.00 MiB, 8192 Frames, each 65536 Byte allocated
Running! Hang up with ^C!

15000002  packets incoming (2 unread on exit)
15000004  packets passed filter
       0  packets failed filter (out of space)
  0.0000% packet droprate
     131  sec, 615943 usec in total

Integrate cmake with contrib/devel/doc/kernel-doc

Integrate cmake with contrib/devel/doc/kernel-doc in order to generate a source documentation.

Documentation, documentation, documentation

Manpages + Readme's needs to be written!

Allow for compile-time option of TPACKET_V2 capture

I use EL6 for my passive monitoring platform and would very much like to use the latest netsniff-ng features. It seems that '91e3a4c5' might be the last commit that support capture from a v2 interface. I understand the rationale for efficiency purposes, but eliminating TPACKET_V2 prevents anyone on RHEL from using netsniff-ng, at least until maybe RHEL7 comes out (if it ships with kernel > 3.2).

Evaluate test frameworks

Test framework would allow us to quickly write some tests to perform sanity checks.
Such tests would be very useful to spot regressions and could be also used to validate a release.

For now, I see two serious possibilty:

• Ctest (http://www.vtk.org/Wiki/CMake_Testing_With_CTest)
• Shareness (https://github.com/mlafeldt/Sharness)

Ctest integrates nicely with CMake and it is a very powerful tool.
By writing small test programs, specific functions can be isolated and fully tested.

Shareness has simplicity on its side and would be an extra tool to integrate. It has a clear output syntax (TAP) and it very efficient to test actions on larger scale.

Maybe the two can be combined to cover specific testing and more generic testing

CMake build process

The goal of this task is to prepare the project to support a CMake build infrastructure.
It will be very useful to optimize compile times, perform cross-compiling, support a test framework and probe needed dependencies.

Cmake >= 2.6 must be supported for a good compatibility.

ERF packet format support

Evaluate the support of ERF packet format to capture/read/replay it

Ubuntu 10.11: Wrong GeoIP data is referenced

libgeoip is installed in version 1.4.8.

The error message:
root@tinkerbell:~/temp/netsniff-ng/build# flowtop
OUI UDP TCP ETH
Error Opening file /usr/share/GeoIP/GeoIPCity.dat
Cannot open GeoIP database!

The following files are present:
root@tinkerbell:~/temp/netsniff-ng/build# ls -la /usr/share/GeoIP/
total 6520
drwxr-xr-x 2 root root 4096 2011-10-15 12:44 .
drwxr-xr-x 267 root root 12288 2012-02-15 00:14 ..
-rw-r--r-- 1 root root 1585923 2011-08-18 01:43 GeoIP.dat
-rw-r--r-- 1 root root 5068179 2011-08-18 01:43 GeoIPv6.dat

Dump ifpps result over time in a file

It would be very practical to be able to dump ifpps results in file.
This way such data could be imported in a classical spreadsheet software for further data crunching or give it to gnuplot to generate a graph out of it.

To achieve this, data could be dump as a CSV file. Such data can be then easily imported.
A shell script called csv2gnuplot.sh can convert to a gnuplot input file.

make error

Well... I've tried to configure and make it work, but it's not working and the error is as follows:
Building netsniff-ng toolkit (0.5.8-rc1) for x86_64-linux-gnu:
Building netsniff-ng:
-e CC dissector.c
In file included from dissector.h:18:0,
from dissector.c:16:
pcap_io.h: In function ‘pcap_devtype_to_linktype’:
pcap_io.h:210:7: error: ‘ARPHRD_IEEE802154_MONITOR’ undeclared (first use in this function)
pcap_io.h:210:7: note: each undeclared identifier is reported only once for each function it appears in
make: *** [netsniff-ng/dissector.o] Error 1

Thanks for the tool!

Geoip database

libGeoIP must access a file which contains the database of the location of each IP address.
There is a free version which can be downloaded from the project website and it is also present in the package management of the all big Linux distribution.

It is also tricky to come up with a "one-size-fits-all" solution because such databases can be installed anywhere on the target but also different versions are delivered between different distributions.

For instance, Debian seems to deliver a GeoIP city database whereas Ubuntu delivers the country edition.

curvetun: stderr outputs in xio.c, where are they going

In debugging curvetun, I'm kicking off the client and receiving no error status whatsoever... nothing in /var/log/messages... Aside from "curvetun client booting!" But clearly my curvetun client is dying for some reason.

Turns out the panic() calls in xio.c are not getting to me... I'm wondering if stderr has been redirected? Changing the panic() calls to syslog_panic() within xio.c is helping me along - now I see my problem has to do with the ioctl call. Should these calls be implemented with syslog_panic()? I tried flushing the stderr after the panic() call in question, no dice (not getting output to terminal).

Expand CPU binding and locality control

Take a look at the locality concepts presented in hwloc and see if they
can be leveraged to have more control over CPU cores like in
the context of hyper threading.

ifpps: show percentage of theoretical line-rate

Show percentage of line-rate would be a useful feature.

Policy about frames larger than the ring frame size

A new switch has been added with 65d0aa1 to allow the user to capture Super Jumbo Frames properly by making the frame size equal 64kB.

However, when the user does not use this option and capture a packet larger than 2kB, his capture PCAP file will be corrupted.

To avoid this problem, two possibilities:

• Drop frames larger than the capture frame size
• Use recvfrom(2) to get hard-copy properly frames larger then the capture frame size to the userspace

Evaluate multi-threaded RX ring vs current RX ring

Using pthread-based multi-threading might make things easier to spread the load over the CPU thus achieving better performance.
It would also make the first step towards more flexibility such as the possibility to pause/resume capture or replay in real-time.

First of all, a proof-of-concept has to be prepared, tested and benchmarked.

Documentation like 0.5.7?

I don't know what happened to documentation like we had in 0.5.7: netsniff-ng-0.5.7/Documentation ?

Please bring it back or let me know how I can generate it!

Thanks!

CMake doesn't test if liburcu-dev is installed

When compiling without the lib the compiler can't find the include files. Error:

[ 87%] Building C object flowtop/CMakeFiles/flowtop.dir/__/flowtop.c.o
.../temp/netsniff-ng/src/flowtop.c:37:18: fatal error: urcu.h: No such file or directory

Add IPv6 support to ashunt

As more and more device on the network are addressed by IPv6, it would be great if ashunt could report them properly.

_FORTIFY_SOURCE redefined

Add a -U_FORTIFY_SOURCE in HARDENING=1 [1] to prevent [2].

x86_64-pc-linux-gnu-gcc -march=native -O2 -pipe -std=gnu99 -pipe -fPIE -pie -Wl,-z,relro,-z,now -fstack-protector-all -Wstack-protector --param=ssp-buffer-size=4 -ftrapv -D_FORTIFY_SOURCE=2 -fexceptions -fomit-frame-pointer -fno-strict-aliasing -fasynchronous-unwind-tables -fno-delete-null-pointer-checks -D_REENTRANT -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -DVERSION_STRING=""0.5.8-rc2"""" -DVERSION_LONG=""0.5.8-rc2"" (Ziggomatic)"" -DPREFIX_STRING="" -Wall -I. -I/usr/include/libnl3 -I/usr/include/libnl3 -DHAVE_DISSECTOR_PROTOS -DNEED_TCPDUMP_LIKE_FILTER -I/usr/include/libnl3 -I/usr/include/libnl3 -DHAVE_DISSECTOR_PROTOS -DNEED_TCPDUMP_LIKE_FILTER -o netsniff-ng/proto_igmp.o -c proto_igmp.c
:0:0: warning: "_FORTIFY_SOURCE" redefined [enabled by default]
proto_igmp.c:1:0: note: this is the location of the previous definition
/*

[1] http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/netsniff-ng/files/netsniff-ng-0.5.8-fortify_source.patch?revision=1.1&view=markup

[2] http://b-4.xmw.de/var/log/portage/build/net-analyzer/netsniff-ng-0.5.8_rc2:20130805-070734.log

NaN values if ifpps

It looks like ifpps prints some values which can be NaN to the user.
Could you investigate on it? thanks!

flowtop missing

Flowtop does not correctly install from the current source or the ubuntu packages.
I can not find the binary anywhere.

curvetun: Client lock-up in randombytes

Curvetun Server-side appears to be up and running fine... syslog says it's ready for connections. That's on a remote Fedora 17 build.

My local client, however, is causing me trouble (Fedora 18 on a macbook pro).

Turns out the ct_client.c call to curve25519_encode is never returning (line 254 of ct_client.c).

254 clen = curve25519_encode(c, p, (unsigned char ) us, us_len,
255 (unsigned char *) &cbuff);
256 if (unlikely(clen <= 0))
257 syslog_panic("Init encrypt error!\n");

I was able to trace it to a spinning loop in randombytes()

xio.c:
162 static void randombytes(unsigned char *x, unsigned long long xlen)
163 {
164 int ret;
165
166 if (fd_rnd < 0) {
167 for (;;) {
168 fd_rnd = open("/dev/urandom", O_RDONLY);
169 if (fd_rnd < 0)
170 break;
171 sleep(1);
172 }
173 }

I have to say, I'm confused as to how this portion of randombytes works! I added some debugging statements to the loops...
First time through fd_rnd = -1, prior to open().
Open returns file descriptor, fd_rnd = 6

This isn't negative, so we don't break. We sleep one second and repeat! Why?

We re-open /dev/urandom... get a NEW file descriptor, now:
fd_rnd = 7!

Etc... Based upon my log my client continues to open /dev/urandom every second... I'm currently on file id 304!

I believe this code came from nacl. I'm just struggling to see how it works =D