jpr5/ngrep

ngrep is like GNU grep applied to the network layer. It's a PCAP-based tool that allows you to specify an extended regular or hexadecimal expression to match against data payloads of packets. It understands many kinds of protocols, including IPv4/6, TCP, UDP, ICMPv4/6, IGMP and Raw, across a wide variety of interface types, and understands BPF filter logic in the same fashion as more common packet sniffing tools, such as tcpdump and snoop.

ngrep's Issues

ngrep -W byline output is garbled

ngrep-1.44 recompiled from source works fine. Output from the ngrep distributed from the EPEL repo is garbled, as is the output of the ngrep recompiled from git master.

Environment: centos 7.2.1511

Command: ngrep -I ngrep_bug.pcap -W byline -t|head -n 4

Expected output (ngrep-1.44):

input: ngrep_bug.pcap
#
U 2016/03/28 15:01:01.852997 10.5.3.33:5062 -> 10.5.3.39:5060
SIP/2.0 200 OK.

Actual output:

input: /home/kenstir/Downloads/ngrep_bug.pcap
filter: (ip)
#
? 2016/03/28 15:01:01.852997 @ч -> 
6760SIPpTag0933165;lr=on>.

ngrep_bug.zip

Add install instructions

Hi.
Thanks for the great tool.
Why doesn't this repo contain installation instructions?

BPF "udp or (vlan and udp)" will fail

If BPF would be like "udp or (vlan and udp)" we'll get tagged frames and not tagged too.
We need to decide to shift or not vlan_offset in process().

If we have tagged frame

1. link_offset + vlan_offset
2. we need to cut variable len - vlan_offset

P.S. I'm using my patch to solve this issue
P.S.S. sorry for my english

Add pcapng support

Building on MacOS with --enable-tcpkill fails

I'm not the best at compiling things so I could missing something obvious. Trying to compile with tcpkill support on MacOS but it errors with tcpkill feature enabled but no libnet found.

I have /usr/local/Cellar/libnet/1.1.6/bin in my PATH and libnet-config is working.

$ libnet-config --defines
-DHAVE_SOCKADDR_SA_LEN -DLIBNET_BSDISH_OS -DLIBNET_BSD_BYTE_SWAP
$ libnet-config --libs
-lnet

$ ./configure --enable-tcpkill

Configuring System ...

checking build system type... x86_64-apple-darwin17.7.0
checking host system type... x86_64-apple-darwin17.7.0
checking target system type... x86_64-apple-darwin17.7.0
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for prefix by checking for tcpdump... /usr/sbin/tcpdump

Configuring GNU Regular Expression library ...

loading cache ./config.cache
checking host system type... x86_64-apple-darwin17.7.0
checking target system type... x86_64-apple-darwin17.7.0
checking build system type... x86_64-apple-darwin17.7.0
checking for gcc... (cached) gcc
checking whether the C compiler (gcc  ) works... yes
checking whether the C compiler (gcc  ) is a cross-compiler... no
checking whether we are using GNU C... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for a BSD compatible install... (cached) /usr/bin/install -c
checking how to run the C preprocessor... (cached) gcc -E
checking for AIX... no
checking for getmntent in -lseq... (cached) no
checking for POSIXized ISC... no
checking for minix/config.h... (cached) no
checking for ANSI C header files... (cached) yes
checking for string.h... (cached) yes
checking for working alloca.h... (cached) yes
checking for alloca... (cached) yes
checking for working const... (cached) yes
checking for prefix by checking for gcc... (cached) /usr/bin/gcc
creating ./config.status
creating Makefile
creating doc/Makefile
creating test/Makefile
checking for libnet_init_packet in -lnet... no
!!! error: tcpkill feature enabled but no libnet found

Filtering for DNS queries does not yield an answer?

I've tried filtering for DNS queries and can't see an IP in the response, what am I doing wrong?

sudo ngrep -W single -l -q -d any -i "" udp and port 53
interface: any
filter: ( udp and port 53 ) and (ip || ip6)

U 10.0.0.2:53278 -> 10.0.0.1:53 "k...........duckduckgo.com.....

U 10.0.0.2:53278 -> 10.0.0.1:53 .............duckduckgo.com.....

U 10.0.0.1:53 -> 10.0.0.2:53278 .............duckduckgo.com..............<.B.ns-175.awsdns-21...awsdns-hostmaster.amazon......... ......u...Q.                                                                                  

U 10.0.0.1:53 -> 10.0.0.2:53278 "k...........duckduckgo.com..............<..2..j.........<..6...

ngrep on windows

i'm tryin to compile the solution with visual studio and getting multiple error messages:

windows versions: 8.1 and server 2016
winpcap installed and also the dev pack.
trying to lock for same errors and cannot find anything.
thanks

ngrep fails when using libpcap v1.8.1 and newer kernel version with warning kernel filter failed

Using ngrep with more than 6 filters on a newer kernel version such as 4.18, 5.0, 5.3 or 5.4 will result in a complete fail when libpcap v1.8.1 is used (e.g. ubuntu 18.04). ngrep with more than 6 filters with libpcap 1.9.1 will run but still returns a warning about kernel filter failed: cannot allocate memory.

The complete fail with libpcap v1.8.1 appears to have been fixed around 1.9.0. As for the warnings "Warning: Kernel filter failed: Cannot allocate memory", which persists even with libpcap 1.9.1, I suspect this is due to recent bug fixes and improvements to the bpf verifier as well as ngrep attempting to apply filters in a non-optimal way. I do not experience any of these same failures or warnings when attempting to use tcpdump with more than 6 filters.

• https://www.tcpdump.org/libpcap-changes.txt
• 1.9.0-rc1 - the-tcpdump-group/libpcap@995849a
  Fix some Linux memory-mapped capture buffer size issues
  Don't fail if kernel filter can't be set on Linux (GitHub issue #549)
• the-tcpdump-group/libpcap#549

Some bug fixes to bpf verifier:

• https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=979d63d50c0c0f7bc537bf821e056cc9fe5abd38
• https://nvd.nist.gov/vuln/detail/CVE-2019-7308
• https://blog.cloudflare.com/ebpf-cant-count/

libpcap 1.8.1 + kernel 4.18, 5.0, 5.3, 5.4 Output:

# uname -sr
Linux 5.4.0-42-generic

# /usr/bin/ngrep -V
ngrep: V1.47.1-git, libpcap version 1.8.1

# ngrep -q -t -d any -W byline '' port 123 or 6001 or 6010 or 6002 or 6003 or 6004 or 6005
interface: any
Warning: Kernel filter failed: Cannot allocate memory
Warning: Kernel filter failed: Cannot allocate memory
pcap: can't remove kernel filter: No such file or directory

libpcap 1.9.1 + kernel 4.18, 5.0, 5.3, 5.4 Output:

# uname -sr
Linux 4.18.0-1020-aws

# LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ./ngrep -V
ngrep: V1.47.1-git, libpcap version 1.9.1 (with TPACKET_V3)

# LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH /usr/local/src/ngrep*/ngrep -q -t -d any -W byline '' \(port 123 or 5999 or 6000 or 6001 or 6002 or 6003 or 6004\)
interface: any
Warning: Kernel filter failed: Cannot allocate memory
filter: ( (port 123 or 5999 or 6000 or 6001 or 6002 or 6003 or 6004) ) and (ip)
U 2020/09/01 13:46:22.778659 10.10.11.123:42044 -> 91.189.89.199:123 #1
#............................................i..

U 2020/09/01 13:46:22.852287 91.189.89.199:123 -> 10.10.11.123:42044 #2
$......U......"{...a.W.X.....i........a8......y.

libpcap 1.8.1 + kernel 4.15

# uname -sr
Linux 4.15.0-1080-aws

# ngrep -q -t -d any -W byline '' \(port 123 or 5999 or 6000 or 6001 or 6002 or 6003 or 6004\)
interface: any
filter: ( (port 123 or 5999 or 6000 or 6001 or 6002 or 6003 or 6004) ) and (ip)

U 2020/09/01 13:57:07.137353 10.10.11.123:54195 -> 91.189.94.4:123 #1
#............................................/..

U 2020/09/01 13:57:07.212640 91.189.94.4:123 -> 10.10.11.123:54195 #2
$...........^..
.....n......./......-..)....-...

Reverse resolution on FreeBSD

Hello,

I am using ngrep on FreeBSD and I cannot see how to enable reverse resolution of the IPs in the first line of the output ...

All of my output is IP only with no DNS names ... I only see a hostname if I dig deeper into the packet and look at HTTP requests (or something like that).

How can I enable reverse lookups of the IP traffic ?

Thank you.

xregex.texi issues on openBSD

Hi,

I can't seem to get ngrep 1.47 to compile on openBSD -current:

make   -C regex-0.12 
gcc -g  -DSTDC_HEADERS=1 -DHAVE_STRING_H=1 -DHAVE_ALLOCA=1  -I. -I. -c regex.c
for d in doc test; do (cd $d; make    CPPFLAGS='' CFLAGS='-g' CC='gcc'  DEFS='-DSTDC_HEADERS=1 -DHAVE_STRING_H=1 -DHAVE_ALLOCA=1 ' LDFLAGS='' LOADLIBES='' default); done
make: don't know how to make xregex.texi (prerequisite of: regex.texi)
Stop in regex-0.12/doc
*** Error 2 in regex-0.12 (Makefile:67 'default')
*** Error 1 in /home/jungle/bin/ngrep-1_47 (Makefile:62 'regex-0.12/regex.o')

Any suggestions?

I have pcre-8.41 installed:

$ ls -l /usr/local/include/pcre*
-rw-r--r--  1 root  bin  31706 Mar 12 13:47 /usr/local/include/pcre.h
-rw-r--r--  1 root  bin  44180 Apr 28 08:42 /usr/local/include/pcre2.h
-rw-r--r--  1 root  bin   5804 Apr 28 08:42 /usr/local/include/pcre2posix.h
-rw-r--r--  1 root  bin   6600 Mar 12 13:47 /usr/local/include/pcre_scanner.h
-rw-r--r--  1 root  bin   6312 Mar 12 13:47 /usr/local/include/pcre_stringpiece.h
-rw-r--r--  1 root  bin  26529 Mar 12 13:47 /usr/local/include/pcrecpp.h
-rw-r--r--  1 root  bin   6783 Mar 12 13:47 /usr/local/include/pcrecpparg.h
-rw-r--r--  1 root  bin   5452 Mar 12 13:47 /usr/local/include/pcreposix.h

DROPPRIVS_USER in manpage is undefined

In the ngrep manpage, in reference to the -R command line option, it reads

Do not try to drop privileges to the DROPPRIVS_USER

However, there is no other use of DROPPRIVS_USER in the man page and it is left undefined.

Written as is, due to conventions commonly used in some manpages and command line help text, a user may mistakenly assume that DROPPRIVS_USER can be specified at runtime such as via an argument to the option, environmental variable, or key in a configuration file despite it being statically set at compile time. Considering the potential unforeseen security implications of a user acting on that assumption (or time spent investigating), it may be prudent to update the manpage to remove the opportunity for confusion.

Potential options could be to remove the reference to DROPPRIVS_USER all together or instead elaborate that DROPPRIVS_USER is a compile time option and optionally mention the default value or insert the value used into the manpage at build time.

Add support for FIX protocol

ngrep currently doesn't search FIX msgs one by one but TCP packet by packet (one TCP packet can contain multiple FIX msgs). The resulting pcap file thus contains not only the filtered FIX msg, but also all the msgs that were in that TCP packet.

Example (filtering by FIX MsgType (e.g. we want only ticks msges (35=W)):

$ ngrep -I network_traffic_from_multiple_FIX_connections_dump.pcap -o only_ticks.pcap -q "35=W" > /dev/null

only_ticks.pcap will contain also quotes messages (35=X). Note that in FIX protocol the MsgType field (in the protocol serialized as "35=[char]") is unique per FIX msg i.e. there is only one instance per FIX msg. So expected result of the ngrep command above would be a pcap file containing only FIX messages with "35=X".

$ ngrep -V
ngrep: V1.47.1-git, libpcap version 1.8.1

highly likely typo: 2&>1 should be 2>&1

Hi,
It is here:
https://github.com/jpr5/ngrep/blob/2a9603bc67dface9606a658da45e1f5c65170444/scripts/multi.pl#L76C61-L76C61
(found using Debian Code Search)

Trailing chars in SIP traffic on SLL in 1.47

From [email protected]:

I test new version and now better filter packets, but trash in sip udp packets remained when -d any.
Example:
SIP/2.0 200 OK.
Via: SIP/2.0/UDP 213.170.84.105:5060;branch=z9hG4bK6476719.
From: sip:[email protected];tag=uloc-591da6eb-7745-005a02-3b80aa86-565d0154.
To: sip:0004*[email protected]:5060;transport=udp;user=phone;tag=2e4daf14d0b51a35.
Call-ID: [email protected].
CSeq: 1 OPTIONS.
User-Agent: Grandstream GXP1200 1.2.3.5.
Contact: sip:0004*[email protected]:5060;transport=udp;user=phone.
Allow: INVITE,ACK,CANCEL,BYE,NOTIFY,REFER,OPTIONS,INFO,SUBSCRIBE,UPDATE,PRACK,MESSAGE.
Supported: replaces, timer.
Content-Length: 0.
.
B6.
Contact: <si

(look to end with B6 Contact ...) - may be any readable trash. If i set as -d eno1 - all packets are clean.

Run as: bin/ngrep -d any -q -W byline '0004*2' udp port 5080 or udp port 5060 or udp port 5068

I think, trouble in SLL, need some backport from tcpdump or option for ip sockets, when only ip can be catched but without SLL.

compiling with --enable-tcpkill completes, but feature doesn't appear to work

Compiling on a fedora14 x64 machine with the --enable-tcpkill option completes and spits out a fresh binary. The option isn't quite enabled, however.. e.g.:

at first, it looks like it might be working:

[root@host] # ./ngrep -K | head -2
./ngrep: option requires an argument -- 'K'
usage: ngrep <-hNXViwqpe...blablabla

but running it with normal syntax does't:

[root@host] # ./ngrep -K 5 host 10.2.3.4 and port 22
usage: ngrep <-hNXViwqpevxl...blablabla

Not a huge concern, but it would be really fun to get this to work! Cheers!

compilation specifics:

[root@host] # ./configure --enable-tcpkill
...
CONFIG: tcpkill feature enabled
...
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config.h

[root@host] # make
make -C regex-0.12 regex.o
make[1]: Entering directory /root/ngrep-master/regex-0.12' gcc -g -DSTDC_HEADERS=1 -DHAVE_STRING_H=1 -DHAVE_ALLOCA_H=1 -DHAVE_ALLOCA=1 -I. -I. -c regex.c make[1]: Leaving directory/root/ngrep-master/regex-0.12'
gcc -DHAVE_CONFIG_H -DLINUX -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -D_BSD_SOURCE=1 -D__FAVOR_BSD=1 -Iregex-0.12 -I/usr/local/include/pcap -g -O2 -g -c ngrep.c
gcc -DHAVE_CONFIG_H -DLINUX -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -D_BSD_SOURCE=1 -D__FAVOR_BSD=1 -Iregex-0.12 -I/usr/local/include/pcap -g -O2 -g -c tcpkill.c
gcc -s -o ngrep ngrep.o tcpkill.o regex-0.12/regex.o -lpcap -lnet

Segfault on OSX Mavericks

To reproduce:

• On OSX Mavericks, install ngrep via homebrew or from source
• Run sudo ngrep -W byline -d lo0 port 4984

It will segfault.

A user said they would report the issue, but it looks like it never happened, so I'm reporting it.