assafmo/joincap

Merge multiple pcap files together, gracefully.

joincap

Merge multiple pcap files together, gracefully.

Installation

• Download a precompiled binary from https://github.com/assafmo/joincap/releases

• Or... Use go get:

  go get -u github.com/assafmo/joincap

• Or use Ubuntu PPA:

  curl -SsL https://assafmo.github.io/ppa/ubuntu/KEY.gpg | sudo apt-key add -
  sudo curl -SsL -o /etc/apt/sources.list.d/assafmo.list https://assafmo.github.io/ppa/ubuntu/assafmo.list
  sudo apt update
  sudo apt install joincap

Basic Usage

Usage:
  joincap [OPTIONS] InFiles...

Application Options:
  -v, --verbose  Explain when skipping packets or entire input files
  -V, --version  Print the version and exit
  -w=            Sets the output filename. If the name is '-', stdout will be used (default: -)
  -c=            An integer argument for limiting the pcap size (default: 9223372036854775807)

Help Options:
  -h, --help     Show this help message

Why?

I believe skipping corrupt packets is better than failing the entire merge job.
When using tcpslice or mergecap sometimes pcapfix is needed to fix bad input pcap files.

1. One option is to try and run merge (mergecap/tcpslice), if we get errors then run pcapfix on the bad pcaps and then run merge again.
   • Adds complexity (run -> check errors -> fix -> rerun)
   • (If errors) Demands more resources (pcapfix processes)
   • (If errors) Extends the total run time
2. Another option is to run pcapfix on the input pcap files and then merge.
   • Extends the total run time by a lot (read and write each pcap twice instead of once)
   • Demands more storage (for the fixed pcaps)
   • Demands more resources (pcapfix processes)
3. We can use pcapfix "in memory" with process substitution: mergecap -w out.pcap <(pcapfix -o /dev/stdout 1.pcap) <(pcapfix -o /dev/stdout 2.pcap).
   • Adds complexity (build a complex command line)
   • Demands more resources (pcapfix processes)
   • Harder for us to use pathname expansion (e.g. tcpslice -w out.pcap *.pcap)
   • We have to mind the command line character limit (in case of long pathnames)
   • Doesn't work for tcpslice (seeks the last packets to calculate time ranges - cannot do this with pipes)

Error handling: joincap vs mergecap vs tcpslice

Results

Use case	joincap	mergecap v2.4.5	tcpslice v1.2a3
Corrupt input global header	✔️	❌	❌
Corrupt input packet header	✔️	❌	❌
Unexpectd EOF (last packet data is truncated)	✔️	✔️	✔️
Input pcap has no packets (global header is ok, no first packet header)	✔️	✔️	❌
Input file size is smaller than 24 bytes (global header is truncated)	✔️	✔️	❌
Input file size is between 24 and 40 bytes (global header is ok, first packet header is truncated)	✔️	❌	❌
Input file doesn't exists	✔️	❌	❌
Input file is a directory	✔️	❌	❌
Input file end is garbage	✔️	✔️	❌
Input file is gzipped (.pcap.gz)	✔️	✔️	❌

Error outputs

Use case	Error outputs
Corrupt input global header	tcpslice: bad tcpdump file test_pcaps/bad_global.pcap: archaic pcap savefile format mergecap: The file "test_pcaps/bad_global.pcap" contains record data that mergecap doesn't support. (pcap: major version 0 unsupported)
Corrupt input packet header	tcpslice: Infinite loop? mergecap: The file "test_pcaps/bad_first_header.pcap" appears to be damaged or corrupt. (pcap: File has 2368110654-byte packet, bigger than maximum of 262144)
Unexpectd EOF (last packet data is truncated)
Input pcap has no packets (global header is ok, no first packet header)	tcpslice: Outputs empty pcap (Only global header)
Input file size is smaller than 24 bytes (global header is truncated)	tcpslice: bad tcpdump file test_pcaps/empty: truncated dump file; tried to read 4 file header bytes, only got 0
Input file size is between 24 and 40 bytes (global header is ok, first packet header is truncated)	tcpslice: bad status reading first packet in test_pcaps/partial_first_header.pcap: truncated dump file; tried to read 16 header bytes, only got 11 mergecap: The file "test_pcaps/partial_first_header.pcap" appears to have been cut short in the middle of a paket.
Input file doesn't exists	tcpslice: bad tcpdump file ./not_here: ./not_here: No such file or directory mergecap: The file "./not_here" doesn't exist.
Input file is a directory	tcpslice: bad tcpdump file examples: error reading dump file: Is a directory mergecap: "examples" is a directory (folder), not a file.
Input file end is garbage	tcpslice: problems finding end packet of file test_pcaps/bad_end.pcap
Input file is gzipped (.pcap.gz)	tcpslice: bad tcpdump file test_pcaps/ok.pcap.gz: unknown file format

How to reproduce

Use case	How to reproduce
Corrupt input global header	joincap -w out_joincap.pcap test_pcaps/bad_global.pcap mergecap -w out_mergecap.pcap test_pcaps/bad_global.pcap tcpslice -D -w out_tcpslice.pcap test_pcaps/bad_global.pcap
Corrupt input packet header	joincap -w out_joincap.pcap test_pcaps/bad_first_header.pcap mergecap -w out_mergecap.pcap test_pcaps/bad_first_header.pcap tcpslice -D -w out_tcpslice.pcap test_pcaps/bad_first_header.pcap
Unexpectd EOF (last packet data is truncated)	joincap -w out_joincap.pcap test_pcaps/unexpected_eof_on_first_packet.pcap mergecap -w out_mergecap.pcap test_pcaps/unexpected_eof_on_first_packet.pcap tcpslice -D -w out_tcpslice.pcap test_pcaps/unexpected_eof_on_first_packet.pcap joincap -w out_joincap.pcap test_pcaps/unexpected_eof_on_second_packet.pcap mergecap -w out_mergecap.pcap test_pcaps/unexpected_eof_on_second_packet.pcap tcpslice -D -w out_tcpslice.pcap test_pcaps/unexpected_eof_on_second_packet.pcap
Input pcap has no packets (global header is ok, no first packet header)	joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/no_packets.pcap mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/no_packets.pcap tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/no_packets.pcap
Input file size is smaller than 24 bytes (global header is truncated)	joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/empty mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/empty tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/empty joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/partial_global_header.pcap mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/partial_global_header.pcap tcpslic -De -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/partial_global_header.pcap
Input file size is between 24 and 40 bytes (global header is ok, first packet header is truncated)	joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/partial_first_header.pcap mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/partial_first_header.pcap tcpslic -De -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/partial_first_header.pcap
Input file doesn't exists	joincap -w out_joincap.pcap test_pcaps/ok.pcap ./not_here mergecap -w out_mergecap.pcap test_pcaps/ok.pcap ./not_here tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap ./not_here
Input file is a directory	joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/ mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/ tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/
Input file end is garbage	joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/bad_end.pcap mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/bad_end.pcap tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/bad_end.pcap
Input file is gzipped (.pcap.gz)	joincap -w out_joincap.pcap test_pcaps/ok.pcap.gz mergecap -w out_mergecap.pcap test_pcaps/ok.pcap.gz tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap.gz

Benchmarks

	Version	Speed	Time
mergecap	3.2.2	590MiB/s	0m5.632s
tcpslice	1.2a3	838MiB/s	0m3.666s
joincap	0.10.2	562MiB/s	0m5.462s

• Merging 3 files with total size of 2.99994GiB.
• Running on Linux 5.4.0-21-generic, with Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (with SSE4.2), with 31765 MB of physical memory, with locale C, with zlib 1.2.11.

joincap's Issues

Limit on merged file

There can also be a limit of number of packets that can be merged or limit on size of merged file i.e merged file should not be greater then specified limit

Arm64 support

I rummaged through several programs and scripts to combine pcap files, but your program is the best. I mainly use Kali Nethunter on an android phone and I would like to have this tool on it as well. Could you release an implementation for arm64? Since I have zero in compilation, but I haven’t found any ready-made solutions on the Internet. Thanks in advance.

error joincap don't merge file

Hi
when i run this command :

./joincap-macos64-v0.10.2 -w test.pcap test_*.pcap -v

nothing appear, just created a file with these errors

2020/10/12 14:03:42 joincap v0.10.2 - https://github.com/assafmo/joincap 2020/10/12 14:03:42 test_00001_20201003010017.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003011513.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003013009.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003014000.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003014800.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003015600.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003020409.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003021202.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003022000.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003022800.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003024400.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003025200.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003030000.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003030800.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003031600.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003032400.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003034000.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003034800.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003035600.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003040400.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003041212.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003042000.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003043600.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003044400.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003045200.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003050000.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003050800.pcap: done (closing) 2020/10/12 14:03:42 test_00001_20201003050800.pcap: EOF before first packet (skipping this file) 2020/10/12 14:03:42 test_00001_20201003051600.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003053200.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003054000.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003054800.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003055600.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003060400.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003061201.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003062800.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003063600.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003064404.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003065200.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003070000.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003070800.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003072401.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003073200.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003074000.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003074800.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003075600.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003080401.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003082000.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003082800.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003083600.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003084400.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003085200.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003090000.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003091600.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003092400.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003093200.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003094000.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003094800.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201003095600.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201005100018.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201005101509.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201005102146.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201005102300.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201005103100.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201005103443.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201005103914.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201005104750.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201005105430.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201005105437.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201005105957.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201006104518.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201006105300.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201006110100.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201007104518.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201007105305.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201007110100.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201008104512.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201008105300.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201008110100.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201009104519.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00001_20201009105300.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00002_20201003063344.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00002_20201003063900.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00002_20201006110618.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00003_20201006110856.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 test_00004_20201006111138.pcap: Unknown magic a0d0d0a (skipping this file) 2020/10/12 14:03:42 merging 0 input files of size 24 B 2020/10/12 14:03:42 writing to test.pcap

FeatReq: follow mode

Since this appears to be the only tool currently allowing a directory to specify the source of pcaps, it would be useful to have a "follow" mode where joincap can merge existing files, but watch the final file for growth as well as watch the specified directory for new pcaps to join. The goal would be to allow one tool to write pcaps while joincap to reads and follows what's written in near-realtime for streaming to other tools that consume pcaps.

If implemented, it may also be necessary to have a start-time option that suppresses output of any packets prior to a given date/time. That would allow one to restart an aborted joincap in follow-mode from a given point without needing to clear out already processed files from the source directory.

Print help if no files were given

TODO

Permission Denied Issue

Hello @assafmo ,

I have been using joincap as a local development tool for merging granular pcap files into larger pcap files. However, when I tried to deploy it onto a linux environment (it was working perfectly on windows locally), I am unable to merge any pcap files. Below is the verbose output, and this is the same output when sudo'd

user@machine:~$ joincap -v -w=output.pcap input.pcap
user@machine:~$ sudo joincap -v -w=output.pcap input.pcap

both yield:

2019/04/17 17:09:20 joincap v0.10.1 - https://github.com/assafmo/joincap
2019/04/17 17:09:20 input.pcap: open input.pcap: permission denied (skipping this file)
2019/04/17 17:09:20 merging 0 input files of size 0 B
2019/04/17 17:09:20 cannot open output.pcap for writing: open output.pcap: permission denied

This happens regardless of user and directory permissions (I'm doing this in the home directory of my user, so it follows that I have read/write privileges). Any ideas as to what is going on here? I would love to get this tool working on this environment (ubuntu 16.04). Any help would be extremely appreciated!

Thanks,
Jonathan

Support pcapng file format

TODO