A solution to the problem posed here: http://www.tsurucapital.com/en/code-sample.html
This application processes packet data stored in a pcap dump file using iteratees. The ideas are described in this paper. Several choices are available for the parsing of the packets themselves: using the iteratee
library, the attoparsec
library, or just using the naive bytestring
operations and the built-in read
.
The source files are structured as follows:
filepath | description |
---|---|
app/Main.hs |
CL executable |
src/Parsing/ |
attoparsec , iteratee , and vanilla ByteString implementation of the quote parser |
src/Streaming/ |
iteratee and myiteratee implementations of the pcap streaming and quote reordering |
iteratee-0.8.9.6/ |
clone of the iteratee library, but with minor modification to satisfy stack's solver |
myiteratee/ |
my own implementation of iteratees |
Make sure you have stack
on your system, and run
stack build
which will install the necessary dependencies and create the executables. The program can then be run with
stack exec tsuru-sample-exe -- PROGRAM_OPTS PCAP_FILE
The following can also be viewed by running stack exec tsuru-sample-exe -- --help
.
option | description |
---|---|
-h ,--help |
Show this help text |
-r ,--reorder |
Reorder packets based on quote accept time |
-s ,--start INT |
Starting index for packet (default: 0) |
-n ,--number INT |
Maximum number of packets to accept |
--silent |
Do not log packets |
-S ,--streaminglib SLIB |
Streaming library to use. Available options: iteratee (default), myiteratee . |
-P ,--parsinglib PLIB |
Parsing library to use. Available options: attoparsec (default), iteratee , bytestring . |
-c ,--chunksize INT |
Chunk size when using the iteratee streaming library (default: 4096) |
--noparse |
Directly print out packets without parsing |
--buffersize INT |
Size of output buffer. System dependent if unspecified. |
The default settings should give optimal performance. The iteratee
library supports chunk processing, which allows faster quote reordering. The fastest packet parser out of the three options is attoparsec
, as it is specialized to parsing bytestrings, and we do not use backtracking. Further speed boost can be achieved by fine-tuning the --chunksize
and the --buffersize
option for your particular system.
Currently, the program works with a hard-coded list (in Parsing.Base
) of Data.Time.LocalTime.TimeZone
objects representing the possible timezones of the sources of the quote data. The program ships with only Tokyo Time (UTC+9) in the list, but the user is free to add more, and the processor will automatically decide for each individual packet which timezone it comes from.