kiminewt/pyshark

Python wrapper for tshark, allowing python packet parsing using wireshark dissectors

pyshark

Python wrapper for tshark, allowing python packet parsing using wireshark dissectors.

Extended documentation: http://kiminewt.github.io/pyshark

Looking for contributors - for various reasons I have a hard time finding time to maintain and enhance the package at the moment. Any pull-requests will be reviewed and if any one is interested and is suitable, I will be happy to include them in the project. Feel free to mail me at dorgreen1 at gmail.

There are quite a few python packet parsing modules, this one is different because it doesn't actually parse any packets, it simply uses tshark's (wireshark command-line utility) ability to export XMLs to use its parsing.

This package allows parsing from a capture file or a live capture, using all wireshark dissectors you have installed. Tested on windows/linux.

Installation

Version support

Python 3.7+ is supported. An unsupported Python 2 version exists as pyshark-legacy.

Supports all modern versions of tshark / wireshark but certain features may be unavailable on older versions.

All Platforms

Simply run the following to install the latest from pypi

pip install pyshark

Or install from the git repository:

git clone https://github.com/KimiNewt/pyshark.git
cd pyshark/src
python setup.py install

Mac OS X

You may have to install libxml which can be unexpected. If you receive an error from clang or an error message about libxml, run the following:

xcode-select --install
pip install libxml

You will probably have to accept a EULA for XCode so be ready to click an "Accept" dialog in the GUI.

Usage

Reading from a capture file:

>>> import pyshark
>>> cap = pyshark.FileCapture('/tmp/mycapture.cap')
>>> cap
<FileCapture /tmp/mycapture.cap (589 packets)>
>>> print cap[0]
Packet (Length: 698)
Layer ETH:
        Destination: BLANKED
        Source: BLANKED
        Type: IP (0x0800)
Layer IP:
        Version: 4
        Header Length: 20 bytes
        Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
        Total Length: 684
        Identification: 0x254f (9551)
        Flags: 0x00
        Fragment offset: 0
        Time to live: 1
        Protocol: UDP (17)
        Header checksum: 0xe148 [correct]
        Source: BLANKED
        Destination: BLANKED
  ...

Other options

• param keep_packets: Whether to keep packets after reading them via next(). Used to conserve memory when reading large caps.
• param input_file: Either a path or a file-like object containing either a packet capture file (PCAP, PCAP-NG..) or a TShark xml.
• param display_filter: A display (wireshark) filter to apply on the cap before reading it.
• param only_summaries: Only produce packet summaries, much faster but includes very little information
• param disable_protocol: Disable detection of a protocol (tshark > version 2)
• param decryption_key: Key used to encrypt and decrypt captured traffic.
• param encryption_type: Standard of encryption used in captured traffic (must be either 'WEP', 'WPA-PWD', or 'WPA-PWK'. Defaults to WPA-PWK.
• param tshark_path: Path of the tshark binary

Reading from a live interface:

>>> capture = pyshark.LiveCapture(interface='eth0')
>>> capture.sniff(timeout=50)
>>> capture
<LiveCapture (5 packets)>
>>> capture[3]
<UDP/HTTP Packet>

for packet in capture.sniff_continuously(packet_count=5):
    print('Just arrived:', packet)

Other options

• param interface: Name of the interface to sniff on. If not given, takes the first available.
• param bpf_filter: BPF filter to use on packets.
• param display_filter: Display (wireshark) filter to use.
• param only_summaries: Only produce packet summaries, much faster but includes very little information
• param disable_protocol: Disable detection of a protocol (tshark > version 2)
• param decryption_key: Key used to encrypt and decrypt captured traffic.
• param encryption_type: Standard of encryption used in captured traffic (must be either 'WEP', 'WPA-PWD', or 'WPA-PWK'. Defaults to WPA-PWK).
• param tshark_path: Path of the tshark binary
• param output_file: Additionally save captured packets to this file.

Reading from a live interface using a ring buffer

>>> capture = pyshark.LiveRingCapture(interface='eth0')
>>> capture.sniff(timeout=50)
>>> capture
<LiveCapture (5 packets)>
>>> capture[3]
<UDP/HTTP Packet>

for packet in capture.sniff_continuously(packet_count=5):
    print('Just arrived:', packet)

Other options

• param ring_file_size: Size of the ring file in kB, default is 1024
• param num_ring_files: Number of ring files to keep, default is 1
• param ring_file_name: Name of the ring file, default is /tmp/pyshark.pcap
• param interface: Name of the interface to sniff on. If not given, takes the first available.
• param bpf_filter: BPF filter to use on packets.
• param display_filter: Display (wireshark) filter to use.
• param only_summaries: Only produce packet summaries, much faster but includes very little information
• param disable_protocol: Disable detection of a protocol (tshark > version 2)
• param decryption_key: Key used to encrypt and decrypt captured traffic.
• param encryption_type: Standard of encryption used in captured traffic (must be either 'WEP', 'WPA-PWD', or 'WPA-PWK'. Defaults to WPA-PWK).
• param tshark_path: Path of the tshark binary
• param output_file: Additionally save captured packets to this file.

Reading from a live remote interface:

>>> capture = pyshark.RemoteCapture('192.168.1.101', 'eth0')
>>> capture.sniff(timeout=50)
>>> capture

Other options

• param remote_host: The remote host to capture on (IP or hostname). Should be running rpcapd.
• param remote_interface: The remote interface on the remote machine to capture on. Note that on windows it is not the device display name but the true interface name (i.e. \Device\NPF_..).
• param remote_port: The remote port the rpcapd service is listening on
• param bpf_filter: A BPF (tcpdump) filter to apply on the cap before reading.
• param only_summaries: Only produce packet summaries, much faster but includes very little information
• param disable_protocol: Disable detection of a protocol (tshark > version 2)
• param decryption_key: Key used to encrypt and decrypt captured traffic.
• param encryption_type: Standard of encryption used in captured traffic (must be either 'WEP', 'WPA-PWD', or 'WPA-PWK'. Defaults to WPA-PWK).
• param tshark_path: Path of the tshark binary

Accessing packet data:

Data can be accessed in multiple ways. Packets are divided into layers, first you have to reach the appropriate layer and then you can select your field.

All of the following work:

>>> packet['ip'].dst
192.168.0.1
>>> packet.ip.src
192.168.0.100
>>> packet[2].src
192.168.0.100

To test whether a layer is in a packet, you can use its name:

>>> 'IP' in packet
True

To see all possible field names, use the packet.layer.field_names attribute (i.e. packet.ip.field_names) or the autocomplete function on your interpreter.

You can also get the original binary data of a field, or a pretty description of it:

>>> p.ip.addr.showname
Source or Destination Address: 10.0.0.10 (10.0.0.10)
# And some new attributes as well:
>>> p.ip.addr.int_value
167772170
>>> p.ip.addr.binary_value
b'\n\x00\x00\n'

Decrypting packet captures

Pyshark supports automatic decryption of traces using the WEP, WPA-PWD, and WPA-PSK standards (WPA-PWD is the default).

>>> cap1 = pyshark.FileCapture('/tmp/capture1.cap', decryption_key='password')
>>> cap2 = pyshark.LiveCapture(interface='wi0', decryption_key='password', encryption_type='wpa-psk')

A tuple of supported encryption standards, SUPPORTED_ENCRYPTION_STANDARDS, exists in each capture class.

>>> pyshark.FileCapture.SUPPORTED_ENCRYPTION_STANDARDS
('wep', 'wpa-pwd', 'wpa-psk')
>>> pyshark.LiveCapture.SUPPORTED_ENCRYPTION_STANDARDS
('wep', 'wpa-pwd', 'wpa-psk')

Reading from a file using a display filter

Pyshark display filters can be helpful in analyzing application focused traffic. BPF filters do not offer as much flexibility as Wireshark's display filters.

>>> cap1 = pyshark.FileCapture('/tmp/capture1.cap', display_filter="dns")
>>> cap2 = pyshark.LiveCapture(interface='en0', display_filter="tcp.analysis.retransmission")

License

This project is licensed under MIT. Contributions to this project are accepted under the same license.

pyshark's Issues

Config.ini is not copied to the pyshark directory during installation

Hi,

I'm not sure if it is a problem with my my OS X 10.9 environment, or a problem with the pyshark installation, but config.ini was not copied to /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pyshark-0.1-py2.7.egg/pyshark/ as part of the installation.

I had to manually copy it there (and modify it per my pull request) in order to get pyshark to work on my system. I am not very familiar with how to troubleshoot a setuptools issue, but I did review the installation log and did not see any errors or acknowledgements that config.ini was successfully copied to the above directory.

Let me know if you want me to run any tests or need any more information.

PyPI version missing fix

Commit 4646e7c fixed a syntax error in RemoteCapture.py . pip install pyshark pulls a version that doesn't have this fix included. Could you update PyPI?

display_filter does not work

I started to explore pyshark today and found that whenever I use the display filter, an empty file is returned. I am using

• TShark (Wireshark) 1.99.0 (v1.99.0-rc1-1493-g2d74838 from master) directly build from a recent git clone.
• pyshark version 0.2.7 installed through pip

I found that tshark is called with "-R display_filter", but this should either be called with "-2 -R display_filter" or with "-Y display_filter" instead to yield a result.

Replacing -R with -Y seems to work (-2 -R not for some reason):

    def get_parameters(self, packet_count=None):
        """
        Returns the special tshark parameters to be used according to the configuration of this class.
        """
        params = []
        if self.display_filter:
            params += ['-Y', self.display_filter]
        if packet_count:
            params += ['-c', str(packet_count)]
        return params

Git diff:

diff --git a/src/pyshark/capture/capture.py b/src/pyshark/capture/capture.py
index 1ec2db4..094e264 100644
--- a/src/pyshark/capture/capture.py
+++ b/src/pyshark/capture/capture.py
@@ -147,7 +147,7 @@ class Capture(object):
         """
         params = []
         if self.display_filter:
-            params += ['-R', self.display_filter]
+            params += ['-Y', self.display_filter]
         if packet_count:
             params += ['-c', str(packet_count)]
         return params
@@ -157,4 +157,4 @@ class Capture(object):
             yield self.next()

     def __repr__(self):
-        return '<%s (%d packets)>' %(self.__class__.__name__, len(self._packets))
\ No newline at end of file
+        return '<%s (%d packets)>' %(self.__class__.__name__, len(self._packets))

not able to setup correctly

chrispooh@ubuntu:~/Desktop/KimiNewt-pyshark-2795378/src$ sudo python setup.py
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help

error: no commands supplied

Any ideas where went wrong?

sniff_continuously has a backlog of packets and doesn't generate them immediately

I see this both in Python 2.7 and 3.4.

In this commit: goretkin@9ef4ab5 I've added a simple test to illustrate the issue and see if anyone else experiences it. The same behavior occurs if you use apply_on_packets callback mechanism.

In my case, I am debugging a USB protocol, which generates pairs of "packets" at once. pyshark isn't dropping any packets, but the sniff_continuously generator appears to yield only once per burst of "packets", even if there's more than one.

Here's how I trigger the behavior. tshark_stub emulates tshark and emits two "packets" in quick succession when it receives SIGUSR1. test_continuous_capture.py sets the tshark path to be this tshark_stub and then prints a line each time sniff_continously yields a new "packet".

So in Terminal 1, I run test_continuous_capture.py

pyshark/tests/capture$ python test_continuous_capture.py 
[2015-02-12 09:32:24.362951] DEBUG: LiveCapture: Creating TShark subprocess with parameters: ./tshark_stub -n -T pdml -i 1
[2015-02-12 09:32:24.365716] DEBUG: LiveCapture: TShark subprocess created

On Terminal 2, I do ps aux | grep tshark_stub to find the PID of the stub, and with an eye on Terminal 1 I do a couple of these

~$ kill -SIGUSR1 8018
~$ kill -SIGUSR1 8018

I expect four lines to be printed in Terminal 1, but instead I only see

[2015-02-12 09:32:24.365716] DEBUG: LiveCapture: TShark subprocess created
sniffed packet
sniffed packet

Finally, in Terminal 2, I run kill 8018. Terminal 1 shows

[2015-02-12 09:32:24.365716] DEBUG: LiveCapture: TShark subprocess created
sniffed packet
sniffed packet
sniffed packet
sniffed packet
[2015-02-12 09:37:52.599719] DEBUG: LiveCapture: EOF reached (sync)

A similar thing happens if tshark_stub emits only one packet at a time. The first packet doesn't cause a yield, but every packet thereafter does. But it's forever offset by 1, the packet just yielded and the actual packet sniffed, until EOF.

Some problems

Hi

I have some issused I've encountered using pyshark:

1. Even when i use lazy to read caps file, if the file is larger then 5MB i get Memory error, is there no way to work with larger files?
2. Why using tshark and not editcap, isn't it work better with cap files?
3. Most important, is there a why to filter on all the packets? like get all the browser packets with hostname = 'x'?

Thanks,

WindowsError exception not available on non-windows platforms

Traceback (most recent call last):
File "", line 1, in
File "pyshark/capture/live_capture.py", line 91, in sniff_continuously
except WindowsError:
NameError: global name 'WindowsError' is not defined

You should probably catch OSError instead of WindowsError

Cheers,

Mario

Large pcap file processing with pyshark

Greetings:

Does pyshark work with large pcap files? I found that each time when I use pyshark to parse a large file python eats all memory on my laptop.

Any workaround for large pcap files?

capture stderr and exit code from tshark

I don't know where to incorporate it, but I think the tshark process' stderr should be logged and possibly displayed in the event that tshark exits with a failure. Currently, the default is to be completely silent if the interface isn't accessible (because of permissions or non-existance), for example.

EDIT I just found this discussion #26

Setup guide required

Hi,
Sorry I am new here, can anyone tell me the basic steps I need to run pyshark? I am getting error ImportError: No module named pyshark.
What should be installed on my machine? I have only installed Python 2.7 and I am on Ubuntu 14.04lts.
Thanking you in advance.

-Okay I got it. I installed all requirements from requirements.txt using pip.

immem_capture send packet one by one, the ack number failed compute normal

import pcap
import pyshark
import os

def test():
    pcapfile = os.path.abspath("/home/engine/pcap/dshell/good.pcap")
    pcap_capture = pcap.pcap(pcapfile)
    immem_capture = pyshark.InMemCapture()
    count = 0

    try:
        while True:
            count += 1
            packet = pcap_capture.next()
            if packet is None:
                break
            ts ,packet_data = packet
            tshark_pkt = immem_capture.feed_packet(packet_data)
            seq = tshark_pkt[2].seq

            print("packet count %d seq is %s " %(count, seq ))

    except Exception as e:
        print e

if __name__ == "__main__":

    test()

using thark the seq number is right

engine@engine:~$ tshark -r ~/pcap/dshell/good.pcap
1 0.000000 2001:6f8:102d:0:2d0:9ff:fee3:e8de -> 2001:6f8:900:7c0::2 TCP 94 59201→80 [SYN] Seq=0 Win=5760 Len=0 MSS=1440 SACK_PERM=1 TSval=664232 TSecr=0 WS=32
2 0.000086 2001:6f8:900:7c0::2 -> 2001:6f8:102d:0:2d0:9ff:fee3:e8de TCP 82 80→59201 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1432 SACK_PERM=1
3 0.000374 2001:6f8:102d:0:2d0:9ff:fee3:e8de -> 2001:6f8:900:7c0::2 TCP 74 59201→80 [ACK] Seq=1 Ack=1 Win=184320 Len=0
4 0.009619 2001:6f8:102d:0:2d0:9ff:fee3:e8de -> 2001:6f8:900:7c0::2 HTTP 314 GET / HTTP/1.0
5 0.014704 2001:6f8:900:7c0::2 -> 2001:6f8:102d:0:2d0:9ff:fee3:e8de TCP 1506 [TCP segment of a reassembled PDU]
6 0.014733 2001:6f8:900:7c0::2 -> 2001:6f8:102d:0:2d0:9ff:fee3:e8de HTTP 901 HTTP/1.1 200 OK (text/html)
7 0.014835 2001:6f8:900:7c0::2 -> 2001:6f8:102d:0:2d0:9ff:fee3:e8de TCP 74 80→59201 [FIN, ACK] Seq=2260 Ack=241 Win=65535 Len=0
8 0.015366 2001:6f8:102d:0:2d0:9ff:fee3:e8de -> 2001:6f8:900:7c0::2 TCP 74 59201→80 [ACK] Seq=241 Ack=1433 Win=274944 Len=0
9 0.015371 2001:6f8:102d:0:2d0:9ff:fee3:e8de -> 2001:6f8:900:7c0::2 TCP 74 59201→80 [ACK] Seq=241 Ack=2260 Win=366592 Len=0
10 0.029609 2001:6f8:102d:0:2d0:9ff:fee3:e8de -> 2001:6f8:900:7c0::2 TCP 74 59201→80 [FIN, ACK] Seq=241 Ack=2261 Win=366592 Len=0

using the test py , seq is not right. How can put pcap one by one , but still get the seq number right?

python test/test_pcap2shark.py
packet count 1 seq is 0
packet count 2 seq is 0
packet count 3 seq is 1
packet count 4 seq is 1
packet count 5 seq is 1
packet count 6 seq is 1
packet count 7 seq is 1
packet count 8 seq is 1
packet count 9 seq is 1
packet count 10 seq is 1

Converting threaded code to asyncio

I've been thinking of converting the code revolving around reading from the tshark subprocess (and parsing the xml) from its current state (threads) into the far-better async io paradigm.

Particularly, I thought about using asyncio (the trollius version, to keep py2.7 support) which has good support for reading from subprocesses, even on Windows. I have a basic implementation working already.

I pose this question because converting to asyncio may hurt the "interactive" usage of the package, as eventloops don't work well with them and so code like:

for packet in cap:
    # Do something with packet

May turn into something more like:

cap.for_each_packet(do_something_with_packet_callback)
# This will now block and call the function for each packet.

Though something can be implemented to still perform the interactivity and simple code as above, it will require to constantly start and stop the event loop, which might be okay if you're working interactively.

The advantages to this sort of approach is twofold:

• Easy insertion of pyshark into other packages that work with eventloops (be it asyncio, twisted, tornado..)
• Removing the issues and uncertainty (which you can see in past issues) that comes with threads.

I'd be happy to hear any opinions on the matter.

accessing frame information

I'd like to access the frame information from the dissected packet.
In the pdml it is there (allthough not strictly speaking a protocol layer), so why can't I acces it?

>>> cap=pyshark.FileCapture(filename)
>>> p=cap[21]
>>> p['frame']._field_names
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/src/pyshark/src/pyshark/packet/packet.py", line 47, in __getitem__
    raise KeyError('Layer does not exist in packet')
>>> p['frame'].frame_number
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/src/pyshark/src/pyshark/packet/packet.py", line 47, in __getitem__
    raise KeyError('Layer does not exist in packet')
KeyError: 'Layer does not exist in packet'

I guess it has todo with this fragment from 'tshark_xml.py':

def _packet_from_pdml_packet(pdml_packet):
    layers = [Layer(proto) for proto in pdml_packet.proto]
    geninfo, frame, layers = layers[0], layers[1], layers[2:]
    frame.raw_mode = True
    return Packet(layers=layers, length=geninfo.get_field_value('len'), sniff_time=geninfo.get_field_value('timestamp', raw=True),
                  captured_length=geninfo.get_field_value('caplen'), interface_captured=frame.get_field_value('interface_id'))

Is there something against 'geninfo' and 'frame' to be also accessible as packet information?

lxml.etree.XMLSyntaxError: internal error: Huge input lookup

I met this exception info when I was parsing a captured .pcap file.

It seems that the XML file of some packets have exceeded the 9.5Mb limit of lxml lib.
(http://making-security-measurable.1364806.n2.nabble.com/libxml2-lxml-Parsing-large-9-5mb-XML-Documents-td7580595.html)

According to the solution above, I added two new lines in "/Lib/site-packages/pyshark/tshark/tshark_xml.py" at line 26:
"
parser = lxml.objectify.makeparser(huge_tree=True)
xml_pkt = lxml.objectify.fromstring(xml_pkt, parser)
"
then the problem was solved.

Pyshark only work good with Linux??

Hi All,
I am not able to use this pyshark Library with windows environment .
import pyshark

cap = pyshark.FileCapture('D:\mallik.pcap')
print cap.len()
for i in cap:
print i

The Result is as follows.
C:\Python27\python.exe "D:/Trailproject/add two number.py" 2 3 4
0
Traceback (most recent call last):
File "D:/Trailproject/add two number.py", line 5, in
for i in cap:
File "C:\Python27\lib\site-packages\pyshark\capture\capture.py", line 159, in _packets_from_tshark_sync
tshark_process = existing_process or self.eventloop.run_until_complete(self._get_tshark_process())
File "C:\Python27\lib\site-packages\trollius\base_events.py", line 300, in run_until_complete
return future.result()
File "C:\Python27\lib\site-packages\trollius\futures.py", line 287, in result
raise self._exception
WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect

Process finished with exit code 1

Please some one help me to get the information whether it can able useful in windows., if so let me know it with example.
I can see other issues far beyond the basic it means it is working for them .
I am use Windows 7 as my OS.

Atrribute Error in packet.py

Pyshark 0.3.5 on Python 3:
While using packet[packet.transport_layer].srcport I received this error:

File "/usr/local/lib/python3.4/dist-packages/pyshark/packet/packet.py", line 49, in getitem
if layer.layer_name == item.lower():
AttributeError: 'NoneType' object has no attribute 'lower'

Can not parse 'X-Requested-With' from http header

Hi, I am using pyshark to parse some pcap files. But it seems that pyshark is missing from information. If I use wireshark, I can see 'X-Request-With' in the http header. While I can not see it if I use pyshark to parse the same file.

Same name for diffrent strings

Hi

When working with packet you can see that there are more then one field with the same name, like :"addr".
For example:
packet.eth.addr also mean packet.eth.src and packet.eth.dst

But when printing packet.eth.addr you will "obvisuly" get only when of them, is there a way to print both ver?

Thanks
Michal

Not able to print the even the basic packet information from the saved file

Hi ,
I have downloaded the pyshark from pip and I am using the python 2.7 version .
I am trying to execute the file capture option using the following code
import SSHLibrary
import pyshark

cap=pyshark.FileCapture('D:\mallik.cap')
print cap[0]

but I am not getting any report from that , I am getting the following error while executing the script.

C:\Python27\python.exe D:/Phython/openssh.py
Traceback (most recent call last):
File "D:/Phython/openssh.py", line 5, in
print cap[0]
File "C:\Python27\lib\site-packages\pyshark\capture\file_capture.py", line 58, in getitem
self.next()
File "C:\Python27\lib\site-packages\pyshark\capture\file_capture.py", line 48, in next
packet = self._packet_generator.send(None)
File "C:\Python27\lib\site-packages\pyshark\capture\capture.py", line 159, in _packets_from_tshark_sync
tshark_process = existing_process or self.eventloop.run_until_complete(self._get_tshark_process())
File "C:\Python27\lib\site-packages\trollius\base_events.py", line 300, in run_until_complete
return future.result()
File "C:\Python27\lib\site-packages\trollius\futures.py", line 287, in result
raise self._exception
WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect

Please provide me the information to resolve this issue

Problems reading a SIP pcap file

Hi,
This is my first short at pyshark and I find it to be pretty smooth except for SIP layer.

Has sip layer been tested well since I have encountered problems printing sip fields through a construct as simple as packet.sip.Call-ID

It throws AttributeError () when in layer.py , method get_field returns a None. This happens only for sip and never for IP layer for example and I see the following:

line 32, in
print('adnan sip.Call-ID=',packet.sip.Call-ID)
File "/home/python/pyshark/pcap3.2/lib/python3.2/site-packages/pyshark-0.3.1-py3.2.egg/pyshark/packet/layer.py", line 50, in getattr
raise AttributeError()
AttributeError

You can try reading the following wireshark sample and see if it works for you:
http://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=aaa.pcap

Apart from sip, for IP layer I never encountered any of above issues.

I have reproduced it in python2.7 and python3.2 running https://github.com/KimiNewt/pyshark/tarball/master i.e. ver 0.3.1 as well as latest git commit: commit f217d0e
Author: Dor Green [email protected]
Date: Thu Sep 25 17:24:59 2014 +0300

Just pasting the following code that failed:
import pyshark
trace='aaa.pcap'
cap = pyshark.FileCapture(trace)
i=1;
for packet in cap:
for l in packet.layers:
if l.layer_name == 'sip':
#print(dir(packet.sip))
print('adnan sip.Method=',packet.sip.Method)
#print('adnan sip.Method i =',packet[i].Method)
print('adnan sip.Call-ID=',packet.sip.Call-ID)
#print('adnan sip.Call-ID=',packet['sip'].sip.Call-ID)
print('-----packet nr ',i,'--------------------')
print('-------------------------')
print('new try',packet['sip'].From)
print('new try2',packet['sip'].msg_hdr)
#try:

                    #print('--sip.Method',packet['sip'].Method)
                    #print('--sip.Call-ID', packet['sip'].Call-Id)
                    #except Exception:
                    #       pass
                    #print('--sip.Call-ID', packet.sip.Call-ID)
            '''
            if l.layer_name == 'ip':
                     print('--ip.src= ',packet['ip'].src)
                     print('--ip.dst=',packet['ip'].dst)
            '''
    i=i+1

Br
Adnan

config.ini missing

pyshark/config.ini is not in the 0.2.1 release file. Also, is not in the Python Package one (easy_install).

stderr buffer filling up

Hi,
Is it possible for an unread stderr pipe to fill up causing a deadlock?
How about a different thread to read from stdrr pipe ..

invalid thread id

Python 2.7.6 (v2.7.6:3a1db0d2747e, Nov 10 2013, 00:42:54)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import pyshark
capture = pyshark.LiveCapture(interface='en0')
capture.sniff(timeout=10)
Traceback (most recent call last):
File "", line 1, in
File "/Library/Python/2.7/site-packages/pyshark-0.2-py2.7.egg/pyshark/capture/live_capture.py", line 44, in sniff
sniff_thread.raise_exc(StopIteration)
File "/Library/Python/2.7/site-packages/pyshark-0.2-py2.7.egg/pyshark/utils.py", line 81, in raise_exc
_async_raise(self._get_my_tid(), exctype)
File "/Library/Python/2.7/site-packages/pyshark-0.2-py2.7.egg/pyshark/utils.py", line 28, in _async_raise
raise ValueError("invalid thread id")
ValueError: invalid thread id

get_tshark_path() raise exception on missing directory.

Windows 7 32bit which have only 'programFiles' directory raises exceptions when setting os.environ['ProgramFiles(x86)']

Using tshark's decode_as option

There does not appear to be an option to use decode_as from pyshark.

How to get posted content from POST request

Hi,

I'm trying to parse POST request and want to extract posted content from those requests. Some of those contents are json strings.

What I got using pyshark is something like this.

Layer JSON:
    Member Key: "userid"
    Object
    String value: 14230381863292564
    Array
    Member Key: "game"
    Member Key: "version"
    Member Key: "othergames"
    String value: com.droidhang.ph
    String value: 2
    String value: com.droidhang.ph
    String value: stericson.busybox
    String value: phongit.quickreboot

What I hope to get is the original json string, like

{
"userid": "14230381863292564",
"game": "com.droidhang.ph",
"version": "com.droidhang.ph",
"othergames": 
                      [
                         "com.droidhang.ph",
                         "stericson.busybox",
                         "phongit.quickreboot"
                       ]
}

Could you please give me some help?

Getting error when using on windows with python 2.7

Hi,

Thanks for making this tool

Actually i dont have choice and I need to use it on windows. But i am getting this error. Can you please help what I am doing wrong ? Also i beginner

file capture functionality

Hi,

I have a huge pcap file and was trying to parse it with pyshark. I tried to load a last packet in pcap and found that python used all free memory on my laptop.
I tried multiple options with lazzy/keep_packets but always the same result. Is this a normal behaviour? Is it possible that pyshark will only keep the last packet in memory?

param lazy = True
param keep_packets = True

Thanks

Can not get request method

Hi,

I have no idea about how to get 'request method' of a http package.

# pkg is a http package
In [70]: pkg.http
Out[70]: <HTTP Layer>

When I try to access the 'request method' filed, it gives me an error

In [71]: pkg.http.request.method
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-71-46ad3cf73be3> in <module>()
----> 1 pkg.http.request.method

/usr/local/lib/python2.7/site-packages/pyshark-0.3.4-py2.7.egg/pyshark/packet/layer.pyc in __getattr__(self, item)
    110
    111     def __getattr__(self, item):
--> 112         return getattr(self.main_field, item)
    113
    114

AttributeError: 'LayerField' object has no attribute 'method'

Then I tried this way, it still does not work

In [74]: pkg.http.http.request.method
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-74-4c2a015dbed7> in <module>()
----> 1 pkg.http.http.request.method

/usr/local/lib/python2.7/site-packages/pyshark-0.3.4-py2.7.egg/pyshark/packet/layer.pyc in __getattr__(self, item)
    140         val = self.get_field(item)
    141         if val is None:
--> 142             raise AttributeError()
    143         if self.raw_mode:
    144             return val.raw_value

AttributeError:

I have to use the following way to access that field. It works but I don't think it's the correct way.

In [73]: getattr(pkg.http, 'http.request.method')
Out[73]: 'GET'

Could you tell me how to access that field?

Thanks

packets_from_tshark() expects callback parameter, but is not given one from sniff_continuously

It looks like sniff_continuously calls packets_from_tshark without the required parameter callback.

Steps to reproduce:

>>> import pyshark
>>> capture = pyshark.LiveCapture(interface='eth0')
>>> capture.sniff_continuously(packet_count=5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pi/.virtualenvs/myproject/local/lib/python2.7/site-packages/pyshark/capture/live_capture.py", line 57, in sniff_continuously
    return self.packets_from_tshark(packet_count=packet_count)
TypeError: packets_from_tshark() takes at least 2 arguments (2 given)

pcap_reader and Python3

I was able to get pyshark working great on python 2.7.* But when I tried with Python3, each time I read in a pcap with cap=pyshark.FileCapture('my_file.pcap') I would get this error:
import pyshark
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pyshark/init.py", line 3, in
from pyshark.capture.remote_capture import RemoteCapture
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pyshark/capture/remote_capture.py", line 21
"""
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 592-593: malformed \N character escape

Resolution:
I played with the comment in the remote_capture.py file, but in the end, by deleting the comment completely, I was able to work around the problem and it ran fine in python3.

It looks like it's the line: \Device\NPF_ that is giving the problem.

Once i remove the \N or Escape it, it works fine. Odd, I didn't think python would care about a \N in a comment.

Version 0.3.3 in Python2

My code from version 0.2.7 doesn't work with version 0.3.3, although the docs on the net doesn't seem to have changed.
Wasn't 0.3.3 meant for Python3? (Just guessing from the number.)

This is what I get when using pyshark 0.3.3:
Exception AttributeError: "'FileCapture' object has no attribute 'running_processes'" in ignored
[...]
capture = pyshark.FileCapture(input_file=capture_file, lazy=True, keep_packets=False)
TypeError: init() got an unexpected keyword argument 'lazy'

Everything works fine in 0.2.7.

Cool module.

Error in the Installation in windows 8

Below Error is appearing which is not letting this module install on Windows 8. Please help.

Downloading https://pypi.python.org/packages/source/l/lxml/lxml-3.3.1.tar.gz#md5
=09401c787d9b2660b0d76c4c6baec0f3
Processing lxml-3.3.1.tar.gz
Writing c:\users\user\appdata\local\temp\easy_install-aik18s\lxml-3.3.1\setup.cfg
Running lxml-3.3.1\setup.py -q bdist_egg --dist-dir c:\users\user\appdata\local
temp\easy_install-aik18s\lxml-3.3.1\egg-dist-tmp-oly1w6
Building lxml version 3.3.1.
Building without Cython.
ERROR: b"'xslt-config' is not recognized as an internal or external command,\r\n
operable program or batch file.\r\n"
** make sure the development packages of libxml2 and libxslt are installed **

Using build configuration of libxslt
C:\Python33\lib\distutils\dist.py:258: UserWarning: Unknown distribution option:
'extra_require'
warnings.warn(msg)
C:\Python33\lib\distutils\dist.py:258: UserWarning: Unknown distribution option:
'bugtrack_url'
warnings.warn(msg)
error: Setup script exited with error: Unable to find vcvarsall.bat

C:\Users\USER\Downloads\pyshark-master\src>

LiveCapture hangs on linux with timeout

On some tshark versions, LiveCapture simply hangs when attempted to run with timeout.

Non-Linear response for sequential iteration of large capture

Thanks for the great tool!

I'm attempting to use pyshark to iterate large capture files (180K packets per file) and see non-linear response from iteration. Is there a way to tune pyshark to behave linearly in such circumstances? At around 3K packets is starts getting really slow.

        cap = pyshark.FileCapture(cap_file, keep_packets=False)
        cap.display_filter = ('stp')
        cap.apply_on_packets(getStpData)

bpduTracker chrconlo$ ./bpduTracker.py
Starting bpduTracker...
Capture File Found: /users/xyz/Desktop/20141202/prog/test.cap

2014-12-11 16:05:00.030286 -> Packet Count: 50
BPDU Accounting Dictionary Size: 0

2014-12-11 16:05:00.402444 -> Packet Count: 100
BPDU Accounting Dictionary Size: 0

2014-12-11 16:05:00.996808 -> Packet Count: 150
BPDU Accounting Dictionary Size: 0

2014-12-11 16:05:01.793828 -> Packet Count: 200
BPDU Accounting Dictionary Size: 0

2014-12-11 16:05:02.814992 -> Packet Count: 250
BPDU Accounting Dictionary Size: 0

2014-12-11 16:05:04.133944 -> Packet Count: 300
BPDU Accounting Dictionary Size: 0

2014-12-11 16:05:05.571902 -> Packet Count: 350
BPDU Accounting Dictionary Size: 0

2014-12-11 16:05:07.212769 -> Packet Count: 400
BPDU Accounting Dictionary Size: 0

2014-12-11 16:05:09.178611 -> Packet Count: 450
BPDU Accounting Dictionary Size: 0

2014-12-11 16:05:11.315323 -> Packet Count: 500
BPDU Accounting Dictionary Size: 0

2014-12-11 16:05:13.638662 -> Packet Count: 550
BPDU Accounting Dictionary Size: 0

pyshark error on print (I think this might be my python setup in Cygwin)

Hi I am trying to use pyshark in Cygwin on Windows 7. I know it works ok, as a colleage of mine has it set up. However when I try to use pyshark I am getting the following error:

print cap[0]
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/site-packages/pyshark-0.3.3-py2.7.egg/pyshark/capture/file_capture.py", line 58, in getitem
self.next()
File "/usr/lib/python2.7/site-packages/pyshark-0.3.3-py2.7.egg/pyshark/capture/file_capture.py", line 48, in next
packet = self._packet_generator.send(None)
File "/usr/lib/python2.7/site-packages/pyshark-0.3.3-py2.7.egg/pyshark/capture/capture.py", line 159, in _packets_from_tshark_sync
tshark_process = existing_process or self.eventloop.run_until_complete(self._get_tshark_process())
File "/usr/lib/python2.7/site-packages/trollius-1.0.4-py2.7.egg/trollius/base_events.py", line 300, in run_until_complete
return future.result()
File "/usr/lib/python2.7/site-packages/trollius-1.0.4-py2.7.egg/trollius/futures.py", line 287, in result
raise self._exception
OSError: [Errno 2] No such file or directory

I think maybe my Python installation on cygwin is messed up or the package trollius is. Has anybody else experienced this before?

missing release tags / files

Hey,

For packaging purpose it would be awesome if you can create git tags for all versions (like the missing 0.3.3 for example). Currently there is only a release for 0.3 (but not the most current one).
Pushing git tags will make github store a tarball that can be downloaded to build the package.
It would be awesome, as the pypi package does not contain the tests and in the packaging step it would be cool to also include your tests 😄

PS: when do you plan to bump to 0.3.4? 😋

cheers
anthraxx

installing from pip or easy_install error

installing in Debian using easy_install or pip causes error, ver. 0.2.1 (ver .2.2 not available in pip):
error: can't copy 'pyshark/config.ini': doesn't exist or not a regular file

the solution is modify in setup.py config.ini to config.py
just a annoying typo.

Build error

building 'lxml.etree' extension

creating build/temp.linux-x86_64-2.7

creating build/temp.linux-x86_64-2.7/src

creating build/temp.linux-x86_64-2.7/src/lxml

x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/local/include -I/usr/include/libxml2 -I/tmp/pip_build_root/lxml/src/lxml/includes -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w

x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -L/usr/local/lib -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-2.7/lxml/etree.so

/usr/bin/ld: cannot find -lz

collect2: error: ld returned 1 exit status

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

How to get the HTTP header by using pyshark?

I am new to this project, and I have already edit the code to let it run on the python3.4 with win7 64bit.
I can get the data in ETH, IP and TCP, but is it possible to get the information in HTTP header by using this?
Or I can only get the data in transport layer by using pyshark?

Thank you so much!

---

I found that I can get the HTTP header by adding the display_filter.
But when there comes a field name called "Expert Info (Chat/Sequence)", how can I get the value of it?
For example:
I can get the host by print (cap[i].http.host).
What should I type when there is a blank or something else in the field name?

Thanks!

Saving packets to a pcap / pcap-ng file.

Is there a way to save a selection of packets to a file?
I can't find a way to do it. I was going to try to make a class based on LiveCapture that has the ability to save the list of self._packets into a file, but I got confused on the making and quit.

where is param bpf_filter?

I can't find param bpf_filter in init function of FileCapture?
So I got a "TypeError: init() got an unexpected keyword argument 'bpf_filter' " error.

Double VLAN detection

How can you get a list of vlans if you have more than one, for example if i have this packet..

Layer ETH:
Destination: 00:99:88:77:66:55 (00:99:88:77:66:55)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Address: 00:99:88:77:66:55 (00:99:88:77:66:55)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Type: 802.1Q Virtual LAN (0x8100)
Source: 00:11:22:33:44:55 (00:11:22:33:44:55)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Address: 00:11:22:33:44:55 (00:11:22:33:44:55)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
Layer VLAN:
...0 .... .... .... = CFI: Canonical (0)
000. .... .... .... = Priority: Best Effort (default) (0)
.... 0000 1100 1000 = ID: 200
Type: 802.1Q Virtual LAN (0x8100)
Layer VLAN:
Trailer: 8dbdde29
...0 .... .... .... = CFI: Canonical (0)
000. .... .... .... = Priority: Best Effort (default) (0)
.... 0000 0110 0100 = ID: 100
Type: IP (0x0800)
Layer IP:
....
Layer UDP:
....

if I use pyshark i got only inner vlan.

print cap[0]['vlan'].id
100

I expect to get it the same as tshark :

tshark -r filename.pcap -T fields -e vlan.id
100,200

objectifying XML pkt of multiple "field" tags with the same "name" attribute

Hi,

https://github.com/KimiNewt/pyshark/blob/master/src/pyshark/packet/layer.py#L43

for field in xml_obj.findall('.//field'):
            self._all_fields[field.attrib['name']] = LayerField(**dict(field.attrib))

I was wondering whats the proper parsing/objectifying of XML packet of multiple "field" tags with the same attribute "name"?
The issue occurs mostly in the highest decoded layer of the packet e.g. DATA-LINES-TEXT, where multiple "field" tags have an empty "name" attribute.

If I may recommend something, (without breaking backward compatibility), adding a prefix to duplicated names e.g. "name_attribute", "name_attribute_1", "name_attribute_2" ..etc

PS
thanks for such a great project

memory error with big capture

If I try to read a big Wireshark capture file with over 88000 captures, pyshark crashes after several thousands of packets (the number varies, even with the same capture file)
with a MemoryError:

File "C:\Python27\lib\site-packages\pyshark\capture\capture.py", line 167, in _packets_from_tshark_sync
self._get_packet_from_stream(tshark_process.stdout, data, psml_structure=psml_structure))
File "C:\Python27\lib\site-packages\trollius\base_events.py", line 300, in run_until_complete
return future.result()
File "C:\Python27\lib\site-packages\trollius\futures.py", line 287, in result
raise self._exception
MemoryError

I shrunk the Python program itself to a basic loop:

cap=pyshark.FileCapture(capture+".pcapng",keep_packets=False)
p=0
start=time.time()
for packet in cap:
p+=1
if p%500 ==0:
print p,time.time()-start
start=time.time()

The output also shows that the program slows down while working its way through the packets
(indication of a memory leak?):
500 2.42000007629
1000 3.33999991417
1500 5.38499999046
2000 7.25200009346
2500 9.28200006485
3000 11.371999979
3500 13.3539998531
4000 15.4560000896
4500 17.6549999714
5000 19.7650001049
5500 21.7889997959
6000 24.0099999905
6500 26.1820001602
7000 28.4779999256
7500 30.6789999008
8000 32.9380002022

load `tshark` executable returned by `which tshark`

On my system I have two versions of tshark. The one I get from typing tshark in bash is not the one that pyshark is loading because of the difference between the search order and my path environment variable order.

I don't suspect it's a common problem, but in my case it would be more intuitive to search for tshark first in the path and then in the default locations.

install error on windows

when I use pip install pyshark on windows,I met the error like that

overlapped.c:1410: error: 'SO_UPDATE_CONNECT_CONTEXT' undeclared (first use in this functi
on)
error: Setup script exited with error: command 'c:\\mingw\\bin\\gcc.exe' failed with exit
status 1

the same error came across when I download the source to use python setup.py install.

Get raw binary frame

Hi !
I'm sorry if my question may appear simplist, I may have missed the method.
Is there a nice way to get a raw binary string "010111000" of the received frames from pyshark ?

Thanks in advance !

Getting pyshark packets from queue / pickle problems

Hello,

As I was trying to capture packets live using pyshark and do some multiprocessing, I encountered problems when getting a packet previously put in a multiprocessing.Queue or when I try to un-pickle it. Here is the error I get:

python2.7/site-packages/pyshark/packet/layer.py", line 48, in getattr
val = self.get_field_value(item, raw=self.raw_mode)
(... multiple times ...)
RuntimeError: maximum recursion depth exceeded while calling a Python object`.

This error comes whatever the sys.recursionlimit is set to (I stopped trying after raising it by a ten-factor).
I noted that this issue does not happen when using Queue.Queue so I suspect this is a problem with "reconstructing" the object when it's unpickled (as multiprocessing.Queue uses some kind of pickler, while Queue.Queue is a collections.deque under the hood).

Is this a "normal" behavior and pyshark's packets cannot be pickled and un-pickled or is it a real issue ?

Here is the code I use to reproduce that issue
I am running a standard 2.7.9 python on RedHat 6.5

import pyshark
import multiprocessing
import Queue
import cPickle as pickle

# Capture on eth0
interface = pyshark.LiveCapture(interface="eth0")

def queue_test(queue):
    """ Puts captured packets in a queue, then un-queue them and display """
    for packet in interface.sniff_continuously(packet_count=5):
        queue.put(packet)

    while not queue.empty():
        packet = queue.get()
        print "Packet {} {}".format(packet.highest_layer,packet._packet_string)


def pickle_test():
    """ Immediately pickle and unpickle the packet to display it"""
    for packet in interface.sniff_continuously(packet_count=5):
        pickled_packet = pickle.loads(pickle.dumps(packet, pickle.HIGHEST_PROTOCOL))
        print "Packet #{}, {} {}".format(pickled_packet.highest_layer,pickled_packet._packet_string)

if __name__ == "__main__":
    # Try two types of queues
    normal_queue = Queue.Queue()
    process_queue = multiprocessing.Queue()

    # Runs fine
    #queue_test(normal_queue)

    # Both crash with a RuntimeError
    queue_test(process_queue)
    #pickle_test()

For cross-references, I had a question posted on StackOverflow about this problem, but I am logging this issue here because it didn't get much attention.

Thanks you

output of `tshark -D` needs to be parsed differently

In my version of tshark (1.12.3), the output of the command may have changed.

In [2]: pyshark.tshark.tshark.get_tshark_interfaces()
Out[2]: [u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'10', u'11']

In [3]: exit()
~/nidaq$ tshark -D
1. wlan0
2. vmnet1
3. vmnet8
4. any
5. lo (Loopback)
6. nflog
7. nfqueue
8. dbus-system
9. dbus-session
10. usbmon1
11. usbmon2