boveus/wifinder

An application to capture and process probe request packets for future analysis.

About

This is a project to use Wireshark to capture packets and view some analysis about them in an html page. There is currently a simple JS front end using data tables and a Rack server and will eventually have google calendar integration.

The current capabilities of this app include the following:

• Ingest data into SQL from an exported Wireshark packet capture (CSV format is currently the only format supported)
• Ingest data and add it to a SQL database on the fly using a rake task that runs tshark
• View and search a list of all of the devices in a datatables view:
• View and search a list of all of the SSIDs in a datatables view:
• View details about an individual device (Including a Google charts view of hours it has been detected):
• View details about a particular SSID:

The SSID and Devices views refer one another, so you can view details about the SSIDs associated with a particular device and vice versa.

For the time being this is intended to passively monitor traffic rather than use packet injection to generate additional traffic. I am also challenging myself to limit the number of dependencies as much as possible.

Pre-requistite Hardware

• A network interface card capable of monitor mode

Pre-requistite Software

• Some version of linux/unix. (Monitoring mode for WiFi NICs is not well supported in OSX/Windows)
• I tested this in Ubuntu 16.04
• Wireshark (Available at https://www.wireshark.org/)

Dependencies

Ruby

• sqlite3 gem
• minitest gem

External

• Datatables
• Sqlite3
• Wireshark

Big shout out to this site: http://youmightnotneedjquery.com/

Getting Started

Identify the wireless interface you want to set to monitor mode using ip a or ifconfig. Set the device to monitor mode using the following commands

`sudo ip link set <interface> down`
`sudo iwconfig <interface> mode monitor`
`sudo ip link set <interface> up`

ifconfig is a deprecated tool, but it is still available in some linux distros and does the same thing:

sudo ifconfig <interface> down
sudo iwconfig <interface> mode monitor
sudo ifconfig <interface> up

Alternatively, there is a rake task built into the project to do so

interface=wlan1 rake set_to_monitor

Start capturing

To display only probe requests, use the following wireshark display filter:

wlan.fc.type_subtype == 0x08

To capture only probe requests, you can use the following capture filter:

subtype probereq

Start capturing on the device (above) that you had set to monitor mode. Run it for as long as you want, but I recommend 3-6 hours or so at a minimum to get a large amount of data.

Export a CSV to import into the application

To generate a CSV to ingest into the application, use Wireshark's built in functionality to save the packets as a CSV:

Once you have gathered the CSV, you can ingest them into the application by doing the following:

• Clone this repo
• In the root directory create a data folder, and put one or more CSVs with Wireshark packet capture data (following the instructions above) in it
• If you don't have them installed already, install ruby and bundler
• In the root directory, run bundle install to install the dependencies of the project
• Run rake setup to get the database created.
• To populate the DB, run rake csv_ingest. Depending on how many/how large the files are this can take a little bit of time
• Alternatively, you can run rake monitor interface=wlan1(or your interface) to capture packets and ingest them into the DB on the fly. If your device is not in monitoring mode, see above.

To view the data

• Run rake serve - This will launch a rack server on port 9000.
• To view the server, you can go to http://localhost:9000/devices
• This will list all of the devices that have been recognized by the capture. This may take a little while to load if you have a large amount of data. You can then click on a device to view more details about it.

Project Organization

The project organization is influenced by the MVC model. There is a Packet model that acts as an ORM between the database and the developer. There are various services that create the database, parse the CSV (that is generated by Wireshark for now) and render the views through ERB and a few Rack server objects.

There are several directories with various types of objects in them:

• Models: This folder contains the ORM objects (Device, Packet, Ssid, ActiveTime so far)
• Services: These objects do various tasks behind the scenes.
• Templates: These contain ERB files for how to format the views for the index / show pages for the models.

Rake Tasks

The current rake tasks for the application are the following:

rake capture         # start capturing packets (device must be in monitor mode)
rake csv_ingest      # Populate the DB with CSVs from the data directory
rake destroy         # Delete the DB
rake light_setup     # setup the database without ingesting packets
rake serve           # serve the viewer folder
rake set_to_monitor  # Usage: interface=wlan1 rake set_to_monitor | set the device to monitor mode
rake setup           # Setup the database
rake view            # view home page in the default browser

wifinder's Issues

Account for anonymized MAC addresses

It’s quite ironic that any device which anonymizes probe requests actually informs you that they do it — in anonymized devices, the locally addressed bit (second-least significant bit) of the address is set to 1. Therefore, running a simple SQL query lets us know that nearly 25% of devices anonymize MAC addresses (891,131 / 3,570,048 probe requests collected).

Add sample seed data

We need to add a sample packet capture so that folks who don't have the requisite hardware can still see how things work.

Add sample packets through a rake seed command

Create a rake seed command that populates the DB with a set of sample data to allow people to test the UI of the app without having to capture a bunch of packets.

Add a home page

I neglected to add a route and a home page to the rack app for the '/' GET request. We will need to add a simple home page with a link to /devices on it and maybe instructions on the app also on there. Right now, the home directory throws an error because there is no 404 page either.

Address the different behavior of .find for packets

The .find method for packets currently behaves in a different way than the rest of the models since they are created from a hash rather than an array of values. The way the SQLITE3 gem returns data seems to lend itself to creating things from arrays, so I think it would be a good refactor to change the packet behavior to be more in line with the other models.

Extend the ERB generation

I would like to have the ERB template generation behave more dynamically. Right now I am just testing things out, but I would prefer that there is a simple mechanism for a user to render the DB in an HTML view.

Integrate Google Charts for visualization

Use this chart: https://developers.google.com/chart/interactive/docs/gallery/calendar to show what day(s) a device has been active (according to the packets captured).

Use https://developers.google.com/chart/interactive/docs/datesandtimes to show what times of day a device is active on a particular day or over several days when it tends to be active.

Paginate the '/devices' page

We need to add pagination for the /devices route. Right now, it shows thousands of links and is kind of unwieldy. I think DataTables would work well here. https://datatables.net/

Improve Wireshark Packet Exporting

Creating a Lua plugin in wireshark to automatically export data into the DB as Wireshark collects it would be awesome: https://wiki.wireshark.org/Lua

Setup rake option to ingest only devices/ssids

Set up a rake option that ingests only ssids and devices. The packets aren't used directly in the app at this time, so it will probably speed up the processing and make the DB a little bit smaller.

Provide a capability for SSIDs to return associated devices

Create a method for retrieving all of the devices for a particular SSID from the SSID model similar to the existing device method here.

Integrate MAC vendors API

Integrate the API here: https://macvendors.com/api. I am imaging this will involve making a call with the MAC of the 'device' model and storing the response info in the DB. I think this should go in the DB in the 'devices' table.

It might be cool to have a button on the device show page that will initiate the request manually on the user's behalf and then stores the info in the DB.

I would prefer to have as few external dependencies as possible, so instead of Faraday or HTTParty, I think we should use: https://docs.ruby-lang.org/en/2.0.0/Net/HTTP.html

Write tests

I need to write some tests. My test suite is looking very sad.

Add the SSID count to the devices table

I tried to do an SSID count column, which was an N+1 query and crashed my dev VM. We should probably try to store this value in the DB and have it calculated as we ingest the CSVs.

Once this is done, we can add an SSID count to the device index view. I think that would be a good piece of info to have on there.

Characterize Devices based on broadcast frequency

https://hal.inria.fr/file/index/docid/858324/filename/Wi-Fi_Stalking.pdf

The emission frequency has been studied by monitoring a Wi-Fi channel and
computing the arrival time delta, i.e. the difference between the arrival times of
all the received probe request frames. The result for two device type (an Apple
and a Samsung smartphone3
) is presented in on Figure 2. For both devices,
peaks can be observed at a regular interval. For the Apple device the peaks
can be observed around 0, 45, 90 and all other multiple of 45 seconds while for
the Samsung device, peaks are appears at 0, 30, 60 and all other multiple of
30 seconds. This indicates that those devices regularly broadcast probe request
frames and that the typical period is 45 seconds for the Apple device and 30 for
the Samsung device. Those results cannot be extended to all Wi-Fi devices, but
they give an idea on the period at which probe requests are emitted by a typical
Wi-Fi device

Add a progress bar for the data ingest

With the addition of a join table, the data ingest is taking a bit of time. A progress bar that indicates what percentage of the CSVs have been parsed would be good.

See: http://joshorourke.com/2012/08/01/how-to-create-terminal-progress-indicators-in-ruby.html

Modify readme to describe the use case / vulnerability

I need to add a more detailed description of the purpose of this project.

My phone was sending probe request packets despite wifi being turned off. I need to add a description to the readme that details how this is possible: https://hal.inria.fr/hal-01575519v2/document

Previously, I thought that probe requests were transmitted only when wifi is turned on, but it appears that probe requests are transmitted even when WiFi is turned off if the "allow scanning" option is enabled.

Write a SQL query that returns the times and dates a device is active

Write some SQL in the Device model that returns a range of dates and times of day that the device had active network traffic for. This might get kind of complicated, as I am not sure what the time interval between probe requests are. I think doing this on an hourly basis would make sense.

I am imagining a display of which hours of the day we have seen traffic from a device. This could be something as simple as converting the times to their equivalent time in military time and returning the unique values as an "active hours" method.

Eventually, I think a graph of this activity or some other sort of front end display on the device show page would be pretty cool.