dakotanelson/pushpin-web

Monitor geotagged social media from multiple platforms in real time.

Pushpin modules broken out of recon-ng and turned into a webapp. Pushpin-web provides an easy web interface to keep track of geotagged social media activity. Deployed using Docker for ease of use.

A project by Black Hills Information Security.

Quickstart:

First, make sure you've got Docker installed: https://docs.docker.com/installation/

Second, make sure you've got Docker Compose installed: https://docs.docker.com/compose/install/

To start, change to the project's root directory and run docker-compose up. This should start all the containers, link them all together, and get everything set up.

Head to localhost:8080 to take a look. Default username/password is 'test'. That's it - you're done!

Run cleanup.sh to remove a few artifacts (logs) left behind from each pushpin deploy - they're RO files, so you can't start up a new deploy until they're gone. Then run docker-compose up to start again.

Acquiring API Keys

• Taken from https://bitbucket.org/LaNMaSteR53/recon-ng/wiki/Usage%20Guide#!acquiring-api-keys
• Google API Key (used for Youtube) - Go to the Google Developer's Console and create a project. Once it's created, under "APIs & auth" on the left bar, select "APIs" and then search for Youtube and click "enable." Then, under "APIs & auth" on the left bar, select "Credentials" and then under "Public API access" select "Create new key."
• Picasa - None required.
• Flickr API Key (flickr_api) - TBD
• Instagram API Key (instagram_api) - Log in here with an existing Instagram account and register a new application. Add http://localhost:31337 as the "OAuth redirect_uri". Click "Manage Clients" at the top of the screen and the API key will be available as the "CLIENT ID".
• Instagram Secret (instagram_secret) - Log in here. Click "Manage Clients" at the top of the screen and the Secret key will be available as the "CLIENT SECRET".
• Twitter Consumer Key (twitter_api) - Create an application https://dev.twitter.com/apps. The Consumer key will be available on the application management page.
• Twitter Consumer Secret (twitter_secret) - The Consumer secret will be available on the application management page for the application created above.
• Shodan API Key (shodan_api) - Create an account or sign in to Shodan using one of the many options available http://www.shodanhq.com/. The API key will be available on the right side of the screen. An upgraded account is required to access advanced search features.

FAQ:

Q: I'm trying to deploy this to somewhere other than localhost, and it's super broken. Help?

Check out the section below on deploying places other than localhost. There are a few special things you have to do.

Q: How do I use this thing?

Once you've run start.sh, go to localhost:8080, where you will be prompted to log in. Enter test and test, unless you configured the PUSHPIN_PASSWORD environment variable to be something else. Either way, username is test. Once you've logged in, you'll be redirected to the admin page, where you can add API keys. You should do so, or else you won't get any data. Once that's done, head back to localhost:8000 and you'll see the pushpin interface. Have fun!

Q: When does data get pulled?

Data is pulled periodically (depends on the module - Flickr is every 15 minutes, for example) by celerybeat and manually every time a new location is added. If you don't see any data for a location you just added, try refreshing - it sometimes takes a minute or two to pull data from the various APIs. Ideallly, the interface will eventually allow you to monitor the progress of background API pulls.

Q: I'm getting an error IOError: [Errno 13] Permission denied: '/some/path/to/pushpin-web/logs/error.log'. What do?

Run cleanup.sh to delete the logs left over from a previous deploy. The logs are currently read-only, and so cannot be overwritten by the new deploy. Hopefully this will have a cleaner solution soon.

Q: Can I run this in the background?

Yep: docker-compose up -d will run the cluster of containers in detached mode, and docker-compose stop will stop the detached cluster.

Q: Can I configure more things?

Yep: open up docker-compose.yml and you can fiddle arround with settings and variables. There are some comments there to help you out.

Deploying other than localhost:

Set the ALLOWED_HOSTS environment variable in docker-compose.yml to be the address of your host. (IP address or domain name, or both) See Django's docs for details.

You'll also want to tighten up security a bit - in docker-compose.yml, changing at least the SECRET_KEY and PUSHPIN_PASSWORD variables is a good start.

Rough notes on local usage for development:

• Install dependencies:
  • python >= 3.3.3
  • RabbitMQ: apt-get install rabbitmq-server or yum install rabbitmq-server
  • postgreSQL >= 9.2.8
  • pip install -r requirements.txt
• Create a symlink in pushpin-web/static that points to your-django-path/django/contrib/admin/static/admin so that your new server (next step) will also serve the static files for Django's admin interface.
• In pushpin-web/static, run python -m http.server 8001. This is to serve static files, because doing so with Django will make you feel many terrible feels.
• Run postmaster -D ./db/ to start the PostgreSQL server, where ./db/ is a directory you create to hold the database.
• Run sudo rabbitmq-server (with an optional -detached) to run the RabbitMQ message queue
• Set some environment variables: POSTGRES_PASSWORD, POSTGRES_USER, POSTGRES_HOST, PUSHPIN_PASSWORD, and DEBUG. Take a look at pushpin-app/pushpin/settings.py for more configuration options.
• cd to pushpin-web/pushpin-app and run python manage.py migrate to set up the database
• Run python manage.py runserver to start the django server
• Run python manage.py celery worker --loglevel=info --concurrency=2 to start the celery server
• Run python manage.py celery beat to run celery's scheduling engine
• Run pushpin-app/deploy.py to set up some users and a place to put you API keys.

To add a module:

• Create the actual module in pushpin-app/modules. Take a look at some other modules in that folder to get an idea of how to make it work.
• In pushpin-app/map/tasks.py, add a task for your module by copying another module and modifying it
• In pushpin-app/map/management/commands/getdata.py, add your task to the list that needs to be run
• Rebuild the Docker container (or restart celery and celery beat if not using Docker)

Wishlist:

• Capable of handling multiple users/authentication.
• Ability to narrow displayed pins by date.
• Break down Celery tasks to the point of getting a single location's data from a single API, use a lock to make sure only one worker pulls from an API at a time. This would help with scaling someday.
• Intelligently deal with rate limiting, rather than just throttling API calls to safe levels without ever checking. (Could help with previous wishlist item.)
• Use RabbitMQ as a backend to have Celery keep task status updated, show user progress of data-pulling tasks.
• Be able to manually trigger a data pull from the webapp. (Especially useful if only one location is triggered.)
• Linkify links in pushpins. The linkify library is already in place, it just needs to be used.

pushpin-web's Issues

Remove unused keys

Remove Google, Facebook, and LinkedIn due to not being used.

Docker-Compose .yml error

"docker-compose up" doesn't recognise the DEBUG value "False" in the environment and celery parameters section in the .yml config for compose

In this file ; https://github.com/DakotaNelson/pushpin-web/blob/master/docker-compose.yml

I changed the 'False' value to 'null' and seems to be working fine. Not sure if this is a change in docker compose. but thought id mention it.

Thanks,

@Tu5k4rr

Loading spinner for media page

The media page loads very slowly (which is its own issue) and could use some sort of indicator that it isn't just frozen.

Two locations with the same name = cannot view either location

The location name is used to look up the correct database entry. Duplicate names cause the database get() call to return two things and therefore die.

Solution: enforce uniqueness in location names. Nobody should need two locations with the same name anyway, that just makes the list confusing and less useful for everyone.

Add separate 'response message' field in facebox overlay

Right now, response messages from the server are just appended directly to the main box of the overlay. That's pretty nasty, and should be replaced with a container (span/div/whatever) that holds these response messages.

Unable to restart pushpin container without destroying database

When pushpin starts up, it tries to create an admin user (in runscripts/init_scripts/make_user.sh. If the database has lived through a previous pushpin startup, it tries to create another user with the same name, and sadness ensues:

*** Running /etc/my_init.d/make-user.sh...
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
psycopg2.IntegrityError: duplicate key value violates unique constraint "auth_user_username_key"
DETAIL:  Key (username)=(admin) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.4/dist-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.4/dist-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.4/dist-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/local/lib/python3.4/dist-packages/django/contrib/auth/management/commands/createsuperuser.py", line 55, in execute
    return super(Command, self).execute(*args, **options)
  File "/usr/local/lib/python3.4/dist-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.4/dist-packages/django/contrib/auth/management/commands/createsuperuser.py", line 160, in handle
    self.UserModel._default_manager.db_manager(database).create_superuser(**user_data)
  File "/usr/local/lib/python3.4/dist-packages/django/contrib/auth/models.py", line 187, in create_superuser
    **extra_fields)
  File "/usr/local/lib/python3.4/dist-packages/django/contrib/auth/models.py", line 178, in _create_user
    user.save(using=self._db)
  File "/usr/local/lib/python3.4/dist-packages/django/db/models/base.py", line 591, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python3.4/dist-packages/django/db/models/base.py", line 619, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/usr/local/lib/python3.4/dist-packages/django/db/models/base.py", line 700, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/usr/local/lib/python3.4/dist-packages/django/db/models/base.py", line 733, in _do_insert
    using=using, raw=raw)
  File "/usr/local/lib/python3.4/dist-packages/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/django/db/models/query.py", line 921, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/python3.4/dist-packages/django/db/models/sql/compiler.py", line 920, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.4/dist-packages/django/db/backends/utils.py", line 81, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/usr/local/lib/python3.4/dist-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.4/dist-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python3.4/dist-packages/django/utils/six.py", line 549, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.4/dist-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "auth_user_username_key"
DETAIL:  Key (username)=(admin) already exists.

*** /etc/my_init.d/make-user.sh failed with status 1

This means that in order to restart the pushpin Docker container, you have to completely reset the database, which currently involves destroying all old data.

Naive Datetime Error on Location Add

When new locations are added, a warning is logged:

/usr/local/lib/python3.4/dist-packages/django/db/models/fields/__init__.py:1278: RuntimeWarning: DateTimeField Location.latest_data received a naive datetime (2015-03-26 10:47:23.673572) while time zone support is active.
  RuntimeWarning)

I don't think it actually harms anything, but it would be nice to not log that error anymore.

Use an actual server

Static files are being served by Python's http.server, and Django is served by its own built-in server. Both should really use an actual server.

Firefox Issues

Chrome is currently the recommended browser for pushpin-web.

I could not reproduce this issue because I was always redirected to a JSON response page when I submitted [the new location form].

I had a couple other issues. The date column always said "Invalid Date" or something like that. And deleting a pin did not work.

These are confirmed bugs in Firefox and not Chrome. I'm not sure why; a fix will happen eventually.

Pins stack

Often, a bunch of pins have exactly the same geolocation, and thus stack with no indication that it's occurring.

Thought: add a slight randomness to pins to space them just a tad apart?

Add Instagram module

Pretty much what it says. The hangup right now is that Instagram seems to use some sort of weird OAuth flow where the user has to manually click a popup; since Docker hates everything, this might be a bit tricky.

Trying to get in touch regarding a security issue

Hey there!

I'd like to report a security issue but cannot find contact instructions on your repository.

If not a hassle, might you kindly add a SECURITY.md file with an email, or another contact method? GitHub recommends this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.

Thank you for your consideration, and I look forward to hearing from you!

(cc @huntr-helper)

Errors in API pulls cause gaps in coverage

Right now, if there's an error and an API doesn't successfully pull data, it still updates its "last pulled" value, which means that window will never be pulled from again (more or less).

I'm thinking some sort of error flag should be set if there's an error in pulling, then checked at the end and acted upon.

Show celery errors in web UI

Currently, if something doesn't work with one of the APIs (i.e. just recently I mis-typed an API key, so Twitter didn't pull any data and I had no idea why), there's no way of knowing why a certain type of data isn't showing up without digging into logs. It'd be nice to have that sort of error reporting in the web interface.

Right click on map to add location

Having to type lat/long pairs is a pain.

Shodan requests occasionally time out

Requests from the Shodan module occasionally return with the following:

[2014-11-30 17:50:17,149: ERROR/MainProcess] Task celery.local.shodanTask[ecfeb5d1-fd61-4022-b4fd-42d4fded7c48] raised unexpected: ModuleException('Request to https://api.shodan.io/shodan/host/search returned with error 500.\n Response body: {"error": "The search request has timed out or your query was invalid."}',)
Traceback (most recent call last):
  File "/home/dnelson/.pyenv/versions/3.3.3/lib/python3.3/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/dnelson/.pyenv/versions/3.3.3/lib/python3.3/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/dnelson/.pyenv/versions/3.3.3/lib/python3.3/site-packages/celery/local.py", line 167, in <lambda>
    __call__ = lambda x, *a, **kw: x._get_current_object()(*a, **kw)
  File "/home/dnelson/.pyenv/versions/3.3.3/lib/python3.3/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return orig(self, *args, **kwargs)
  File "/home/dnelson/.pyenv/versions/3.3.3/lib/python3.3/site-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/projects/pushpin-web/pushpin-app/map/tasks.py", line 68, in shodanTask
    module.run(location.name,location.latitude,location.longitude,location.radius)
  File "/projects/pushpin-web/pushpin-app/modules/shodan.py", line 19, in run
    results = self.search_shodan_api(query, limit)
  File "/projects/pushpin-web/pushpin-app/modules/module.py", line 142, in search_shodan_api
    resp = self.request(url, content=payload)
  File "/projects/pushpin-web/pushpin-app/modules/module.py", line 87, in request
    raise ModuleException("Request to " + url + " returned with error " + str(r.status_code) + ".\n Response body: " + r.text)
modules.module.ModuleException: Request to https://api.shodan.io/shodan/host/search returned with error 500.
 Response body: {"error": "The search request has timed out or your query was invalid."}

Which is almost certainly caused by the request timing out, since this was an issue in recon-ng. Since celery is set up how it is, one Shodan request timing out causes the rest to be canceled, which is unfortunate.