mitrecnd/whodat

Pivotable Reverse WhoIs / PDNS Fusion with Registrant Tracking & Alerting plus API for automated queries (JSON/CSV/TXT)

whodat's Issues

500 Internal Server Error

I'm running this in a vagrant Ubuntu 16.04. VBox is port forwarding 8080 to 80, and here are the errors I see:

10.0.2.2 - - [09/Feb/2017:20:16:17 +0000] "GET / HTTP/1.1" 500 528
[Thu Feb 09 20:16:17.667820 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] mod_wsgi (pid=6): Target WSGI script '/opt/WhoDat/pydat/pydat/wsgi.py' cannot be loaded as Python module.
[Thu Feb 09 20:16:17.667847 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] mod_wsgi (pid=6): Exception occurred processing WSGI script '/opt/WhoDat/pydat/pydat/wsgi.py'.
[Thu Feb 09 20:16:17.667868 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] Traceback (most recent call last):
[Thu Feb 09 20:16:17.667889 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]   File "/opt/WhoDat/pydat/pydat/wsgi.py", line 31, in <module>
[Thu Feb 09 20:16:17.667920 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]     application = get_wsgi_application()
[Thu Feb 09 20:16:17.667930 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]   File "/usr/local/lib/python2.7/dist-packages/django/core/wsgi.py", line 13, in get_wsgi_application
[Thu Feb 09 20:16:17.667958 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]     django.setup()
[Thu Feb 09 20:16:17.667967 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]   File "/usr/local/lib/python2.7/dist-packages/django/__init__.py", line 17, in setup
[Thu Feb 09 20:16:17.667988 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]     configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
[Thu Feb 09 20:16:17.667997 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 55, in __getattr__
[Thu Feb 09 20:16:17.668019 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]     self._setup(name)
[Thu Feb 09 20:16:17.668027 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 43, in _setup
[Thu Feb 09 20:16:17.668047 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]     self._wrapped = Settings(settings_module)
[Thu Feb 09 20:16:17.668055 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 99, in __init__
[Thu Feb 09 20:16:17.668075 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]     mod = importlib.import_module(self.SETTINGS_MODULE)
[Thu Feb 09 20:16:17.668083 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]   File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
[Thu Feb 09 20:16:17.668106 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]     __import__(name)
[Thu Feb 09 20:16:17.668114 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]   File "/opt/WhoDat/pydat/pydat/settings.py", line 233, in <module>
[Thu Feb 09 20:16:17.668133 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022]     execfile(csfile)
[Thu Feb 09 20:16:17.668152 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] IOError: [Errno 21] Is a directory: '/opt/WhoDat/pydat/pydat/custom_settings.py'

Support NOT filter in ES

It would be awesome to have a way to express queries using a NOT filter in pyDat.

For example:

email:@foo.com and not email:"[email protected]"

pyDat 4.0 status - what branch to use today

Thanks for the useful project.

I recently purchased whoisxmlapi data and am looking into pyDat for running my local API on top of the raw data. I was wondering whether I should start using the current release (master branch) or if one of the other branches with support for newer ES versions is production-ready (or close enough).

Thanks in advance. (If there's a better way to communicate about general questions about the project other than an Issue, just let me know.)

apache error

just got around to playing with the API and testing using curl and noticed an error in the apache error.log. querying via the pydat interface works fine. the error in the log:

==> error.log <==
[Sat Mar 26 19:45:44.512042 2016] [:error] [pid 8206:tid 139663402592000] {'sort': [{'dataVersion': {'order': 'asc'}}], 'query': {'filtered': {'filter': {'and': [{'term': {'domainName': u'google.com'}}, {'term': {'dataVersion': 7}}]}, 'query': {'match_all': {}}}}, 'size': 50000}

the curl command: curl http://localhost/ajax/domain/google.com/latest/

the output of the curl command is;

{"avail": 0, "total": 0, "data": [], "success": true}

whoisxmlapi database

Hi guys, please contact me at [email protected], I would like to discuss collaboration on improving this tool.
thanks!

local variable 'changed' referenced before assignment

i have successfully done an import using elasticsearch (started with v6), however when trying to import another version (doesnt matter which), the import fails with the following message:

Exception in thread Worker 0:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File "./elasticsearch_populate.py", line 167, in process_worker
process_entry(insert_queue, es, work['header'], work['row'], options)
File "./elasticsearch_populate.py", line 246, in process_entry
for ch in changed:
UnboundLocalError: local variable 'changed' referenced before assignment

ive tried importing several different versions (v7, v8, v9, v4, v2..) and get the same error each time

PassiveTotal usage

Hi all,

As far as I can understand, for pDNS information only DNSDB can be used!
It would be a good idea to use PassiveTotal as well (or as an alternative compared to Farsight's commercial solution)?
https://www.passivetotal.org/

Cheers,
Andreas

Pymongo Connection deprecated

In pydat/pydat/scripts/mongo.py you're using pymongo.Connection to connect to MongoDB. After pymongo 3.0.2 it has been removed entirely.

Unexpected behavior for files with no extension

elasticsearch_populate always adds files without extension to queue to be processed (here) when using the -d flag. CSV exceptions are handled gracefully, but we could avoid adding files that don't match the extension by changing conditional to:

if ext[1:] != self.options.extension:

importing error

after the recent round of updates to the elasticsearch_import; im getting an error:

Errors making bulk api request!!
Bulk Requests saved to disk (/tmp/pydat-bulk--.txt) and should be submitted manually!!
It is possible you are running too many bulk workers or the bulk size is too big!
ElasticSearch Bulk Syntax (using curl):
curl -s -XPOST <es_server:port>/_bulk --data-binary @
Bulk API error -- forcing program shutdown
Cleaning Up ... Please Wait ...
Warning!! Forcefully killing this might leave Elasticsearch in an inconsistent state!
Shutting down input reader threads ...
Stopping workers ...
Waiting for ElasticSearch bulk uploads to finish ...
Finalizing metadata
Finalizing settings
... Done
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed
send(obj)
IOError: [Errno 32] Broken pipe

when i try to import manually with the curl command above:

curl -s -XPOST http://127.0.0.1:9200/_bulk --data-binary @/tmp/pydat-bulk-3-5b37e8e6-0b14-11e6-a9cd-005056a1f827.txt

{"took":270,"errors":true,"items":[{"create":{"_index":"whois-3","_type":"com","_id":"emctools","status":500,"error":"IllegalArgumentException[Document contains at least one immense term in field="details.registrant_email" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[97, 114, 109, 105, 110, 46, 108, 101, 110, 107, 64, 102, 114, 101, 101, 110, 101, 116, 46, 100, 101, 32, 116, 101, 99, 104, 95, 32, 97, 114]...', original message: bytes can be at most 32766 in length; got 39327]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 39327]; "}}]}