mitrecnd / whodat Goto Github PK
View Code? Open in Web Editor NEWPivotable Reverse WhoIs / PDNS Fusion with Registrant Tracking & Alerting plus API for automated queries (JSON/CSV/TXT)
Pivotable Reverse WhoIs / PDNS Fusion with Registrant Tracking & Alerting plus API for automated queries (JSON/CSV/TXT)
I'm running this in a vagrant Ubuntu 16.04. VBox is port forwarding 8080 to 80, and here are the errors I see:
10.0.2.2 - - [09/Feb/2017:20:16:17 +0000] "GET / HTTP/1.1" 500 528
[Thu Feb 09 20:16:17.667820 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] mod_wsgi (pid=6): Target WSGI script '/opt/WhoDat/pydat/pydat/wsgi.py' cannot be loaded as Python module.
[Thu Feb 09 20:16:17.667847 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] mod_wsgi (pid=6): Exception occurred processing WSGI script '/opt/WhoDat/pydat/pydat/wsgi.py'.
[Thu Feb 09 20:16:17.667868 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] Traceback (most recent call last):
[Thu Feb 09 20:16:17.667889 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] File "/opt/WhoDat/pydat/pydat/wsgi.py", line 31, in <module>
[Thu Feb 09 20:16:17.667920 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] application = get_wsgi_application()
[Thu Feb 09 20:16:17.667930 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] File "/usr/local/lib/python2.7/dist-packages/django/core/wsgi.py", line 13, in get_wsgi_application
[Thu Feb 09 20:16:17.667958 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] django.setup()
[Thu Feb 09 20:16:17.667967 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] File "/usr/local/lib/python2.7/dist-packages/django/__init__.py", line 17, in setup
[Thu Feb 09 20:16:17.667988 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
[Thu Feb 09 20:16:17.667997 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 55, in __getattr__
[Thu Feb 09 20:16:17.668019 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] self._setup(name)
[Thu Feb 09 20:16:17.668027 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 43, in _setup
[Thu Feb 09 20:16:17.668047 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] self._wrapped = Settings(settings_module)
[Thu Feb 09 20:16:17.668055 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 99, in __init__
[Thu Feb 09 20:16:17.668075 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] mod = importlib.import_module(self.SETTINGS_MODULE)
[Thu Feb 09 20:16:17.668083 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
[Thu Feb 09 20:16:17.668106 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] __import__(name)
[Thu Feb 09 20:16:17.668114 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] File "/opt/WhoDat/pydat/pydat/settings.py", line 233, in <module>
[Thu Feb 09 20:16:17.668133 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] execfile(csfile)
[Thu Feb 09 20:16:17.668152 2017] [wsgi:error] [pid 6:tid 140672902043392] [remote 10.0.2.2:33022] IOError: [Errno 21] Is a directory: '/opt/WhoDat/pydat/pydat/custom_settings.py'
It would be awesome to have a way to express queries using a NOT filter in pyDat.
For example:
email:@foo.com and not email:"[email protected]"
Thanks for the useful project.
I recently purchased whoisxmlapi data and am looking into pyDat for running my local API on top of the raw data. I was wondering whether I should start using the current release (master branch) or if one of the other branches with support for newer ES versions is production-ready (or close enough).
Thanks in advance. (If there's a better way to communicate about general questions about the project other than an Issue, just let me know.)
just got around to playing with the API and testing using curl and noticed an error in the apache error.log. querying via the pydat interface works fine. the error in the log:
==> error.log <==
[Sat Mar 26 19:45:44.512042 2016] [:error] [pid 8206:tid 139663402592000] {'sort': [{'dataVersion': {'order': 'asc'}}], 'query': {'filtered': {'filter': {'and': [{'term': {'domainName': u'google.com'}}, {'term': {'dataVersion': 7}}]}, 'query': {'match_all': {}}}}, 'size': 50000}
the curl command: curl http://localhost/ajax/domain/google.com/latest/
the output of the curl command is;
{"avail": 0, "total": 0, "data": [], "success": true}
Hi guys, please contact me at [email protected], I would like to discuss collaboration on improving this tool.
thanks!
i have successfully done an import using elasticsearch (started with v6), however when trying to import another version (doesnt matter which), the import fails with the following message:
Exception in thread Worker 0:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File "./elasticsearch_populate.py", line 167, in process_worker
process_entry(insert_queue, es, work['header'], work['row'], options)
File "./elasticsearch_populate.py", line 246, in process_entry
for ch in changed:
UnboundLocalError: local variable 'changed' referenced before assignment
ive tried importing several different versions (v7, v8, v9, v4, v2..) and get the same error each time
Hi all,
As far as I can understand, for pDNS information only DNSDB can be used!
It would be a good idea to use PassiveTotal as well (or as an alternative compared to Farsight's commercial solution)?
https://www.passivetotal.org/
Cheers,
Andreas
In pydat/pydat/scripts/mongo.py you're using pymongo.Connection to connect to MongoDB. After pymongo 3.0.2 it has been removed entirely.
elasticsearch_populate always adds files without extension to queue to be processed (here) when using the -d
flag. CSV exceptions are handled gracefully, but we could avoid adding files that don't match the extension by changing conditional to:
if ext[1:] != self.options.extension:
after the recent round of updates to the elasticsearch_import; im getting an error:
Errors making bulk api request!!
Bulk Requests saved to disk (/tmp/pydat-bulk--.txt) and should be submitted manually!!
It is possible you are running too many bulk workers or the bulk size is too big!
ElasticSearch Bulk Syntax (using curl):
curl -s -XPOST <es_server:port>/_bulk --data-binary @
Bulk API error -- forcing program shutdown
Cleaning Up ... Please Wait ...
Warning!! Forcefully killing this might leave Elasticsearch in an inconsistent state!
Shutting down input reader threads ...
Stopping workers ...
Waiting for ElasticSearch bulk uploads to finish ...
Finalizing metadata
Finalizing settings
... Done
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed
send(obj)
IOError: [Errno 32] Broken pipe
when i try to import manually with the curl command above:
{"took":270,"errors":true,"items":[{"create":{"_index":"whois-3","_type":"com","_id":"emctools","status":500,"error":"IllegalArgumentException[Document contains at least one immense term in field="details.registrant_email" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[97, 114, 109, 105, 110, 46, 108, 101, 110, 107, 64, 102, 114, 101, 101, 110, 101, 116, 46, 100, 101, 32, 116, 101, 99, 104, 95, 32, 97, 114]...', original message: bytes can be at most 32766 in length; got 39327]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 39327]; "}}]}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.