feeicn / esd Goto Github PK
View Code? Open in Web Editor NEWEnumeration sub domains(枚举子域名)
Home Page: https://yuque.com/esd
License: GNU General Public License v3.0
Enumeration sub domains(枚举子域名)
Home Page: https://yuque.com/esd
License: GNU General Public License v3.0
Sub domain dict count: 735300
2018-04-11 12:17:05,165 [ESD] [INFO] Generate coroutines...
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/encodings/idna.py", line 165, in encode
raise UnicodeError("label empty or too long")
UnicodeError: label empty or too long
The above exception was the direct cause of the following exception:
参照Sublist3r,支持Baidu、Yahoo、Google、Bing甚至GitHub等
root@kali:~/esd# python3 ESD.py baidu.com
File "ESD.py", line 117
async def query(self, sub):
^
SyntaxError: invalid syntax
目前,uvloop不支持Windows系统和Python 3.5 及其以上版本,这在它源码的setup.py文件中可以看到:
if sys.platform in ('win32', 'cygwin', 'cli'):
raise RuntimeError('uvloop does not support Windows at the moment')
vi = sys.version_info
if vi < (3, 5):
raise RuntimeError('uvloop requires Python 3.5 or greater')
所以,使用Windows的小猿们要运行异步爬虫,就要把uvloop那两行注释掉哦。
安装esd过程中出现 uvloop报错, 有么有更好的解决方法
比如字典枚举、相似相似度对比、HTTPS证书透明度、DNS域传送、搜索引擎子域名,默认全部使用也可选择性使用。
pip install aiodns
pip3 install aiodns
在虚拟环境和真实环境都失败
pip已经更新到最新了
stack overflow的方法也都没解决,请问如何解决
报错如下:
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
error: command 'clang' failed with exit status 1
搜索引擎枚举子域后卡了
域名有泛域名解析,例如域名xxx.com有泛域名,
但同时存在oa.xxx.com,admin.xxx.com指向的ip不是泛域名ip,这种情况如何处理。
程序中是直接判断,如果存在泛域名就不继续执行了,会遗漏上述域名
对泛解析域名枚举时,最大的速度问题不是网络请求耗时,而是进行响应相似度比对。
Python中difflib.SequenceMatcher
有三个字符串相似度比较方法:
real_quick_ratio
(速度4) > quick_ratio
(速度2) > ratio
(速度1)
使用最快的real_quick_ratio
在不本地字符串比对时,速度低于50/s
。
即使网络请求耗时忽略不计,仅对17万子域名进行响应相似度比对就得接近1个小时。
目前看来只能重写一套页面相似度算法。
Python 3.7.2 (default, Feb 12 2019, 08:15:36)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
➜ ~ pip3 install esd
Collecting esd
Using cached https://files.pythonhosted.org/packages/eb/76/3962b04a456d13f4ebb0cac1577e4ef707c2c899ff867f16622093ce99b0/ESD-0.0.21-py3-none-any.whl
Requirement already satisfied: colorlog in /usr/local/lib/python3.7/site-packages (from esd) (4.0.2)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.7/site-packages (from esd) (3.5.4)
Collecting aiodns (from esd)
Using cached https://files.pythonhosted.org/packages/c0/9d/82d910965520ac17956a4b412e36298466de1b87a8fd0ab6dea601cdb8aa/aiodns-1.2.0-py2.py3-none-any.whl
Requirement already satisfied: requests in /usr/local/lib/python3.7/site-packages (from esd) (2.20.1)
Requirement already satisfied: uvloop in /usr/local/lib/python3.7/site-packages (from esd) (0.12.1)
Requirement already satisfied: async-timeout in /usr/local/lib/python3.7/site-packages (from esd) (3.0.1)
Requirement already satisfied: dnspython in /usr/local/lib/python3.7/site-packages (from esd) (1.15.0)
Requirement already satisfied: backoff in /usr/local/lib/python3.7/site-packages (from esd) (1.8.0)
Requirement already satisfied: chardet<4.0,>=2.0 in /usr/local/lib/python3.7/site-packages (from aiohttp->esd) (3.0.3)
Requirement already satisfied: multidict<5.0,>=4.0 in /usr/local/lib/python3.7/site-packages (from aiohttp->esd) (4.5.2)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.7/site-packages (from aiohttp->esd) (1.3.0)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/site-packages (from aiohttp->esd) (18.2.0)
Collecting pycares>=1.0.0 (from aiodns->esd)
Using cached https://files.pythonhosted.org/packages/86/0c/3b662b2629fbde4b81ac36f2735495751fbde74a478ba5fe0c30e4db8f59/pycares-2.4.0.tar.gz
Requirement already satisfied: typing in /usr/local/lib/python3.7/site-packages (from aiodns->esd) (3.6.6)
Requirement already satisfied: idna<2.8,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests->esd) (2.5)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests->esd) (2017.4.17)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests->esd) (1.24.1)
Building wheels for collected packages: pycares
Building wheel for pycares (setup.py) ... error
Complete output from command /usr/local/opt/python/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/xx/gtlvnn65597bczwl0btk0x6w0000gn/T/pip-install-bl17e652/pycares/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/xx/gtlvnn65597bczwl0btk0x6w0000gn/T/pip-wheel-3py3no4o --python-tag cp37:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.14-x86_64-3.7
creating build/lib.macosx-10.14-x86_64-3.7/pycares
copying pycares/errno.py -> build/lib.macosx-10.14-x86_64-3.7/pycares
copying pycares/_version.py -> build/lib.macosx-10.14-x86_64-3.7/pycares
copying pycares/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/pycares
copying pycares/__main__.py -> build/lib.macosx-10.14-x86_64-3.7/pycares
running build_ext
building 'pycares._core' extension
creating build/temp.macosx-10.14-x86_64-3.7
creating build/temp.macosx-10.14-x86_64-3.7/src
creating build/temp.macosx-10.14-x86_64-3.7/deps
creating build/temp.macosx-10.14-x86_64-3.7/deps/c-ares
creating build/temp.macosx-10.14-x86_64-3.7/deps/c-ares/src
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -DHAVE_CONFIG_H=1 -D_LARGEFILE_SOURCE=1 -D_FILE_OFFSET_BITS=64 -D_DARWIN_USE_64_BIT_INODE=1 -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m -Ideps/c-ares/src -Ideps/c-ares/src/config_darwin -c src/pycares.c -o build/temp.macosx-10.14-x86_64-3.7/src/pycares.o
In file included from src/pycares.c:4:
src/cares.c:302:72: error: no member named 'ttl' in 'struct ares_mx_reply'
PyStructSequence_SET_ITEM(tmp, 2, PyInt_FromLong((long)mx_ptr->ttl));
~~~~~~ ^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/structseq.h:38:69: note: expanded from macro 'PyStructSequence_SET_ITEM'
#define PyStructSequence_SET_ITEM(op, i, v) PyTuple_SET_ITEM(op, i, v)
^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/tupleobject.h:62:75: note: expanded from macro 'PyTuple_SET_ITEM'
#define PyTuple_SET_ITEM(op, i, v) (((PyTupleObject *)(op))->ob_item[i] = v)
^
In file included from src/pycares.c:4:
src/cares.c:415:95: error: too many arguments to function call, expected 6, have 7
parse_status = ares_parse_ptr_reply(answer_buf, answer_len, NULL, 0, AF_UNSPEC, &hostent, &hostttls);
~~~~~~~~~~~~~~~~~~~~ ^~~~~~~~~
/usr/local/include/ares.h:581:14: note: 'ares_parse_ptr_reply' declared here
CARES_EXTERN int ares_parse_ptr_reply(const unsigned char *abuf,
^
In file included from src/pycares.c:4:
src/cares.c:522:81: error: no member named 'ttl' in 'struct ares_txt_ext'
PyStructSequence_SET_ITEM(tmp_obj, 1, PyInt_FromLong((long)txt_ptr->ttl));
~~~~~~~ ^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/structseq.h:38:69: note: expanded from macro 'PyStructSequence_SET_ITEM'
#define PyStructSequence_SET_ITEM(op, i, v) PyTuple_SET_ITEM(op, i, v)
^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/tupleobject.h:62:75: note: expanded from macro 'PyTuple_SET_ITEM'
#define PyTuple_SET_ITEM(op, i, v) (((PyTupleObject *)(op))->ob_item[i] = v)
^
In file included from src/pycares.c:4:
src/cares.c:597:78: error: no member named 'ttl' in 'struct ares_soa_reply'
PyStructSequence_SET_ITEM(dns_result, 7, PyInt_FromLong((long)soa_reply->ttl));
~~~~~~~~~ ^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/structseq.h:38:69: note: expanded from macro 'PyStructSequence_SET_ITEM'
#define PyStructSequence_SET_ITEM(op, i, v) PyTuple_SET_ITEM(op, i, v)
^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/tupleobject.h:62:75: note: expanded from macro 'PyTuple_SET_ITEM'
#define PyTuple_SET_ITEM(op, i, v) (((PyTupleObject *)(op))->ob_item[i] = v)
^
In file included from src/pycares.c:4:
src/cares.c:664:73: error: no member named 'ttl' in 'struct ares_srv_reply'
PyStructSequence_SET_ITEM(tmp, 4, PyInt_FromLong((long)srv_ptr->ttl));
~~~~~~~ ^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/structseq.h:38:69: note: expanded from macro 'PyStructSequence_SET_ITEM'
#define PyStructSequence_SET_ITEM(op, i, v) PyTuple_SET_ITEM(op, i, v)
^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/tupleobject.h:62:75: note: expanded from macro 'PyTuple_SET_ITEM'
#define PyTuple_SET_ITEM(op, i, v) (((PyTupleObject *)(op))->ob_item[i] = v)
^
In file included from src/pycares.c:4:
src/cares.c:735:75: error: no member named 'ttl' in 'struct ares_naptr_reply'
PyStructSequence_SET_ITEM(tmp, 6, PyInt_FromLong((long)naptr_ptr->ttl));
~~~~~~~~~ ^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/structseq.h:38:69: note: expanded from macro 'PyStructSequence_SET_ITEM'
#define PyStructSequence_SET_ITEM(op, i, v) PyTuple_SET_ITEM(op, i, v)
^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/tupleobject.h:62:75: note: expanded from macro 'PyTuple_SET_ITEM'
#define PyTuple_SET_ITEM(op, i, v) (((PyTupleObject *)(op))->ob_item[i] = v)
^
6 errors generated.
error: command 'clang' failed with exit status 1
----------------------------------------
Failed building wheel for pycares
Running setup.py clean for pycares
Failed to build pycares
Installing collected packages: pycares, aiodns, esd
Running setup.py install for pycares ... error
Complete output from command /usr/local/opt/python/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/xx/gtlvnn65597bczwl0btk0x6w0000gn/T/pip-install-bl17e652/pycares/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/xx/gtlvnn65597bczwl0btk0x6w0000gn/T/pip-record-9w92d9q1/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.macosx-10.14-x86_64-3.7
creating build/lib.macosx-10.14-x86_64-3.7/pycares
copying pycares/errno.py -> build/lib.macosx-10.14-x86_64-3.7/pycares
copying pycares/_version.py -> build/lib.macosx-10.14-x86_64-3.7/pycares
copying pycares/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/pycares
copying pycares/__main__.py -> build/lib.macosx-10.14-x86_64-3.7/pycares
running build_ext
building 'pycares._core' extension
creating build/temp.macosx-10.14-x86_64-3.7
creating build/temp.macosx-10.14-x86_64-3.7/src
creating build/temp.macosx-10.14-x86_64-3.7/deps
creating build/temp.macosx-10.14-x86_64-3.7/deps/c-ares
creating build/temp.macosx-10.14-x86_64-3.7/deps/c-ares/src
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -DHAVE_CONFIG_H=1 -D_LARGEFILE_SOURCE=1 -D_FILE_OFFSET_BITS=64 -D_DARWIN_USE_64_BIT_INODE=1 -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m -Ideps/c-ares/src -Ideps/c-ares/src/config_darwin -c src/pycares.c -o build/temp.macosx-10.14-x86_64-3.7/src/pycares.o
In file included from src/pycares.c:4:
src/cares.c:302:72: error: no member named 'ttl' in 'struct ares_mx_reply'
PyStructSequence_SET_ITEM(tmp, 2, PyInt_FromLong((long)mx_ptr->ttl));
~~~~~~ ^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/structseq.h:38:69: note: expanded from macro 'PyStructSequence_SET_ITEM'
#define PyStructSequence_SET_ITEM(op, i, v) PyTuple_SET_ITEM(op, i, v)
^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/tupleobject.h:62:75: note: expanded from macro 'PyTuple_SET_ITEM'
#define PyTuple_SET_ITEM(op, i, v) (((PyTupleObject *)(op))->ob_item[i] = v)
^
In file included from src/pycares.c:4:
src/cares.c:415:95: error: too many arguments to function call, expected 6, have 7
parse_status = ares_parse_ptr_reply(answer_buf, answer_len, NULL, 0, AF_UNSPEC, &hostent, &hostttls);
~~~~~~~~~~~~~~~~~~~~ ^~~~~~~~~
/usr/local/include/ares.h:581:14: note: 'ares_parse_ptr_reply' declared here
CARES_EXTERN int ares_parse_ptr_reply(const unsigned char *abuf,
^
In file included from src/pycares.c:4:
src/cares.c:522:81: error: no member named 'ttl' in 'struct ares_txt_ext'
PyStructSequence_SET_ITEM(tmp_obj, 1, PyInt_FromLong((long)txt_ptr->ttl));
~~~~~~~ ^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/structseq.h:38:69: note: expanded from macro 'PyStructSequence_SET_ITEM'
#define PyStructSequence_SET_ITEM(op, i, v) PyTuple_SET_ITEM(op, i, v)
^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/tupleobject.h:62:75: note: expanded from macro 'PyTuple_SET_ITEM'
#define PyTuple_SET_ITEM(op, i, v) (((PyTupleObject *)(op))->ob_item[i] = v)
^
In file included from src/pycares.c:4:
src/cares.c:597:78: error: no member named 'ttl' in 'struct ares_soa_reply'
PyStructSequence_SET_ITEM(dns_result, 7, PyInt_FromLong((long)soa_reply->ttl));
~~~~~~~~~ ^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/structseq.h:38:69: note: expanded from macro 'PyStructSequence_SET_ITEM'
#define PyStructSequence_SET_ITEM(op, i, v) PyTuple_SET_ITEM(op, i, v)
^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/tupleobject.h:62:75: note: expanded from macro 'PyTuple_SET_ITEM'
#define PyTuple_SET_ITEM(op, i, v) (((PyTupleObject *)(op))->ob_item[i] = v)
^
In file included from src/pycares.c:4:
src/cares.c:664:73: error: no member named 'ttl' in 'struct ares_srv_reply'
PyStructSequence_SET_ITEM(tmp, 4, PyInt_FromLong((long)srv_ptr->ttl));
~~~~~~~ ^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/structseq.h:38:69: note: expanded from macro 'PyStructSequence_SET_ITEM'
#define PyStructSequence_SET_ITEM(op, i, v) PyTuple_SET_ITEM(op, i, v)
^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/tupleobject.h:62:75: note: expanded from macro 'PyTuple_SET_ITEM'
#define PyTuple_SET_ITEM(op, i, v) (((PyTupleObject *)(op))->ob_item[i] = v)
^
In file included from src/pycares.c:4:
src/cares.c:735:75: error: no member named 'ttl' in 'struct ares_naptr_reply'
PyStructSequence_SET_ITEM(tmp, 6, PyInt_FromLong((long)naptr_ptr->ttl));
~~~~~~~~~ ^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/structseq.h:38:69: note: expanded from macro 'PyStructSequence_SET_ITEM'
#define PyStructSequence_SET_ITEM(op, i, v) PyTuple_SET_ITEM(op, i, v)
^
/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/include/python3.7m/tupleobject.h:62:75: note: expanded from macro 'PyTuple_SET_ITEM'
#define PyTuple_SET_ITEM(op, i, v) (((PyTupleObject *)(op))->ob_item[i] = v)
^
6 errors generated.
error: command 'clang' failed with exit status 1
----------------------------------------
Command "/usr/local/opt/python/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/xx/gtlvnn65597bczwl0btk0x6w0000gn/T/pip-install-bl17e652/pycares/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/xx/gtlvnn65597bczwl0btk0x6w0000gn/T/pip-record-9w92d9q1/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/xx/gtlvnn65597bczwl0btk0x6w0000gn/T/pip-install-bl17e652/pycares/
例:
python3 ESD.py
输出帮助信息
问题描述: ESD支持配置shodan_key来进行使用domain语法搜索子域名,但是普通的shodan账号会出现shodan.exception.APIError: Please upgrade your API plan to use filters or paging.错误。一旦出错后不管如何重新安装ESD库也还是会出错
问题根本:
ESD在使用shodan库会进行初始化,会在用户家目录的api_key或者ESD库目录下取key.ini得shodan节段的内容
解决办法:
步骤1. 切换到当前用户的家目录下
cd ~/.shodan
rm -f api_key
步骤2. 切换到ESD库目录底下
#这里的ESD目录请切换成自己的
find / -name "key.ini" 2>/dev/null
cd /usr/local/lib/python3.7/site-packages/ESD/
rm key.ini
1.做了泛解析的域名,不存在的子域名能解析到一个IP,但这个子域名或对应的IP 不一定开放http/https服务啊!,,后续进行相似比较,只要是域名存在但没有开发http 服务的,都会被漏掉吧。
现在字典已经达到60w,在扫描一些子域名较少的域名时,实际在扫描但因为没有新子域名扫到没有输出导致可能以为Hang住。
泛解析域名并不是不能扫描,可以试着通过随机1000个子域解析出的IP,来添加一个泛解析的IP黑名单。在接下来的扫描中,只需要忽略解析到了黑名单IP的域名就好... 😆
目前支持A和CNAME记录,计划支持AAAA,SOA,TXT,MX等类型
可以使用 https://github.com/john-kurkowski/tldextract 来判断域名有效性,扫描前过滤掉类似 123.aa
这种无效的域名
Traceback (most recent call last): File "/usr/local/bin/esd", line 11, in <module> load_entry_point('ESD==0.0.24', 'console_scripts', 'esd')() File "/usr/local/lib/python3.7/dist-packages/ESD-0.0.24-py3.7.egg/ESD/__init__.py", line 1460, in main esd.run() File "/usr/local/lib/python3.7/dist-packages/ESD-0.0.24-py3.7.egg/ESD/__init__.py", line 1135, in run is_all_stable_dns = stable_dns.count(stable_dns[0]) == len(stable_dns) IndexError: list index out of range
报错了老哥。
2020-01-07 07:05:57,616 [ESD] [INFO] Shodan subdomain count: 0
Traceback (most recent call last):
File "/usr/local/bin/esd", line 10, in
sys.exit(main())
File "/usr/local/lib/python3.7/dist-packages/ESD/init.py", line 13, in main
esd.run()
File "/usr/local/lib/python3.7/dist-packages/ESD/engine.py", line 574, in run
is_success = zoomeye.initialize(base_dir)
File "/usr/local/lib/python3.7/dist-packages/ESD/lib/thirdpartApi.py", line 113, in initialize
resp_json = json.loads(resp.text)
File "/usr/lib/python3.7/json/init.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
如果失效可以去掉,或者单纯做个接口放着做备用
每个厂商都有自己专属特征域名,比如蘑菇街的(molidai.mogujie.com
)。
对最终子域名结果集的响应内容进行爬取,分析其中包含的子域名,加入结果集作为最终结果。
挂个二维码吧 给你打钱
ModuleNotFoundError: No module named 'dns'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
prepare(preparation_data)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 263, in run_path
return _run_module_code(code, init_globals, run_name,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/Users/admin/Documents/hack/SRC/test.py", line 3, in <module>
domains = EnumSubDomain('darkless.cn').run()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ESD/engine.py", line 533, in run
subdomains_queue = multiprocessing.Manager().list()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 57, in Manager
m.start()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/managers.py", line 579, in start
self._process.start()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Traceback (most recent call last):
File "/Users/admin/Documents/hack/SRC/test.py", line 3, in <module>
domains = EnumSubDomain('darkless.cn').run()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ESD/engine.py", line 533, in run
subdomains_queue = multiprocessing.Manager().list()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 57, in Manager
m.start()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/managers.py", line 583, in start
self._address = reader.recv()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
域名爆破遇到泛解析挺常见的,还是应该解决下,我觉得chu的这个文章中,ip+ttl的方法效果应该还是不错的
http://sh3ll.me/archives/201704041222.txt
每次都需先进行暴力破解,能否跳过此阶段直接进入后续枚举。
Lines 161 to 163 in 116ff70
传入参数 ns_addr
为 Answer object
类型,dns.zone.from_xfr
调用时会出现异常
TypeError: str, bytes or bytearray expected, not Answer
修改 ns_addr
为 ns_addr.rrset.items[0].address
解决异常情况
程序内调用方法:
from ESD import EnumSubDomain
domains = EnumSubDomain('feei.cn').run()
错误提示:
Traceback (most recent call last):
File "webmon.py", line 17, in <module>
main()
File "webmon.py", line 13, in main
domains = EnumSubDomain('feei.cn').run()
File "/usr/local/lib/python3.6/site-packages/ESD/__init__.py", line 1235, in run
fofa = FofaEngine(self.fofa_struct, self.conf, self.domain)
File "/usr/local/lib/python3.6/site-packages/ESD/__init__.py", line 271, in __init__
self.email = fofa_struct['femail']
KeyError: 'femail'
老板,找不到配置key的文件了
部分域名会增加访问速度,会针对不同线路(电信、联通、移动、教育)解析不同IP,而不同DNS服务商对于网络出口的线路判定不一定准确,导致同一网络、同一域名下针对不同的DNS会出现不同的IP。
W-IPS
和W-Response Text
W-IPS
不一致的子域名加入结果列表
结果列表
中的子域组成新字典
新字典
子域名的Response Text
和W-Response Text
比较相似度90%
的为泛解析域名则丢弃,否则加入结果列表
结果列表
包含了常规子域名爆破和泛解析子域名的响应相似度的集合改为一次生成最终的字典文件,同时可以配置动态字典规则
如下
bigfun.fangdd.com 103.235.228.41
philippinesfoodrecipes.fangdd.com103.235.228.41
oo49.fangdd.com 103.235.228.41
vd55.fangdd.com 103.235.228.41
fe44.fangdd.com 103.235.228.41
maxmegacuriosidades.fangdd.com103.235.228.41
bc93.fangdd.com 103.235.228.41
这样不方便其他工具解析结果
@
centos7 py3.6 pip安装 无法加载命令行
(ESD) [22:00:21] xxxxxx ➜ code/github/esd» esd uulian.com
2018-07-12 22:00:40,224 [ESD] [INFO] Debug: False
2018-07-12 22:00:40,224 [ESD] [INFO] Total target domains: 1
2018-07-12 22:00:40,224 [ESD] [INFO] Version: 0.0.11
2018-07-12 22:00:40,224 [ESD] [INFO] ----------
2018-07-12 22:00:40,224 [ESD] [INFO] Start domain: uulian.com
2018-07-12 22:00:40,360 [ESD] [INFO] Sub domain dict count: 170084
2018-07-12 22:00:40,360 [ESD] [INFO] Generate coroutines...
2018-07-12 22:00:40,413 [ESD] [INFO] 114.114.114.114 feei-esd-735 ['121.40.62.25']
2018-07-12 22:00:51,521 [ESD] [INFO] 223.6.6.6 feei-esd-735 None
2018-07-12 22:00:55,818 [ESD] [INFO] 223.5.5.5 feei-esd-735 ['121.40.62.25']
2018-07-12 22:00:55,818 [ESD] [INFO] Is all stable dns: NO, use the default dns server
2018-07-12 22:00:55,825 [ESD] [INFO] This is a wildcard domain, will enumeration subdomains use by DNS+RSC.
2018-07-12 22:00:55,826 [ESD] [INFO] Wildcard IPS: ['121.40.62.25']
Traceback (most recent call last):
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/urllib3/util/connection.py", line 56, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/Users/xxxxxx/.pyenv/versions/3.6.4/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Users/xxxxxx/.pyenv/versions/3.6.4/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Users/xxxxxx/.pyenv/versions/3.6.4/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Users/xxxxxx/.pyenv/versions/3.6.4/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Users/xxxxxx/.pyenv/versions/3.6.4/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/Users/xxxxxx/.pyenv/versions/3.6.4/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/urllib3/connection.py", line 196, in connect
conn = self._new_conn()
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x106120898>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
timeout=timeout
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='feei-esd-735.uulian.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x106120898>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/xxxxxx/code/github/ESD/bin/esd", line 11, in
sys.exit(main())
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/ESD/init.py", line 533, in main
esd.run()
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/ESD/init.py", line 433, in run
self.wildcard_html = requests.get('http://{w_sub}.{domain}'.format(w_sub=self.wildcard_sub, domain=self.domain), headers=self.request_headers, timeout=10).text
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
resp = self.send(prep, **send_kwargs)
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
r = adapter.send(request, **kwargs)
File "/Users/xxxxxx/code/github/ESD/lib/python3.6/site-packages/requests/adapters.py", line 513, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='feei-esd-735.uulian.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x106120898>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
you could try spyse API to reduce the problems and increase the speed. Hope it helps
root@kali:~/ESD# python setup.py
Traceback (most recent call last):
File "setup.py", line 18, in
from ESD import version
File "/root/ESD/ESD/init.py", line 176
async def query(self, sub):
^
SyntaxError: invalid syntax
root@kali:~/ESD# python3 setup.py
Traceback (most recent call last):
File "setup.py", line 18, in
from ESD import version
File "/root/ESD/ESD/init.py", line 25, in
import uvloop
ModuleNotFoundError: No module named 'uvloop'
能把shodan,censys,等接口加进去吗
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.