timwhitez / crawlergo_x_xray Goto Github PK

View Code? Open in Web Editor NEW

1.2K 1.2K 214.0 101 KB

360/0Kee-Team/crawlergo动态爬虫结合长亭XRAY扫描器的被动扫描功能

Python 100.00%

crawlergo_x_xray's Introduction

About Me

Offensive Security Researcher.
Interested in Defense Evasion techniques.
📫 How to reach me: [email protected]

关于我

红队攻防技术研究
免杀/对抗爱好者
📫 联系我: [email protected]

Github stats & metrics

etc

开源的样本大部分可能已经无法免杀,需要自行修改
许多项目都有借鉴github上其它项目
若用本人项目去进行：HW演练/红蓝对抗/APT/黑产/恶意行为/违法行为/割韭菜，等行为，本人概不负责，也与本人无关
本人早已不参与大小HW活动的攻击方了，若溯源到timwhite id与本人无关

Visit Count

crawlergo_x_xray's People

Contributors

Stargazers

Watchers

Forkers

gh-jy 1f3lse crackercat eastsunwong t3st0r-git thr0cyte xieqiwen1993 maxncu jguoguo sry309 pt001 qing-q grayguest totoro1520 b4jewer arryboom devsecops-src xiaofengtongxue d4wner doge-dog dataorz echocipher sie504 wangyi0127 alanlonglong yizhimanpadewoniu supermike96 huangyuan666 p4yche kenanat webvul fngoo xschur longxiang090 dennyx liwuqi haoirui 5icorgi lw1988 icysun h1d3r trevor3000 saomase om1jerry iamjazz july0426 phpplay test2504 thinks520 0xf4vul zk953874391 r4b3rt htesiege wifido shamo666 smitnald cgaiide lxkxc lyhistory kevil-hui fk-prog jojoli123 dayhappy-jpg simon0105 xiaohutuchong lhtest429 ilkkai mrpointsun pyking kenuoseclab layman404 popmedd quinn-yan harmoc noosec bigbrobro hariyopmail eniac888 slowmistio wonderkun fbisbb cnonce test502git wxyadmin omiter flaray npc7 sunian19 jiushill liuyifei0226 dclangx lxngoddess5321 fwxiong insu002 lsr00ter strawerlar hyperia-cn deepwebhacker coollce guaguala

crawlergo_x_xray's Issues

win7 x64 error

C:\Users\Public\crawlergo_x_XRAY>python3 launcher.py
Traceback (most recent call last):
File "launcher.py", line 83, in
main(data1)
File "launcher.py", line 59, in main
rsp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "C:\Program Files\Python37\lib\subprocess.py", line 800, in init
restore_signals, start_new_session)
File "C:\Program Files\Python37\lib\subprocess.py", line 1207, in _execute_child
startupinfo)

#tclose=0 #while tclose==0 or urls_queue.empty() == False: #tclose=1

crawlergo_x_XRAY/launcher.py

Line 13 in ef81270

tclose=0

使用launcher_new.py出现fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached的解决方案

launcher_new.py使用了fake-useragent可能会出现报错：
fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached

本人尝试了如下解决方案：

先：
pip3 install --upgrade fake_useragent

1.修改代码launcher_new.py第十一行
ua = UserAgent(use_cache_server=False,verify_ssl=False,cache=False)

2.利用临时文件
下载https://fake-useragent.herokuapp.com/browsers/0.1.11 //好像国内无法访问
改名为fake_useragent_0.1.11.json
放入临时文件夹(如/tmp)

import tempfile
tempfile.gettempdir()

都无法解决

查看代码site-packages/fake_useragent/settings.py
发现要请求BROWSERS_STATS_PAGE = 'https://www.w3schools.com/browsers/default.asp'
但是该url好像国内无法访问所以报该错误

所以建议使用国外vps，或者弃用fake-useragent库。

请问支持过滤静态么

可以增加同时运行xray和w13scan的支持吗？

ca证书

没有装ca证书也能用代理模式抓https？

PermissionError: [WinError 5] 拒绝访问。

Traceback (most recent call last):
File "launcher.py", line 83, in
main(data1)
File "launcher.py", line 59, in main
rsp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "C:\Users\DELL\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 854, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\DELL\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1307, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
PermissionError: [WinError 5] 拒绝访问。

请问这个情况怎么解决

运行报错

环境python3 Windows server16

OSError: [WinError 193] %1 不是有效的 Win32 应用程序。

crawlergo_x_XRAY-master>python launcher_new.py
File "launcher_new.py", line 29
cmd = ["./crawlergo", "-c", "C:\Users\gkd\Desktop\chrome-win\chrome.exe","-t", "5","-f","smart","--fuzz-path", "--push-to-proxy", "http://127.0.0.1:7888/", "--push-pool-max", "10","--output-mode", "json" , target]
^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape

C:\Users\gkd\Desktop\crawlergo_x_XRAY-master>python launcher_new.py
Traceback (most recent call last):
File "launcher_new.py", line 50, in
main(data1)
File "launcher_new.py", line 30, in main
rsp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "C:\Users\gkd\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 854, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\gkd\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1307, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
OSError: [WinError 193] %1 不是有效的 Win32 应用程序。

大佬，这种拒绝访问是什么情况

大佬，这个问题是什么原因造成，您帮助给解答一下。

在targets中的url，爬虫显示爬取到了。x详见下图

但是在xray缺看不到扫描，扫描的都是一些别的网址

subprocess方法的communicate()函数会卡住

subprocess方法的communicate()函数会卡住。

command命令在powershell能成功执行，但是放到subprocess的communicate()就会卡死。
我自己写了个test，用command命令会卡死，用dir/b命令就不会卡。

#!/usr/bin/python3
\# coding: utf-8
import simplejson
import threading
import subprocess
import requests
import warnings
warnings.filterwarnings(action='ignore')

def runcmd():
    command = ["./crawlergo", "-c", "D:\tools\crawlergo_x_XRAY\chromium\chrome.exe","-t", "20","-f","smart","--fuzz-path", "--output-mode", "json", "http://testphp.vulnweb.com/"]
    #command = ['dir','/b']
    #print(command)
    ret = subprocess.Popen(command, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    output, error = ret.communicate()
    print(output.decode('gbk'))
    print(error.decode('gbk'))

#runcmd(["dir","/b"])#序列参数
#runcmd("exit 1")#字符串参数

runcmd()

请问这个是这个函数的bug吗?还是我8g内存太小了？
有没有解决方法？

FileNotFoundError: [WinError 2] 系统找不到指定的文件。

运行提示FileNotFoundError: [WinError 2] 系统找不到指定的文件。
查了路径是没有问题的
报错信息如下：
Traceback (most recent call last):
File "launcher.py", line 89, in
main(data1)
File "launcher.py", line 59, in main
rsp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "H:\python\python3.7\lib\subprocess.py", line 775, in init
restore_signals, start_new_session)
File "H:\python\python3.7\lib\subprocess.py", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] 系统找不到指定的文件。

HTTP请求头会泄露Spider-Name: crawlergo，如何去除？

HTTP请求头字段会泄露Spider-Name: crawlergo，如何去除？