- Email:
deedy
.ru
@gmail
.com
- Telegram:
msboto
deedy5 / requests_tor Goto Github PK
View Code? Open in Web Editor NEWMultithreading requests via TOR with automatic TOR new identity
License: MIT License
Multithreading requests via TOR with automatic TOR new identity
License: MIT License
deedy
.ru
@gmail
.com
msboto
I get an error:
from requests_tor import RequestsTor
# If you use the Tor browser
rt = RequestsTor()
# If you use the Tor
# rt = RequestsTor(tor_ports=(9050,), tor_cport=9051)
url = 'https://httpbin.org/anything'
r = rt.get(url)
print(r.text)
urls = ['https://foxnews.com', 'https://nbcnews.com', 'https://wsj.com/news/world',
'https://abcnews.go.com', 'https://cbsnews.com', 'https://nytimes.com',
'https://usatoday.com','https://reuters.com/world', 'http://bbc.com/news',
'https://theguardian.com/world', 'https://cnn.com', 'https://apnews.com']
r = rt.get_urls(urls)
print(r[-1].text)
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/socks.py](https://localhost:8080/#) in connect(self, dest_pair, catch_errors)
786 # Initial connection to proxy server.
--> 787 super(socksocket, self).connect(proxy_addr)
788
OSError: [Errno 99] Cannot assign requested address
During handling of the above exception, another exception occurred:
ProxyConnectionError Traceback (most recent call last)
19 frames
[/usr/local/lib/python3.7/dist-packages/urllib3/contrib/socks.py](https://localhost:8080/#) in _new_conn(self)
87 timeout=self.timeout,
---> 88 **extra_kw
89 )
[/usr/local/lib/python3.7/dist-packages/socks.py](https://localhost:8080/#) in create_connection(dest_pair, timeout, source_address, proxy_type, proxy_addr, proxy_port, proxy_rdns, proxy_username, proxy_password, socket_options)
208 if err:
--> 209 raise err
210
[/usr/local/lib/python3.7/dist-packages/socks.py](https://localhost:8080/#) in create_connection(dest_pair, timeout, source_address, proxy_type, proxy_addr, proxy_port, proxy_rdns, proxy_username, proxy_password, socket_options)
198
--> 199 sock.connect((remote_host, remote_port))
200 return sock
[/usr/local/lib/python3.7/dist-packages/socks.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
46 self.setblocking(True)
---> 47 return function(*args, **kwargs)
48 except Exception as e:
[/usr/local/lib/python3.7/dist-packages/socks.py](https://localhost:8080/#) in connect(self, dest_pair, catch_errors)
799 log.debug("%s due to: %s", msg, error)
--> 800 raise ProxyConnectionError(msg, error)
801 else:
ProxyConnectionError: Error connecting to SOCKS5 proxy localhost:9150: [Errno 99] Cannot assign requested address
During handling of the above exception, another exception occurred:
NewConnectionError Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py](https://localhost:8080/#) in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
599 body=body, headers=headers,
--> 600 chunked=chunked)
601
[/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py](https://localhost:8080/#) in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
342 try:
--> 343 self._validate_conn(conn)
344 except (SocketTimeout, BaseSSLError) as e:
[/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py](https://localhost:8080/#) in _validate_conn(self, conn)
838 if not getattr(conn, 'sock', None): # AppEngine might not have `.sock`
--> 839 conn.connect()
840
[/usr/local/lib/python3.7/dist-packages/urllib3/connection.py](https://localhost:8080/#) in connect(self)
300 # Add certificate verification
--> 301 conn = self._new_conn()
302 hostname = self.host
[/usr/local/lib/python3.7/dist-packages/urllib3/contrib/socks.py](https://localhost:8080/#) in _new_conn(self)
109 self,
--> 110 "Failed to establish a new connection: %s" % error
111 )
NewConnectionError: <urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x7f10527dc7d0>: Failed to establish a new connection: [Errno 99] Cannot assign requested address
During handling of the above exception, another exception occurred:
MaxRetryError Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/requests/adapters.py](https://localhost:8080/#) in send(self, request, stream, timeout, verify, cert, proxies)
498 retries=self.max_retries,
--> 499 timeout=timeout,
500 )
[/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py](https://localhost:8080/#) in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
637 retries = retries.increment(method, url, error=e, _pool=self,
--> 638 _stacktrace=sys.exc_info()[2])
639 retries.sleep()
[/usr/local/lib/python3.7/dist-packages/urllib3/util/retry.py](https://localhost:8080/#) in increment(self, method, url, response, error, _pool, _stacktrace)
398 if new_retry.is_exhausted():
--> 399 raise MaxRetryError(_pool, url, error or ResponseError(cause))
400
MaxRetryError: SOCKSHTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /anything (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x7f10527dc7d0>: Failed to establish a new connection: [Errno 99] Cannot assign requested address'))
During handling of the above exception, another exception occurred:
ConnectionError Traceback (most recent call last)
[<ipython-input-3-70877019d4e9>](https://localhost:8080/#) in <module>()
8
9 url = 'https://httpbin.org/anything'
---> 10 r = rt.get(url)
11 print(r.text)
12
[/usr/local/lib/python3.7/dist-packages/requests_tor.py](https://localhost:8080/#) in get(self, url, **kwargs)
88 def get(self, url, **kwargs):
89 method = 'GET'
---> 90 return self.request(method, url, **kwargs)
91
92 def post(self, url, **kwargs):
[/usr/local/lib/python3.7/dist-packages/requests_tor.py](https://localhost:8080/#) in request(self, method, url, **kwargs)
79
80 kwargs["headers"] = kwargs.get("headers", TOR_HEADERS)
---> 81 resp = requests.request(method, url, **kwargs, proxies=proxies)
82 if self.verbose:
83 print(f"SocksPort={port} status={resp.status_code} url={resp.url}")
[/usr/local/lib/python3.7/dist-packages/requests/api.py](https://localhost:8080/#) in request(method, url, **kwargs)
57 # cases, and look like a memory leak in others.
58 with sessions.Session() as session:
---> 59 return session.request(method=method, url=url, **kwargs)
60
61
[/usr/local/lib/python3.7/dist-packages/requests/sessions.py](https://localhost:8080/#) in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
585 }
586 send_kwargs.update(settings)
--> 587 resp = self.send(prep, **send_kwargs)
588
589 return resp
[/usr/local/lib/python3.7/dist-packages/requests/sessions.py](https://localhost:8080/#) in send(self, request, **kwargs)
699
700 # Send the request
--> 701 r = adapter.send(request, **kwargs)
702
703 # Total elapsed time of the request (approximately)
[/usr/local/lib/python3.7/dist-packages/requests/adapters.py](https://localhost:8080/#) in send(self, request, stream, timeout, verify, cert, proxies)
563 raise SSLError(e, request=request)
564
--> 565 raise ConnectionError(e, request=request)
566
567 except ClosedPoolError as e:
ConnectionError: SOCKSHTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /anything (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x7f10527dc7d0>: Failed to establish a new connection: [Errno 99] Cannot assign requested address'))
Hi, Tor ip will change after every God damn request.
ports = []
for id in range(1000):
ports.append(5000 + id)
rt = RequestsTor(tor_ports=ports, autochange_id=0)
url = 'https://httpbin.org/anything'
first = rt.get(url)
print(first.text)
second = rt.get(url)
print(second.text)
Python 3.9.5 (tags/v3.9.5:0a7dcbd, May 3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32
from requests_tor import RequestsTor
rt = RequestsTor(tor_ports=(9000, 9001, 9002, 9003, 9004), autochange_id=1)
urls = (f'https://habr.com/ru/post/{x}' for x in range(1, 50))
r = rt.get_urls(urls)
print(r[-1].text)
Traceback (most recent call last):
File "", line 6, in
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\site-packages\requests_tor.py", line 105, in get_urls
temp_results = [resp for resp in
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\site-packages\requests_tor.py", line 105, in
temp_results = [resp for resp in
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\concurrent\futures_base.py", line 608, in result_iterator
yield fs.pop().result()
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\concurrent\futures_base.py", line 445, in result
return self.__get_result()
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\concurrent\futures_base.py", line 390, in __get_result
raise self._exception
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\concurrent\futures\thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\site-packages\requests_tor.py", line 77, in get
return self.request(method, url, **kwargs)
File "C:\Users\jbard\AppData\Local\Programs\Python\Python39\lib\site-packages\requests_tor.py", line 69, in request
resp = requests.request(method, url, **kwargs, proxies=proxies)
File "C:\Users\jbard\PycharmProjects\Parallel\venv\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\jbard\PycharmProjects\Parallel\venv\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\jbard\PycharmProjects\Parallel\venv\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "C:\Users\jbard\PycharmProjects\Parallel\venv\lib\site-packages\requests\adapters.py", line 412, in send
conn = self.get_connection(request.url, proxies)
File "C:\Users\jbard\PycharmProjects\Parallel\venv\lib\site-packages\requests\adapters.py", line 309, in get_connection
proxy_manager = self.proxy_manager_for(proxy)
File "C:\Users\jbard\PycharmProjects\Parallel\venv\lib\site-packages\requests\adapters.py", line 182, in proxy_manager_for
manager = self.proxy_manager[proxy] = SOCKSProxyManager(
File "C:\Users\jbard\PycharmProjects\Parallel\venv\lib\site-packages\requests\adapters.py", line 47, in SOCKSProxyManager
raise InvalidSchema("Missing dependencies for SOCKS support.")
requests.exceptions.InvalidSchema: Missing dependencies for SOCKS support.
All is write on the title. So i should write it on the file? I mean "ControlPort : 9051" must be in a comment for being uncomment.
Is your feature request related to a problem? Please describe.
I want to send post request concurrently like we do in get_urls() function
Describe the solution you'd like
similar to get_urls like to see post_urls function with additional fields like data and headers
Describe alternatives you've considered
If this kind feature can't be implemented can you mention it here that how can i send post request concurrently using RequestTor object.
Additional context
Add any other context or screenshots about the feature request here.
Hi,
I have just tried your package. It seems you get request don't accept params, headers and other arguments from requests library.
For example this code:
r = rt.get("https://oss.uredjenazemlja.hr/rest/katHr/parcelInfo",
params={
"id": 2786835,
"status": encode(number, int(12414714730308))
},
headers={
"Origin": "https://www.katastar.hr",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"
}
returns TypeError: get() got an unexpected keyword argument 'params'
# trying code;
from requests_tor import RequestsTor
rt = RequestsTor()
rt.check_ip()
Error ->
�Traceback (most recent call last):
File "C:\Users\deniz\AppData\Local\Programs\Python\Python310\lib\site-packages\socks.py", line 787, in connect
super(socksocket, self).connect(proxy_addr)
ConnectionRefusedError: [WinError 10061] The connection could not be established because the target machine actively refused.
Please help.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.