Python controller library for Tor
Home Page: https://stem.torproject.org/
License: GNU Lesser General Public License v3.0
Stem is a Python controller library for Tor. With it you can use Tor's control protocol to script against the Tor process, or build things such as Nyx.
Documentation and tutorials available at stem.torproject.org.
There is code inside parse_file function which determinate type of descryptor by file name:
if filename == 'cached-descriptors' or filename == 'cached-descriptors.new':
return stem.descriptor.server_descriptor._parse_file(descriptor_file, validate = validate, **kwargs)
elif filename == 'cached-extrainfo' or filename == 'cached-extrainfo.new':
return stem.descriptor.extrainfo_descriptor._parse_file(descriptor_file, validate = validate, **kwargs)
elif filename == 'cached-microdescs' or filename == 'cached-microdescs.new':
return stem.descriptor.microdescriptor._parse_file(descriptor_file, validate = validate, **kwargs)
elif filename == 'cached-consensus':
return stem.descriptor.networkstatus._parse_file(descriptor_file, validate = validate, document_handler = document_handler, **kwargs)
elif filename == 'cached-microdesc-consensus':
return stem.descriptor.networkstatus._parse_file(descriptor_file, is_microdescriptor = True, validate = validate, document_handler = document_handler, **kwargs)
else:
raise TypeError("Unable to determine the descriptor's type. filename: '%s', first line: '%s'" % (filename, first_line))So we can add for each type of descriptor some class variable which store default cache file name. Profit: We can use it here above. And we can use it when we want save to disk such document with this default file name.
Migrated from ticket 13910.
Txtorcon uses a few neat things, one of them is Coveralls. This does a nice job of gamifying code coverage, and might be fun to do for Stem too. We already have automated test runs, so we just need to figure out how to integrate it.
With config values like
Nickname zwiebeltoralf
Address 5.9.158.75
OutboundBindAddress 5.9.158.75
OutboundBindAddress [2a01:4f8:190:514a::2]
DirPort 5.9.158.75:80
ORPort 5.9.158.75:443
DirPort [2a01:4f8:190:514a::2]:80 NoAdvertise
ORPort [2a01:4f8:190:514a::2]:443
ControlPort 9051
I run into that issue. A circumvention is
or_ports = controller.get_ports(Listener.OR)
if not or_ports:
or_ports = [ 443 ] if control_port == 9051 else [ 9001 ]
print("warn: have to guess OR port(s):" + str(or_ports))
in my script (derived mostly from your example /usr/share/doc/stem-1.8.0/_static/example/relay_connections.py , my is here: https://github.com/toralf/torutils/blob/master/info.py)
https://github.com/atx/prometheus-tor_exporter uses this module and using a valid IPv6 address returns:
$ sudo /usr/bin/prometheus-tor-exporter.py -m tcp -a '2a01:XXXX:XXXX:XXX::XX'
Traceback (most recent call last):
File "/usr/bin/prometheus-tor-exporter.py", line 197, in <module>
port=args.control_port)
File "/usr/lib/python3/dist-packages/stem/control.py", line 1014, in from_port
raise ValueError('Invalid IP address: %s' % address)
ValueError: Invalid IP address: 2a01:XXXX:XXXX:XXX::XX
with open('cached-microdesc-consensus', 'rb') as consensus_file:
network_status = parse_file(consensus_file,
'network-status-microdesc-consensus-3 1.0',
document_handler=DocumentHandler.DOCUMENT,
validate=True)So here is parse_file always return generator. In my opinion will be better if we parse as single document return object but not generator.
Or you mean it is possible to load several documents from one file?
Migrated from ticket 31234.
When Tor is running as a daemon, its stderr is only used for real errors that we should probably be paying attention to. But there are still cases (see #52 (comment)) where we suppress Tor's stderr logging. This made it hard to diagnose https://trac.torproject.org/projects/tor/ticket/33006 .
I'll write a patch when I can.
I'm trying to use stem to control tor as described here, but instead of privoxy, I'm using the built-in HTTP tunnel. This is my tor config:
ControlPort 9051
HTTPTunnelPort 8118
It works locally, but not on the server, i.e. when I run my Python program on a server machine where tor is running as a background service. When I raise logging level, I get a few
[stem] INFO: Error while receiving a control message (SocketClosed): empty socket content
and then it all fails with an exception
stem.ProtocolError: SIGNAL response contained unrecognized status code: 514
Could someone please explain me what could be wrong or help me to debug this? I don't even know where I should look. I raised the logging level of tor itself, but couldn't find anything very interesting there. Attaching the log just in case someone could understand what's going on.
For now it's list of strings:
['Authority', 'BadExit', 'Exit', 'Fast', 'Guard', 'HSDir', 'NoEdConsensus', 'Running', 'Stable', 'StaleDesc', 'V2Dir', 'Valid']
Migrated from ticket 8780.
Stem's site presently has a two level navigational menu based on this.
The tutorial warns that adding additional levels can be tricky but having an additional level or two would greatly improve the navigability of our API. With it we could do something like...
API
|- Control
| |- stem.control
| |- stem.connection
| |- stem.socket
| |- stem.process
| +- stem.response
|
|- Types
| |- stem.exit_policy
| +- stem.version
... etc
... and with another level we might also be able to include methods/functions.
With Python 3.9.0a3, the tests fail to run with this error:
+ /usr/bin/python3 run_tests.py --unit
======================================================================
INITIALISING
======================================================================
stem version... 1.8.0
python version... 3.9.0a3
operating system... Linux
cryptography version... missing
mock version... failed
module 'unittest.mock' has no attribute '__version__'
It looks like __version__ was removed:
python/cpython#17977
Originally reported downstream in Fedora:
https://bugzilla.redhat.com/show_bug.cgi?id=1797690
Once we fix Tor's Travis make test-stem timeouts, we should update Stem's Travis CI config, based on Chutney's Travis CI config.
Chutney's config provides a combination of OSes, Tor versions, Python versions, and PyPy versions. (We won't need the chutney-specific network tests.)
Here's the relevant Tor ticket:
https://trac.torproject.org/projects/tor/ticket/29437
I'm happy to do this task, once we get Tor/Stem working in Travis CI again.
When running stem's tests on commit 99c3043, we see the following error in tor's CI:
Traceback (most recent call last):
File "/home/travis/build/torproject/tor/stem/run_tests.py", line 482, in
main()
File "/home/travis/build/torproject/tor/stem/run_tests.py", line 197, in main
args = test.arguments.Arguments.parse(sys.argv[1:])
File "/home/travis/build/torproject/tor/stem/test/arguments.py", line 109, in parse
args['specific_test'].append(crop_module_name(arg))
KeyError: 'specific_test'
https://travis-ci.org/github/torproject/tor/jobs/686422295#L3615
The previous commit d1174a8 does not appear to have this issue.
Edit: specify the correct working commit.
This is a independent repro of a problem related to #60 but was not tested previously.
Using the current master, when caching descriptor files, CollecTor fails to validate the cached files:
Here is my repro code
import os
import sys
import datetime
import stem.descriptor.collector as sd_collector
fdir = '/tmp/stem_bg60' # I independently made this directory
def main():
print(f'{fdir} contents')
print(os.listdir(fdir))
start = datetime.datetime(year=2020, month=8, day=1)
end = datetime.datetime(year=2020, month=8, day=3)
genr = sd_collector.get_server_descriptors(
start=start,
end=end,
cache_to=fdir,
timeout=600
)
recs = 0
for rec in genr:
recs = recs + 1
print(f'got {recs} recs')
print('running again with cache dir')
print(f'{fdir} contents')
print(os.listdir(fdir))
genr = sd_collector.get_server_descriptors(
start=start,
end=end,
cache_to=fdir,
timeout=600
)
recs = 0
for rec in genr:
recs = recs + 1
print(f'got {recs} recs')
return 0
if __name__ == '__main__':
sys.exit(main())
This fails as soon as I try to use the cached data with the following errors:
(stem379) epiphyte@vertex05:~/git/stem$ python bg60.py
/tmp/stem_bg60 contents
[]
got 24465 recs
running again with cache dir
/tmp/stem_bg60 contents
['server-descriptors-2020-08.tar']
Traceback (most recent call last):
File "bg60.py", line 52, in <module>
sys.exit(main())
File "bg60.py", line 44, in main
for rec in genr:
File "/home/epiphyte/git/stem/stem/descriptor/collector.py", line 103, in get_server_descriptors
for desc in get_instance().get_server_descriptors(start, end, cache_to, bridge, timeout, retries):
File "/home/epiphyte/git/stem/stem/descriptor/collector.py", line 434, in get_server_descriptors
for desc in f.read(cache_to, desc_type, start, end, timeout = timeout, retries = retries):
File "/home/epiphyte/git/stem/stem/descriptor/collector.py", line 273, in read
path = self.download(directory, True, timeout, retries)
File "/home/epiphyte/git/stem/stem/descriptor/collector.py", line 335, in download
raise OSError("%s already exists but mismatches CollecTor's checksum (expected: %s, actual: %s)" % (path, expected_hash, actual_hash))
OSError: /tmp/stem_bg60/server-descriptors-2020-08.tar already exists but mismatches CollecTor's checksum (expected: 5f5c62fa5691d520017ef107c1d6ea4f29af2e5aabf959373da31755c30d21d8, actual: 352b10fae3e221fb3287d8e1dfd754eb43f3058d94ee8940d090f34971b01f70)
Running it again fails right away
(stem379) epiphyte@vertex05:~/git/stem$ python bg60.py
/tmp/stem_bg60 contents
['server-descriptors-2020-08.tar']
Traceback (most recent call last):
File "bg60.py", line 51, in <module>
sys.exit(main())
File "bg60.py", line 25, in main
for rec in genr:
File "/home/epiphyte/git/stem/stem/descriptor/collector.py", line 103, in get_server_descriptors
for desc in get_instance().get_server_descriptors(start, end, cache_to, bridge, timeout, retries):
File "/home/epiphyte/git/stem/stem/descriptor/collector.py", line 434, in get_server_descriptors
for desc in f.read(cache_to, desc_type, start, end, timeout = timeout, retries = retries):
File "/home/epiphyte/git/stem/stem/descriptor/collector.py", line 273, in read
path = self.download(directory, True, timeout, retries)
File "/home/epiphyte/git/stem/stem/descriptor/collector.py", line 335, in download
raise OSError("%s already exists but mismatches CollecTor's checksum (expected: %s, actual: %s)" % (path, expected_hash, actual_hash))
OSError: /tmp/stem_bg60/server-descriptors-2020-08.tar already exists but mismatches CollecTor's checksum (expected: 5f5c62fa5691d520017ef107c1d6ea4f29af2e5aabf959373da31755c30d21d8, actual: 352b10fae3e221fb3287d8e1dfd754eb43f3058d94ee8940d090f34971b01f70)
And environment information (using ubuntu 18.04)
(stem379) epiphyte@vertex05:~/git/stem$ git rev-parse HEAD
ab835c1a2972a654af991d9690776b755a9450c1
(stem379) epiphyte@vertex05:~/git/stem$ python --version
Python 3.7.9
(stem379) epiphyte@vertex05:~/git/stem$ python -m pip freeze
appdirs==1.4.4
cffi==1.14.3
cryptography==3.1.1
distlib==0.3.1
filelock==3.0.12
importlib-metadata==1.7.0
mock==4.0.2
packaging==20.4
pluggy==0.13.1
py==1.9.0
pycodestyle==2.6.0
pycparser==2.20
pyflakes==2.2.0
pyparsing==2.4.7
six==1.15.0
toml==0.10.1
tox==3.20.0
virtualenv==20.0.31
zipp==3.2.0
(stem379) epiphyte@vertex05:~/git/stem$ uname -a
Linux vertex05 4.15.0-117-generic #118-Ubuntu SMP Fri Sep 4 20:02:41 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
For now it's list of strings:
flags = ['Fast', 'Guard', 'HSDir', 'Running', 'Stable', 'V2Dir', 'Valid']Better to use enum.
Tor git master just merged its support to automatically have an IPv6 ORPort by default. Which means that ORPort 9001 for instance will both open a listener on 0.0.0.0 and [::] leading in this error with test_get_listeners and test_get_ports:
https://travis-ci.org/github/torproject/tor/jobs/710747893#L3773
Second list contains 1 additional elements.
First extra element 1:
('::', 1113)
- [('0.0.0.0', 1113)]
+ [('0.0.0.0', 1113), ('::', 1113)]
This https://github.com/toralf/torutils/blob/f307866d2149bcd3c95df5269b3b48d051f871e9/info.py gave today at relay:
0.4.2.5 10 days, 17:01:05 Fast Guard HSDir Running Stable V2Dir Valid
resolver=proc pid=6069 conns=8336
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/stem/descriptor/__init__.py", line 236, in decompress
return self._decompression_func(self._module, content)
File "/usr/lib64/python3.6/site-packages/stem/descriptor/__init__.py", line 257, in <lambda>
('LZMA', _Compression('lzma', 'lzma', 'x-tor-lzma', '.xz', lambda module, content: module.decompress(content))),
File "/usr/lib64/python3.6/lzma.py", line 342, in decompress
raise LZMAError("Compressed data ended before the "
_lzma.LZMAError: Compressed data ended before the end-of-stream marker was reached
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/torutils/info.py", line 232, in <module>
main()
File "/opt/torutils/info.py", line 127, in main
for s in stem.descriptor.collector.get_server_descriptors(start = back):
File "/usr/lib64/python3.6/site-packages/stem/descriptor/collector.py", line 103, in get_server_descriptors
for desc in get_instance().get_server_descriptors(start, end, cache_to, bridge, timeout, retries):
File "/usr/lib64/python3.6/site-packages/stem/descriptor/collector.py", line 439, in get_server_descriptors
for f in self.files(desc_type, start, end):
File "/usr/lib64/python3.6/site-packages/stem/descriptor/collector.py", line 674, in files
self._cached_files = sorted(CollecTor._files(self.index(), []), key = lambda x: x.start if x.start else FUTURE)
File "/usr/lib64/python3.6/site-packages/stem/descriptor/collector.py", line 648, in index
response = compression.decompress(stem.util.connection.download(url, self.timeout, self.retries))
File "/usr/lib64/python3.6/site-packages/stem/descriptor/__init__.py", line 238, in decompress
raise IOError('Failed to decompress as %s: %s' % (self, exc))
OSError: Failed to decompress as lzma: Compressed data ended before the end-of-stream marker was reached
Migrated from ticket 30596.
python version... 3.8.0a3+
operating system... failed
module 'platform' has no attribute 'linux_distribution'
The command "./run_tests.py --unit" exited with 1.
Migrated from ticket 8248.
We presently have a get_server_descriptor() and get_network_status() method, but it's horribly inefficient. Our present messaging infrastructure provided by the BaseController class sends or receives complete messages rather than streaming the message as it comes in. This means that we load the compete "GETINFO desc/all" response into memory before providing our caller with even the first descriptor. This eats up memory and it means an unnecessarily long latency before we provide results. Unfortunately this will take a bit of re-architecting in the BaseController to allow for streamed messages.
Migrated from ticket 20500.
At PyCon sounded like wheels (*.whl) are the new hotness when it comes to distributing python modules.
Don't think we want to jump on this right away for a few reasons...
That said, if this becomes the standard then once it's universally supported we should obviously swap.
Hi,
The development button on the navigation bar of this page (https://stem.torproject.org/index.html) links to the old trac page, fyi.
Migrated from ticket 32719.
Stem has a few references to tor's pre-0.3.5 directory structure in its documentation. We should update them based on the new subdirectories in 0.3.5 and later.
We're seeing some hangs in the stem unit tests in tor's CI:
https://travis-ci.org/github/teor2345/tor/jobs/681198159#L3671
They aren't happening very often right now, so this issue is currently a low priority. (But our experience of CI is that intermittent issues can suddenly happen all the time, due to code or infrastructure changes.)
These hangs could be caused by a tor bug, a stem bug, a race condition, or a Travis CI infrastructure issue.
I can't see any obvious bugs here, but if you find a stem or tor issue, please let us know.
To help us diagnose these kinds of issues, we'd like stem's blocking and busy-waiting operations to have a timeout. Travis CI times out after 10 minutes, and we terminate stem after 9 minutes. So any timeout should be less than 8 minutes. (Or lower, if we need to allow more test setup and run time.)
If stem should block indefinitely, and only timeout during tests, we're happy to set a command-line argument or environmental variable.
We can add timeouts as needed, we don't need to fix everything in one go.
So let's focus on these two timeouts in this ticket:
queue.get(), in response to the timelimit warning USR1 signal at 9 minutes:
Running tests...
control.base_controller... success (1.51s)
control.controller... timelimit: sending warning signal 10
...
File "/home/travis/build/teor2345/tor/stem/test/integ/control/controller.py", line 1468, in test_transition_to_relay
controller.set_options({'OrPort': '9090', 'DisableNetwork': '1'})
File "/home/travis/build/teor2345/tor/stem/stem/control.py", line 2413, in set_options
response = self.msg(query)
File "/home/travis/build/teor2345/tor/stem/stem/control.py", line 644, in msg
response = self._reply_queue.get()
File "/usr/lib/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File "/usr/lib/python3.6/threading.py", line 295, in wait
waiter.acquire()
...
Looping time.sleep(0.1), in response to the timelimit termination ABRT signal 30 seconds later:
File "/home/travis/build/teor2345/tor/stem/test/integ/installation.py", line 64, in run_tests
stem.util.test_tools.ASYNC_TESTS['test.integ.installation.test_sdist'].run(test_install.pid())
File "/home/travis/build/teor2345/tor/stem/stem/util/test_tools.py", line 187, in run
self._process.start()
File "/usr/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
...
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/travis/build/teor2345/tor/stem/stem/util/test_tools.py", line 155, in _wrapper
runner(*args) if args else runner()
File "/home/travis/build/teor2345/tor/stem/test/integ/installation.py", line 135, in test_sdist
time.sleep(0.1) # we need to run these tests serially
...
I'm cc'ing @ahf on this issue, because I'm transitioning CI responsibilities to him.
Tor 0.4.5.0-alpha (yet unreleased) and later are going to support a new DROPTIMEOUTS command:
https://gitlab.torproject.org/tpo/core/tor/-/issues/40002
https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n2005
That new command should be quite similar to DROPGUARDS from Stem's perspective.
Is there a way to execute the new DROPTIMEOUTS command from Stem? If not, can you add a drop_timeouts function to Controller?
Thanks in advance!
Migrated from ticket 30257.
In #30234, we got the tor logs, but the USR1 and ABRT signals sent by timelimit to test_stem.py aren't being propagated to tor:
Apr 22 03:32:30.000 [notice] Monitored process 20402 is dead.
Apr 22 03:32:30.000 [notice] Owning controller process has vanished -- exiting now.
Apr 22 03:32:30.000 [notice] Catching signal TERM, exiting cleanly.
https://travis-ci.org/teor2345/tor/jobs/522893523#L4944
We need to work out how to get the signals from this stem test process down to the tor it launches:
================================================================================
Signal SIGABRT received by thread MainThread in process 20402
--------------------------------------------------------------------------------
Event notifier thread stacktrace
File "/usr/lib/python3.4/threading.py", line 888, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
self.run()
File "/usr/lib/python3.4/threading.py", line 868, in run
self._target(*self._args, **self._kwargs)
File "/home/travis/build/teor2345/tor/stem/stem/control.py", line 984, in _event_loop
self._event_notice.wait(0.05)
File "/usr/lib/python3.4/threading.py", line 553, in wait
signaled = self._cond.wait(timeout)
File "/usr/lib/python3.4/threading.py", line 294, in wait
gotit = waiter.acquire(True, timeout)
--------------------------------------------------------------------------------
MainThread thread stacktrace
File "/home/travis/build/teor2345/tor/stem/run_tests.py", line 451, in <module>
main()
File "/home/travis/build/teor2345/tor/stem/run_tests.py", line 287, in main
integ_runner.start(target, args.attribute_targets, args.tor_path)
File "/home/travis/build/teor2345/tor/stem/test/runner.py", line 262, in start
self._owner_controller = self.get_tor_controller(True)
File "/home/travis/build/teor2345/tor/stem/test/runner.py", line 482, in get_tor_controller
controller.authenticate(password = CONTROL_PASSWORD, chroot_path = self.get_chroot())
File "/home/travis/build/teor2345/tor/stem/stem/control.py", line 1103, in authenticate
stem.connection.authenticate(self, *args, **kwargs)
File "/home/travis/build/teor2345/tor/stem/stem/connection.py", line 530, in authenticate
protocolinfo_response = get_protocolinfo(controller)
File "/home/travis/build/teor2345/tor/stem/stem/connection.py", line 1007, in get_protocolinfo
protocolinfo_response = _msg(controller, 'PROTOCOLINFO 1')
File "/home/travis/build/teor2345/tor/stem/stem/connection.py", line 1036, in _msg
return controller.msg(message)
File "/home/travis/build/teor2345/tor/stem/stem/control.py", line 654, in msg
response = self._reply_queue.get()
File "/usr/lib/python3.4/queue.py", line 167, in get
self.not_empty.wait()
File "/usr/lib/python3.4/threading.py", line 290, in wait
waiter.acquire()
File "/home/travis/build/teor2345/tor/stem/run_tests.py", line 98, in log_traceback
for thread_name, stacktrace in test.output.thread_stacktraces().items():
File "/home/travis/build/teor2345/tor/stem/test/output.py", line 110, in thread_stacktraces
stacktraces[thread.name] = ''.join(traceback.format_stack(frame))
--------------------------------------------------------------------------------
Tor listener thread stacktrace
File "/usr/lib/python3.4/threading.py", line 888, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
self.run()
File "/usr/lib/python3.4/threading.py", line 868, in run
self._target(*self._args, **self._kwargs)
File "/home/travis/build/teor2345/tor/stem/stem/control.py", line 939, in _reader_loop
control_message = self._socket.recv()
File "/home/travis/build/teor2345/tor/stem/stem/socket.py", line 474, in recv
return self._recv(lambda s, sf: recv_message(sf))
File "/home/travis/build/teor2345/tor/stem/stem/socket.py", line 274, in _recv
return handler(my_socket, my_socket_file)
File "/home/travis/build/teor2345/tor/stem/stem/socket.py", line 474, in <lambda>
return self._recv(lambda s, sf: recv_message(sf))
File "/home/travis/build/teor2345/tor/stem/stem/socket.py", line 676, in recv_message
line = control_file.readline()
File "/usr/lib/python3.4/socket.py", line 374, in readinto
return self._sock.recv_into(b)
================================================================================
Hi,
I couldn't find any ticket or work on this so thought I'd open a ticket to track it.
Now that Client Authentication has landed for v3 and has been popularized via Tor Browser 9.5's release, the OnionShare team is being regularly asked why we don't support it yet.
As far as I can tell, since we use ephemeral onions exclusively via Stem, we would need Stem to support it. By the looks of it, only v2 auth is supported at this time via the basic_auth parameter to create_ephemeral_onion_service().
This commit to the Tor core project added Control Port support for adding v3 Client Auth via the ONION_CLIENT_AUTH_ADD command torproject/tor@00fdaae
Will gladly help if I can, but not sure exactly how to go about it in Stem. I can see you call ADD_ONION with the BasicAuth flag, I'm not sure if we have to call ONION_CLIENT_AUTH_ADD after we've called ADD_ONION, meaning we have already published the descriptor? I'm guessing so since Tor validates the v3 address as part of that function torproject/tor@00fdaae#diff-20089616ed928fbc29bd570b740c0189R100
Did you have any specific ideas on how you wanted to go about the implementation?
Hi again,
There are two issues with this page https://stem.torproject.org/api/descriptor/microdescriptor.html
Firstly, the ticket #7953 seems to be already closed. I think "To do so you need to match the microdescriptor's digest against its corresponding router status entry. For added fun as of this writing the controller doesn't even surface those router status entries (ticket 7953)." part mentioned in the description is already implemented.
Secondly, the first example on that page doesn't work anymore since version 1.8.0 (as @atagar mentioned in the changelog: Replaced our digest attribute with a much more flexible digest() method. Unfortunately I cannot do this in a backward compatible way because of the name conflict.)
Here is the updated version of the example that works with the latest Stem:
import os
from stem.control import Controller
from stem.descriptor import parse_file
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
exit_digests = set()
data_dir = controller.get_conf('DataDirectory')
for desc in controller.get_microdescriptors():
if desc.exit_policy.is_exiting_allowed():
exit_digests.add(desc.digest())
print 'Exit Relays:'
for desc in parse_file(os.path.join(data_dir, 'cached-microdesc-consensus')):
if desc.microdescriptor_digest in exit_digests:
print ' %s (%s)' % (desc.nickname, desc.fingerprint)
Thank you for creating this library, it is super useful!
Migrated from ticket 31551.
I do get :
# for p in 9051 29051 ; do python /usr/share/doc/stem-1.7.1/_static/example/relay_connections.py --ctrlport $p --resolver ss; done
0.4.1.5 uptime: 3-01:35:39 flags: Fast, Running, Stable, V2Dir, Valid
+------------------------------+------+------+
| Type | IPv4 | IPv6 |
+------------------------------+------+------+
| Inbound to our ORPort | 0 | 1 |
+------------------------------+------+------+
| Total | 0 | 1 |
+------------------------------+------+------+
0.4.1.5 uptime: 3-01:36:37 flags: Fast, Running, Stable, V2Dir, Valid
Traceback (most recent call last):
File "/usr/share/doc/stem-1.7.1/_static/example/relay_connections.py", line 130, in <module>
main()
File "/usr/share/doc/stem-1.7.1/_static/example/relay_connections.py", line 66, in main
for conn in get_connections(resolver = args.resolver, process_pid = pid):
File "/usr/lib64/python3.6/site-packages/stem/util/connection.py", line 300, in get_connections
raise IOError('No results found using: %s' % resolver_command)
OSError: No results found using: ss -nptu
at a hardened stable Gentoo Linux with kernel 5.2.10 and sys-apps/iproute2-4.19.0-r1 (provides "ss"), but th ecomamdn itself prints :
mr-fox ~ # ss -nptu | head
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
tcp ESTAB 0 0 5.9.158.75:39658 51.77.52.216:443 users:(("tor",pid=2365,fd=4313))
tcp ESTAB 0 0 5.9.158.75:9001 52.14.166.220:51466 users:(("tor",pid=2397,fd=5105))
tcp ESTAB 0 1057 5.9.158.75:46379 158.174.255.235:4430 users:(("tor",pid=2365,fd=4663))
tcp ESTAB 0 0 5.9.158.75:443 82.35.159.254:57674 users:(("tor",pid=2365,fd=3870))
tcp ESTAB 0 0 5.9.158.75:35005 94.16.113.67:9001 users:(("tor",pid=2365,fd=4975))
tcp ESTAB 0 0 5.9.158.75:36837 88.3.139.225:48998 users:(("tor",pid=2365,fd=6473))
tcp ESTAB 0 0 5.9.158.75:44183 82.118.21.60:9001 users:(("tor",pid=2365,fd=720))
tcp ESTAB 0 0 5.9.158.75:36997 50.7.115.67:443 users:(("tor",pid=2397,fd=1543))
tcp ESTAB 1571 1057 5.9.158.75:9001 46.165.250.224:34639 users:(("tor",pid=2397,fd=2352))
All other 3 resolvers are fine and do print something like
# for p in 9051 29051 ; do python /usr/share/doc/stem-1.7.1/_static/example/relay_connections.py --ctrlport $p; done
0.4.1.5 uptime: 3-01:51:38 flags: Fast, Running, Stable, V2Dir, Valid
+------------------------------+------+------+
| Type | IPv4 | IPv6 |
+------------------------------+------+------+
| Inbound to our ORPort | 2267 | 1 |
| Inbound to our DirPort | 3 | 0 |
| Inbound to our ControlPort | 1 | 0 |
| Outbound to a relay | 3965 | 0 |
| Outbound uncategorized | 12 | 0 |
+------------------------------+------+------+
| Total | 6248 | 1 |
+------------------------------+------+------+
0.4.1.5 uptime: 3-01:51:40 flags: Fast, Running, Stable, V2Dir, Valid
+------------------------------+------+------+
| Type | IPv4 | IPv6 |
+------------------------------+------+------+
| Inbound to our ORPort | 1629 | 2 |
| Inbound to our ControlPort | 1 | 0 |
| Outbound to a relay | 4083 | 0 |
| Outbound uncategorized | 12 | 0 |
+------------------------------+------+------+
| Total | 5725 | 2 |
+------------------------------+------+------+
Migrated from ticket 30595.
doctest... success (0.08s)
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated
Migrated from ticket 7632.
Presently we're including a summary for modules at the top of their page. These summaries include classes, methods, and show their inheritance/membership. For instance this.
This is fine and well, but I'm doing it via a preformatted text block to take advantage of the monospaced text to show membership. This is a hack, and doing it this way means that we can't include links to the class/method documentation.
We should check to see if there's a better way of doing module overviews, maybe asking on the sphinx list.
Migrated from ticket 15999.
While adding support for ephemeral hidden services I tried to add a end-to-end integ test without success. Ideally the test would create a new hidden service, then use tor to retrieve its content.
Really shouldn't be a big whoop. We have a tutorial for something similar after all. But for some reason even the tutorial wasn't working for me (hidden service was inaccessible)...
@require_online
@require_controller
@require_version(Requirement.ADD_ONION)
def test_using_ephemeral_hidden_services(self):
"""
Create and use a live ephemeral hidden service.
"""
# TODO: Not having success getting... well, just about any damn hidden
# serivce working. Even our prior tutorial is failing right now. >:(
return
with test.runner.get_runner().get_tor_controller() as controller:
incoming_address = None, None
def run_server():
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind(('localhost', 4567))
serversocket.listen(5)
incoming_socket, incoming_address = serversocket.accept()
incoming_socket.write('hello world')
serversocket.shutdown(socket.SHUT_RDWR)
server_thread = threading.Thread(target = run_server)
server_thread.setDaemon(True)
server_thread.start()
response = controller.create_ephemeral_hidden_service({80: 4567})
with test.network.Socks(controller.get_socks_listeners()[0]) as s:
s.settimeout(30)
s.connect(('%s.onion' % response.service_id, 80))
print s.read()
self.assertTrue(incoming_address is not None)
server_thread.join()
rj76 mentioned having a branch that might cover this.
Migrated from ticket 32049.
Running my ps.py for about 30 minutes at a Tor relay gives
port # opened closed max ( proc:9051, 6630 conns 0.24 sec )
5222 62 62 3 1 (Jabber)
5280 1 1 (None)
6660 4 2 (IRC)
6666 1 1 1 (IRC)
6667 1 3 1 1 (IRC)
6697 3 3 (IRC)
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/stem/util/proc.py", line 448, in _inodes_for_sockets
fd_name = os.readlink(fd_path)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/2284/fd/5280'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/torutils/ps.py", line 178, in <module>
main()
File "/opt/torutils/ps.py", line 102, in main
connections = get_connections(resolver=resolver,process_pid=pid,process_name='tor')
File "/usr/lib64/python3.6/site-packages/stem/util/connection.py", line 236, in get_connections
return stem.util.proc.connections(pid = process_pid)
File "/usr/lib64/python3.6/site-packages/stem/util/proc.py", line 374, in connections
inodes = _inodes_for_sockets(pid) if pid else set()
File "/usr/lib64/python3.6/site-packages/stem/util/proc.py", line 457, in _inodes_for_sockets
raise IOError('unable to determine file descriptor destination (%s): %s' % (exc, fd_path))
OSError: unable to determine file descriptor destination ([Errno 2] No such file or directory: '/proc/2284/fd/5280'): /proc/2284/fd/5280
FWIW:
mr-fox ~ # ls /proc/2284/fd | wc
7148 7148 34630
mr-fox ~ # for p in 9051 ; do python /usr/share/doc/stem-1.7.1/_static/example/relay_connections.py --ctrlport $p; done
0.4.2.2-alpha uptime: 1-16:08:14 flags: Fast, Guard, Running, Stable, V2Dir, Valid
+------------------------------+------+------+
| Type | IPv4 | IPv6 |
+------------------------------+------+------+
| Inbound to our ORPort | 3661 | 4 |
| Inbound to our DirPort | 2 | 0 |
| Inbound to our ControlPort | 1 | 0 |
| Outbound to a relay | 2889 | 0 |
| Outbound exit traffic | 63 | 6 |
| Outbound uncategorized | 8 | 0 |
+------------------------------+------+------+
| Total | 6624 | 10 |
+------------------------------+------+------+
+------------------------------+------+------+
| Exit Port | IPv4 | IPv6 |
+------------------------------+------+------+
| 5222 (Jabber) | 54 | 5 |
| 5280 | 1 | 0 |
| 6667 (IRC) | 6 | 0 |
| 6697 (IRC) | 2 | 1 |
+------------------------------+------+------+
| Total | 63 | 6 |
+------------------------------+------+------+
Stem version: 1.8.0
Python version: 3.7.3
Tor: 0.4.2.6 (git-971a6beff5a53434) running on Windows 8 [or later] with Libevent 2.1.8-stable, OpenSSL 1.1.1d, Zlib 1.2.11, Liblzma N/A, and Libzstd N/A.
Traceback (most recent call last):
File "C:\Drive-E\Web\Today\ZeroNet-win-dist-win64 árvíztűrő tükörfúrógép\ZeroNet-win-dist-win64\test2.py", line 20, in <module>
controller.authenticate()
File "C:\Python3\lib\site-packages\stem\control.py", line 1112, in authenticate
stem.connection.authenticate(self, *args, **kwargs)
File "C:\Python3\lib\site-packages\stem\connection.py", line 534, in authenticate
protocolinfo_response = get_protocolinfo(controller)
File "C:\Python3\lib\site-packages\stem\connection.py", line 1029, in get_protocolinfo
stem.response.convert('PROTOCOLINFO', protocolinfo_response)
File "C:\Python3\lib\site-packages\stem\response\__init__.py", line 124, in convert
message._parse_message(**kwargs)
File "C:\Python3\lib\site-packages\stem\response\protocolinfo.py", line 110, in _parse_message
self.cookie_path = line.pop_mapping(True, True, get_bytes = True)[1].decode(sys.getfilesystemencoding())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 44: invalid continuation byte
Starting tor.exe:
@ c:\Drive-E\Web\Today\ZeroNet-win-dist-win64 árvíztűrő tükörfúrógép\ZeroNet-win-dist-win64\core\tools\tor
$ tor.exe -f torrc --defaults-torrc torrc-defaults --ignore-missing-torrc
torrc-defaults file:
DataDirectory data
HiddenServiceStatistics 0
GeoIPFile geoip\geoip
GeoIPv6File geoip\geoip6
ControlPort 49051
SOCKSPort 49050
CookieAuthentication 1
Script:
import stem
import stem.connection
from stem.control import Controller
controller = Controller.from_port(port=49051)
controller.authenticate()PROTOCOLINFO result:
import socket
conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
conn.connect(("127.0.0.1", 49051))
conn.sendall(b"PROTOCOLINFO\r\n")
print(conn.recv(1024 * 64).decode("utf8"))250-PROTOCOLINFO 1
250-AUTH METHODS=COOKIE,SAFECOOKIE COOKIEFILE="c:\\Drive-E\\Web\\Today\\ZeroNet-win-dist-win64 \341rv\355zt\373r\365 t\374k\366rf\372r\363g\351p\\ZeroNet-win-dist-win64\\core\\tools\\tor\\data\\control_auth_cookie"
250-VERSION Tor="0.4.2.6"
250 OK
Migrated from ticket 19679.
To make things manageable Stem's FAQ starts with a header linking to the questions. A nicer approach would be for us to only show the questions, with the answers shown when users click on them.
Not hard and no doubt there's lots of examples we can pull from. One note is that our anchor links need to continue to work. That is to say: providing a link to question X should start with it expanded.
Hello,
The following tests are failing in CentOS 8. They pass fine in Fedora.
Let me know if you need any other information.
+ /usr/bin/python3.6 run_tests.py --unit
======================================================================
INITIALISING
======================================================================
stem version... 1.8.0
python version... 3.6.8
operating system... Linux (Red Hat Enterprise Linux 8.2)
cryptography version... 2.3
mock version... 1.0
pyflakes version... 2.1.1
pycodestyle version... 2.5.0
checking for orphaned .pyc files... done (0.0s)
checking for unused tests... done (0.0s)
importing test modules... done (0.8s)
running pyflakes... running
running pycodestyle... running
======================================================================
UNIT TESTS
======================================================================
util.enum... success (0.00s)
util.connection... success (0.07s)
util.conf... success (0.00s)
util.log... success (0.00s)
util.proc... failed (0.08s)
test_connections 1 ms [SUCCESS]
test_connections_ipv6 [FAILURE]
test_connections_ipv6_by_user [FAILURE]
test_cwd 0 ms [SUCCESS]
test_file_descriptors_used 0 ms [SUCCESS]
test_high_connection_count [FAILURE]
test_memory_usage 0 ms [SUCCESS]
test_physical_memory 0 ms [SUCCESS]
test_stats 20 ms [SUCCESS]
test_system_start_time 0 ms [SUCCESS]
test_uid 0 ms [SUCCESS]
======================================================================
FAIL: test_connections_ipv6
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/mock.py", line 1183, in patched
return func(*args, **keywargs)
File "/builddir/build/BUILD/stem-1.8.0/test/unit/util/proc.py", line 277, in test_connections_ipv6
self.assertEqual(expected, proc.connections(pid = pid))
AssertionError: Lists differ: [Conn[17 chars]ess='2a01:04f8:0190:514a:0000:0000:0000:0002',[311 chars]rue)] != [Conn[17 chars]ess='f804:012a:4a51:9001:0000:0000:0200:0000',[311 chars]rue)]
First differing element 0:
Conne[16 chars]ess='2a01:04f8:0190:514a:0000:0000:0000:0002',[118 chars]True)
Conne[16 chars]ess='f804:012a:4a51:9001:0000:0000:0200:0000',[118 chars]True)
Diff is 1347 characters long. Set self.maxDiff to None to see it.
======================================================================
FAIL: test_connections_ipv6_by_user
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/mock.py", line 1183, in patched
return func(*args, **keywargs)
File "/builddir/build/BUILD/stem-1.8.0/test/unit/util/proc.py", line 306, in test_connections_ipv6_by_user
self.assertEqual(expected, proc.connections(user = 'me'))
AssertionError: Lists differ: [Conn[37 chars]0000:0000:ffff:0509:9e4b', local_port=5222, re[486 chars]rue)] != [Conn[37 chars]0000:ffff:0000:4b9e:0905', local_port=5222, re[486 chars]rue)]
First differing element 0:
Conne[36 chars]0000:0000:ffff:0509:9e4b', local_port=5222, re[99 chars]True)
Conne[36 chars]0000:ffff:0000:4b9e:0905', local_port=5222, re[99 chars]True)
Diff is 2042 characters long. Set self.maxDiff to None to see it.
======================================================================
FAIL: test_high_connection_count
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/mock.py", line 1183, in patched
return func(*args, **keywargs)
File "/builddir/build/BUILD/stem-1.8.0/test/unit/util/proc.py", line 348, in test_high_connection_count
self.assertEqual(expected, proc.connections(pid))
AssertionError: Lists differ: [Conn[64 chars]ess='51.34.34.34', remote_port=8772, protocol=[294 chars]lse)] != [Conn[64 chars]ess='34.34.34.51', remote_port=8772, protocol=[294 chars]lse)]
First differing element 0:
Conne[63 chars]ess='51.34.34.34', remote_port=8772, protocol=[16 chars]alse)
Conne[63 chars]ess='34.34.34.51', remote_port=8772, protocol=[16 chars]alse)
Diff is 1358 characters long. Set self.maxDiff to None to see it.
----------------------------------------------------------------------
Ran 11 tests in 0.080s
FAILED (failures=3)
util.str_tools... success (0.00s)
util.system... success (0.01s)
util.term... success (0.00s)
util.tor_tools... success (0.00s)
util.__init__... success (0.00s)
installation... success (0.00s)
descriptor.descriptor... success (0.00s)
descriptor.compression... success (0.00s)
descriptor.export... success (0.00s)
descriptor.reader... success (0.04s)
descriptor.collector... success (0.70s)
descriptor.remote... success (0.18s)
descriptor.server_descriptor... success (0.06s)
descriptor.extrainfo_descriptor... success (0.05s)
descriptor.microdescriptor... success (0.00s)
descriptor.router_status_entry... success (0.02s)
descriptor.tordnsel... success (0.00s)
descriptor.networkstatus.detached_signature... success (0.00s)
descriptor.networkstatus.directory_authority... success (0.01s)
descriptor.networkstatus.key_certificate... success (0.01s)
descriptor.networkstatus.document_v2... success (0.00s)
descriptor.networkstatus.document_v3... success (0.08s)
descriptor.networkstatus.bridge_document... success (0.00s)
descriptor.hidden_service_v2... success (0.01s)
descriptor.hidden_service_v3... success (0.01s)
descriptor.certificate... success (0.00s)
descriptor.bandwidth_file... success (0.01s)
exit_policy.rule... success (0.01s)
exit_policy.policy... success (0.04s)
endpoint... success (0.00s)
version... success (0.00s)
manual... success (0.01s)
directory.authority... success (0.00s)
directory.fallback... success (0.02s)
tutorial... success (0.01s)
tutorial_examples... success (0.02s)
response.add_onion... success (0.00s)
response.control_message... success (0.00s)
response.control_line... success (0.00s)
response.events... success (0.02s)
response.getinfo... success (0.00s)
response.getconf... success (0.00s)
response.singleline... success (0.00s)
response.authchallenge... success (0.00s)
response.protocolinfo... success (0.00s)
response.mapaddress... success (0.00s)
client.size... success (0.00s)
client.address... success (0.00s)
client.link_protocol... success (0.00s)
client.certificate... success (0.00s)
client.link_specifier... success (0.00s)
client.kdf... success (0.00s)
client.cell... success (0.01s)
connection.authentication... success (0.05s)
connection.connect... success (0.00s)
control.controller... success (0.29s)
interpreter.arguments... success (0.01s)
interpreter.autocomplete... success (0.00s)
interpreter.help... success (0.00s)
interpreter.commands... success (0.01s)
doctest... success (0.06s)
STATIC CHECKS
* /builddir/build/BUILD/stem-1.8.0/stem/util/ed25519.py
line 59 - undefined name 'xrange' | range = xrange
TESTING FAILED (10 seconds)
[UNIT TEST] test_connections_ipv6 (test.unit.util.proc.TestProc) ... FAIL
[UNIT TEST] test_connections_ipv6_by_user (test.unit.util.proc.TestProc) ... FAIL
[UNIT TEST] test_high_connection_count (test.unit.util.proc.TestProc) ... FAIL
You can re-run just these tests with:
run_tests.py --unit --test util.proc
I tried using the code from exits_used.py, but I'm not getting any output. I removed any conditions, and it appears I don't get any Stream events at all. My torrc file only specifies the control port, and I'm using the same curl command provided with the script. I'm running Python 3.8.6 and stem==1.8.0. Any help would be appreciated, thanks!
Migrated from ticket 27479.
Integration test failure spotted by Nick...
======================================================================
ERROR: test_connections_by_ss
----------------------------------------------------------------------
Traceback (most recent call last):
File "stem/test/integ/util/connection.py", line 49, in test_connections_by_ss
self.check_resolver(Resolver.SS)
File "stem/test/require.py", line 58, in wrapped
return func(self, *args, **kwargs)
File "stem/test/integ/util/connection.py", line 28, in check_resolver
connections = get_connections(resolver, process_pid = runner.get_pid())
File "stem/stem/util/connection.py", line 300, in get_connections
raise IOError('No results found using: %s' % resolver_command)
OSError: No results found using: ss -nptu
----------------------------------------------------------------------
This test is disabled until it's fixed.
Migrated from ticket 10522.
Our client usage tutorial (To Russia With Love) is reportedly incompatible with certain SocksiPy versions.
The tutorial works with the standard SocksiPy Version 1.4.2 as well as Version 1.0 but not version 1.01...
Socks5:
[...]
File "/usr/lib64/python2.7/site-packages/socks.py", line 163, in sendall
if 'encode' in dir(bytes):
RuntimeError: maximum recursion depth exceeded while calling a Python object
Socks4:
[...]
File "/usr/lib64/python2.7/site-packages/socks.py", line 399, in connect
self.__negotiatesocks4(destpair[0],destpair[1])
File "/usr/lib64/python2.7/site-packages/socks.py", line 313, in __negotiatesocks4
req = "\x04\x01" + struct.pack(">H",destport).decode() + ipaddr
UnicodeDecodeError: 'ascii' codec can't decode byte 0xbb in position 1: ordinal not in range(128)
Migrated from ticket 22627.
Damn folks seem excited about asyncio. Seems to be a builtin intended to replace twisted starting with Python 3.4.
We should consider adding an AsyncController class that uses it, so we can integrate with folks who want a controller using that pattern.
Our Descriptor class' from_str function converts strings to descriptors. When used from a subclass we should (and are documented as) providing a descriptor of that type. However, this fails with...
from stem.descriptor.networkstatus import NetworkStatusDocumentV3
with open('/home/atagar/Desktop/stem/test/unit/descriptor/data/cached-consensus') as consensus_file:
consensus = NetworkStatusDocumentV3.from_str(consensus_file.read())
Traceback (most recent call last):
File "broken_demo.py", line 4, in <module>
consensus = NetworkStatusDocumentV3.from_str(consensus_file.read())
File "/home/atagar/Desktop/stem/stem/descriptor/__init__.py", line 870, in from_str
results = list(parse_file(io.BytesIO(stem.util.str_tools._to_bytes(content)), **kwargs))
File "/home/atagar/Desktop/stem/stem/descriptor/__init__.py", line 448, in parse_file
for desc in parse(descriptor_file): # type: ignore
File "/home/atagar/Desktop/stem/stem/descriptor/__init__.py", line 446, in parse
raise TypeError("Unable to determine the descriptor's type. filename: '%s', first line: '%s'" % (filename, stem.util.str_tools._to_unicode(first_line)))
TypeError: Unable to determine the descriptor's type. filename: '<undefined>', first line: 'network-status-version 3'
We only have from_str unit tests for server and extrainfo descriptors, which coincidentally work because our parse method can infer their type from the content. This ticket is for two things...
Migrated from ticket 15260.
Minor thing but our tutorials could look a little nicer. We use iconic images for the tutorial listing, but the tutorials themselves are often a wall of text. Might be worth experimenting to see if a title banner of some sort makes the pages more approachable.
Inside NetworkStatusDocumentV3 you have is_fresh method:
def is_fresh(self):
"""
Checks if the current time is between this document's **valid_after** and
**fresh_until** timestamps. To be fresh means this should be the latest
consensus.
.. versionadded:: 1.8.0
:returns: **True** if this consensus is presently fresh and **False**
otherwise
"""
return self.valid_after < datetime.datetime.utcnow() < self.fresh_untilis_reasonably_live like networkstatus_consensus_reasonably_live. It is used when tor client try use old consensus but not so old.< and > or <= and >=From torpy sources:
@property
def is_live(self):
# tor ref: networkstatus_is_live
return self.valid_after <= datetime.utcnow() <= self.valid_until
@property
def is_reasonably_live(self):
# tor ref: networkstatus_consensus_reasonably_live
return self.valid_after - timedelta(hours=24) <= datetime.utcnow() <= self.valid_until + timedelta(hours=24)When the exit policy accepts traffic to that port only to a subnet.
For instance, a relay with exit policy:
accept 133.0.0.0/8:443
>>> rs = controller.get_network_status("63BF46A63F9C21FD315CD061B3EAA3EB05283A0A")
>>> sd = controller.get_server_descriptor("63BF46A63F9C21FD315CD061B3EAA3EB05283A0A")
>>> rs.exit_policy.can_exit_to(port=443)
False
>>> sd.exit_policy.can_exit_to(port=443)
True
Why is can_exit_to returning different things?, should it be queried in a different way?, is it possible that this happens because sbws is not fetching microdescriptors.
Migrated from ticket 16348.
Python3 added exception chaining. Under python2 you only get the useful stacktrace but with python3 you get the internal one too.
At present there isn't a method for suppressing this. There's a proposal for adding suppression but it isn't yet merged, and even after it is this will be an ongoing annoyance for python 3.x before the addition.
To transparently support both synchronous and asynchronous usage we use metaprogrmming. Basically, our mixin matches method calls to the context type we run within.
When a synchronous controller is used within an event handler we mistakenly treat it as being asynchronous. For example...
from stem.control import Controller, EventType
with Controller.from_port() as controller:
controller.authenticate()
print(controller.get_version()) # this prints our version
def listener_func(event):
print(controller.get_version()) # this prints "<coroutine object Controller.get_version at 0x7f87d29da640>"
controller.add_event_listener(listener_func, EventType.BW)
input()
The trouble here is that our listener invokes within our controller, which is an asynchronous context. Needless to say, this is confusing as hell.
I'm going to revisit and likely remove our asyncio metaprogramming. It's complicated, breaks type checks, and liable to keep causing problems. Pitching this on the python list and stack overflow I get the sense that this isn't a promising approach long term.
I'm not sure if this is a stem or tor issue, or even a Travis load issue:
Traceback (most recent call last):
File "/home/travis/build/torproject/tor/stem/stem/util/test_tools.py", line 143, in
self.method = lambda test: self.result(test) # method that can be mixed into TestCases
File "/home/travis/build/torproject/tor/stem/stem/util/test_tools.py", line 210, in result
test.fail(self._result.msg)
AssertionError: tor didn't quit after the controller that owned it disconnected
See https://travis-ci.org/torproject/tor/jobs/637520130#L3650 for details
Migrated from ticket 23057.
Recently our Jenkins, which run Stem's tests, hosts upgraded their Debian distribution. Doing so caused our test_connections_by_sockstat to start failing...
======================================================================
FAIL: test_connections_by_sockstat
----------------------------------------------------------------------
Traceback (most recent call last):
File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/integ/util/connection.py", line 55, in test_connections_by_sockstat
self.check_resolver(Resolver.SOCKSTAT)
File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/require.py", line 58, in wrapped
return func(self, *args, **kwargs)
File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/integ/util/connection.py", line 37, in check_resolver
self.fail('Unable to find our controller connection with %s (%s). Connections found were...\n\n%s\n\nCommand output was...\n\n%s' % (resolver, resolver_command, '\n'.join(map(str, connections)), resolver_output))
AssertionError: Unable to find our controller connection with sockstat (sockstat). Connections found were...
Connection(local_address=u'127.0.0.1', local_port=1024, remote_address=u'127.0.0.1', remote_port=38974, protocol=u'tcp', is_ipv6=False)
Connection(local_address=u'127.0.0.1', local_port=1024, remote_address=u'127.0.0.1', remote_port=38966, protocol=u'tcp', is_ipv6=False)
Command output was...
[u'USER PROCESS PID PROTO SOURCE ADDRESS FOREIGN ADDRESS STATE', u'jenkins tor 4759 tcp4 127.0.0.1:1024 *:* LISTEN', u'jenkins tor 4759 tcp4 127.0.0.1:1024 *:* LISTEN', u'jenkins tor 4759 tcp4 127.0.0.1:1024 127.0.0.1:38974 ESTABLISHED', u'jenkins tor 4759 tcp4 127.0.0.1:1024 127.0.0.1:38966 ESTABLISHED', u'jenkins sockstat 4930 tcp4 127.0.0.1:38912 127.0.0.1:1111 ESTABLISHED', u'jenkins sockstat 4930 tcp4 127.0.0.1:38912 127.0.0.1:1111 ESTABLISHED', u'jenkins python 18063 tcp4 127.0.0.1:38912 127.0.0.1:1111 ESTABLISHED', u'jenkins python 18063 tcp4 127.0.0.1:38912 127.0.0.1:1111 ESTABLISHED']
Here's the sockstat output...
USER PROCESS PID PROTO SOURCE ADDRESS FOREIGN ADDRESS STATE
jenkins python 18337 tcp4 127.0.0.1:41728 127.0.0.1:1111 ESTABLISHED
jenkins python 18337 tcp4 127.0.0.1:41728 127.0.0.1:1111 ESTABLISHED
jenkins tor 20588 tcp4 127.0.0.1:1024 *:* LISTEN
jenkins tor 20588 tcp4 127.0.0.1:1024 *:* LISTEN
jenkins tor 20588 tcp4 127.0.0.1:1024 127.0.0.1:41814 ESTABLISHED
jenkins tor 20588 tcp4 127.0.0.1:1024 127.0.0.1:41806 ESTABLISHED
jenkins sockstat 20594 tcp4 127.0.0.1:41728 127.0.0.1:1111 ESTABLISHED
jenkins sockstat 20594 tcp4 127.0.0.1:41728 127.0.0.1:1111 ESTABLISHED
Note that our socksport (1024) is listed twice, but our controlport (1111) isn't among the tor process entries at all. However, we're showing connections to the controlport.
Did some searching around but stumped. If we can fix sockstat to once again work on the jenkins hosts I'm all ears - otherwise we'll drop this connection resolution method in Stem 2.0.0.
Some of stem's tests hang in Tor's Travis CI.
So we'd like to be able to say something like:
run_tests.py --test process --exclude-test process.test_take_ownership_via_controller
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.