elifesciences/threadbare

A partial replacement for eLife's usage of Fabric 1.x and Paramiko.

Threadbare

A partial replacement for the slice of Fabric 1.x and Paramiko used in elifesciences/builder.

Threadbare supports Python 3.6 and greater.

Threadbare removed support for Python 2 in major version 3.0.0.

Documentation.

installation

As a library:

pip install threadbare

For development:

./install.sh -dev

usage

Threadbare can be used like this:

import threadbare
threadbare.operations.remote("echo 'hello world!'", host_string='my.server', warn_only=True)

or like this:

from threadbare.state import settings
from threadbare.operations import remote
with settings(host_string='my.server', warn_only=True):
    remote("echo 'hello world!'")

See example.py for more.

local tests

Run the test suite with ./test.sh

Individual tests can be run with ./test.sh example.py::test_fn_name

remote tests

A set of tests that also serve as working examples of threadbare's functionality can be executed by starting a dummy SSH server:

./tests-remote/sshd-server.sh

And then, in another terminal, run example.py with:

./test-examples.sh

While the examples are harmless they do involve uploading and downloading files to /tmp, testing permissions and invoking sudo. Individual tests can be run with ./test-examples.sh -k test_fn_name.

The dummy ssh server creates the temporary directory /tmp/sshd-dummy and writes the host and user certificates there.

The only user that may connect to this dummy SSH server is the current user using the certificate generated for them.

See ./tests-remote/ssh-client.sh for a working example on connecting to the dummy SSH server.

local+remote tests

This runs a dummy ssh server and runs both the local unit tests as well as the remote tests. It's used in CI and generates a set of temporary credentials with deliberately insecure ssh configuration. Be cautious.

./project_tests.sh

a guide to Threadbare for developers

Threadbare is comprised of just three modules:

1. state
2. operations
3. execute

state

(source)

Underpinning all of Fabric is it's manipulation of a global state dictionary called the 'environment'.

For the most part this manipulation is sane, single threaded and predictable.

It has many defaults and functions will dip into the environment dictionary and pull out settings as they need them.

Global state can be built up by nesting context managers like settings or hide or lcd, pushing new state into the environment dictionary, and then popping it off as execution leaves the context manager.

The state module replicates this mechanism but with no initial defaults. Default values are pushed back into the functions and may be overridden globally in a general way or overridden specifically by passing in a keyword argument to the function.

This change:

• improves locality of reference. All options that a function operates on are defined right there with the function.
• keeps the state module small and dumb and predictable

operations

(source)

Fabric provides a set of very useful commands, like local or sudo or cd.

The operations module re-implements these commands and they make heavy use of the state module.

Not all commands have been re-implemented, just those in use by Builder.

A big part of the operations module is running commands on a remote host. Where Fabric uses it's sister project Paramiko, Threadbare uses a library called PSSH, or ParallelSSH. ParallelSSH itself wraps another project of theirs called ssh-python, an interface between libssh and Python. ParallelSSH does a little more than just wrap ssh-python, it also provides a higher level (and more convenient) interface to basic network operations and ensures thread safety.

Threadbare does not use the parallel capabilities of ParallelSSH. See execute.

execute

(source)

Fabric provides an interface for running commands in parallel using their global state manipulation mechanism and set of operations. It uses Python's multiprocessing module.

Threadbare re-implements this mechanism in execute, also using multiprocessing.

In Fabric the parallelism is used primarily to execute commands on multiple hosts simultaneously, so the interfaces are skewed towards that. Threadbare is more general purpose but provides the function execute_with_hosts that behaves as Fabric's execute does.

Both Threadbare and Fabric share the same caveats of running code in different processes using multiprocessing with a mechanism for manipulating global state.

• changes to global state within the worker function does not propagate back to the parent
• SSH connections cannot be passed to child processes so new connections are made within the child process if necessary
• child processes cannot prompt for input. They have no access to stdin.
• child processes may die or throw exceptions that can't be properly handled in the parent

Licence

Copyright © 2019-2023 eLife Sciences

Distributed under the GNU Affero General Public Licence, version 3.

threadbare's Issues

upload/download, overwrite behaviour is not consistent or configurable

"Unlike with the SFTP functionality, remote files that already exist are not overwritten and an exception is raised instead."

• https://github.com/ParallelSSH/parallel-ssh#user-content-scp

this implies that:

• SFTP overwrites files on upload
• SCP does not overwrite uploaded file

With no mention of behaviour when downloading files.

Threadbare has relied on the default behaviour of SFTP so far and switching to SCP may have unintended side effects unless overwrite behaviour for local and remote files is made explicit with a default enforced that can be overridden.

Fabric/Threadbare inconsistency: UnicodeDecodeError

bldr switch_revision_update_instance:search--continuumtest,approved
...
54.221.28.185   out: Collecting pip
54.221.28.185   out:   Downloading https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce1
4b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl (1.4MB)
exception while executing task 'switch_revision_update_instance': 'ascii' codec can't encode character u'\u258e' in position 10
: ordinal not in range(128)

Traceback (most recent call last):
  File "src/taskrunner.py", line 218, in exec_task
    return_map['result'] = task_map['fn'](*task_args, **task_kwargs)
  File "/home/giorgio/code/builder/src/decorators.py", line 80, in call
    return func(stackname, *args, **kwargs)
  File "/home/giorgio/code/builder/src/deploy.py", line 18, in switch_revision_update_instance
    bootstrap.update_stack(stackname, service_list=['ec2'], concurrency=concurrency_for(stackname, concurrency))
  File "/home/giorgio/code/builder/src/buildercore/bootstrap.py", line 398, in update_stack
    fn(stackname, context, **actual_arguments)
  File "/home/giorgio/code/builder/src/buildercore/context_handler.py", line 84, in decorated_with_only_if
    return fn(stackname, context, **kwargs)
  File "/home/giorgio/code/builder/src/buildercore/decorators.py", line 18, in _wrapper
    return func(stackname, *args, **kwargs)
  File "/home/giorgio/code/builder/src/buildercore/bootstrap.py", line 549, in update_ec2_stack
    stack_all_ec2_nodes(stackname, _update_ec2_node, username=BOOTSTRAP_USER, concurrency=concurrency)
  File "/home/giorgio/code/builder/src/buildercore/core.py", line 370, in stack_all_ec2_nodes
    return serial_work(single_node_work, params)
  File "/home/giorgio/code/builder/src/buildercore/core.py", line 382, in serial_work
    return execute(serial(single_node_work), hosts=list(params['public_ips'].values()))
  File "/home/giorgio/code/builder/src/buildercore/threadbare/execute.py", line 222, in execute_with_hosts
    results = execute(func, param_key="host_string", param_values=host_list)
  File "/home/giorgio/code/builder/src/buildercore/threadbare/execute.py", line 202, in execute
    return _serial_execution(func, param_key, param_values)
  File "/home/giorgio/code/builder/src/buildercore/threadbare/execute.py", line 151, in _serial_execution
    result_list.append(func())
  File "/home/giorgio/code/builder/src/buildercore/threadbare/execute.py", line 15, in inner
    return func(*args, **kwargs)
  File "/home/giorgio/code/builder/src/buildercore/core.py", line 343, in single_node_work
    return workfn(**work_kwargs)
  File "/home/giorgio/code/builder/src/buildercore/bootstrap.py", line 508, in _update_ec2_node
    run_script('bootstrap.sh', salt_version, minion_id, install_master_flag, master_ip, **environment_vars)
  File "/home/giorgio/code/builder/venv/local/lib/python2.7/site-packages/backoff/_sync.py", line 99, in retry
    ret = target(*args, **kwargs)
  File "/home/giorgio/code/builder/src/buildercore/bootstrap.py", line 54, in run_script
    result = remote_sudo(" ".join(env_string + cmd))
  File "/home/giorgio/code/builder/src/buildercore/threadbare/operations.py", line 371, in remote_sudo
    return remote(command, **kwargs)
  File "/home/giorgio/code/builder/src/buildercore/threadbare/operations.py", line 331, in remote
    stdout = _process_output(sys.stdout, result["stdout"], **output_kwargs)
  File "/home/giorgio/code/builder/src/buildercore/threadbare/operations.py", line 250, in _process_output
    new_results = [_print_line(output_pipe, line, **kwargs) for line in result_list]
  File "/home/giorgio/code/builder/src/buildercore/threadbare/operations.py", line 237, in _print_line
    output_pipe.write(template.format(**template_kwargs))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u258e' in position 10: ordinal not in range(128)