flyingcircusio / batou Goto Github PK

View Code? Open in Web Editor NEW

47.0 7.0 12.0 4.45 MB

batou is a universal, fractal deployment utility using Python.

Home Page: https://batou.readthedocs.org

License: Other

Shell 0.38% Python 99.19% Jinja 0.02% Nix 0.41%

devops deployment automation continous-deployment continous-delivery

batou's Introduction

batou helps you to automate your application deployments:

You create a model of your deployment using a simple but powerful Python API.
You configure how the model applies to hosts in different environments.
You verify and run the deployment with the batou utility.

Getting started with a new project is easy:

mkdir myproject
cd myproject
git init
curl -sL https://raw.githubusercontent.com/flyingcircusio/batou/main/bootstrap | sh
git commit -m "Start a batou project."

Here's a minimal application model:

$ mkdir -p components/myapp
$ cat > components/myapp/component.py
from batou.component import Component
from batou.lib.python import VirtualEnv, Package
from batou.lib.supervisor import Program

class MyApp(Component):

    def configure(self):
        venv = VirtualEnv('2.7')
        self += venv
        venv += Package('myapp')
        self += Program('myapp',
            command='bin/myapp')

And here's a minimal environment:

$ mkdir environments
$ cat > environments/dev/environment.cfg
[environment]
connect_method = local

[hosts]
localhost = myapp

To deploy this, you run:

$ ./batou deploy dev

Check the detailed documentation to get going with a more ambitious project.

Features

Separate your application model from environments
Supports idempotent operation for incremental deployments
Deploy to multiple hosts simultaneously
Automated dependency resolution for multi-host scenarios
No runtime requirements on your application
Encrypted secrets with multiple access levels: store your SSL certificates, SSH keys, service secrets and more to get true 1-button deployments.
Deploy to local machines, Vagrant, or any SSH host
Broad SSH feature support by using OpenSSH through execnet
Only few dependencies required on the remote host
Ships with a library of components for regularly needed tasks
self-bootstrapping and self-updating - no additional scripting needed

License

The project is licensed under the 2-clause BSD license.

Hacking

Make sure mercurial and subversion are installed and in $PATH.
Run ./develop.sh to create a local virtualenv with everything set up.
Run the test suite using: bin/tox
Build the documentation using: cd doc; make
Set up GPG for the examples with export GNUPGHOME=<DIRECTORY OF BATOU HERE>/src/batou/secrets/tests/fixture/gnupg
Make sure age is installed and in $PATH for age encryption support.

Changelog

See CHANGES.md.

batou's People

Contributors

Stargazers

Watchers

Forkers

frlan ksuess admdev8 swipswaps witsch sallner jugmac00 mpxgit risclog-solution gforcada sweh mcdonc

batou's Issues

Extra newline in requirements.txt causes crash

Example requirements.txt:

batou_ext @ https://github.com/flyingcircusio/batou_ext/archive/b8b38483419476924b426b072d29d668d0718d9f.zip#sha256=f75bd0fbe3094c98f0a9f2d1d2fde5fe3510ab6ed61813da9c16c447339cbac0

Laads to a crash of batou:

% ./batou --help
Installing batou_ext @ https://github.com/flyingcircusio/batou_ext/archive/b8b38483419476924b426b072d29d668d0718d9f.zip#sha256=f75bd0fbe3094c98f0a9f2d1d2fde5fe3510ab6ed61813da9c16c447339cbac0
Installing 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/test.batou/.batou/lib/python2.7/site-packages/batou/bootstrap.py", line 98, in bootstrap
    '"{}"'.format(pip_options, req))
  File "/tmp/test.batou/.batou/lib/python2.7/site-packages/batou/utils.py", line 340, in cmd
    cmd, process.returncode, stdout, stderr)
batou.utils.CmdExecutionError: ('.batou/bin/pip install -U  --no-deps ""', 1, '', "DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support\nERROR: Invalid requirement: ''\n")

supervisorctl died after upgrade

This happened and didn't get unstuck on its own:

     ManagedMySQL > Grant(GRANT ALL)
     Supervisor > Buildout > File(work/supervisor/buildout.cfg) > Content(work/supervisor/buildout.cfg)
     Supervisor > Buildout > VirtualEnv(3) > VirtualEnvPy
     Supervisor > Buildout > VirtualEnv(3) > Package(setuptools==46.1.3)
     Supervisor > Buildout > VirtualEnv(3) > Package(zc.buildout==2.13.3)
     Supervisor > Buildout
     Supervisor > RunningSupervisor(<Service (default) "Supervisor > Service(bin/supervisord)">)
default: Deploying component instanceadmin ...
     InstanceAdmin > Instance > Buildout
     InstanceAdmin > Instance > Program(admin-instance)
ERROR: /home/vagrant/deployment/work/supervisor/bin/supervisorctl reread
     Return code: 1
STDOUT
     error: <class 'xmlrpc.client.ProtocolError'>, <ProtocolError for 127.0.0.1/RPC2: 500 Internal Server Error>: file: /home/vagrant/deployment/work/supervisor/eggs/supervisor-4.1.0-py3.7.egg/supervisor/xmlrpc.py line: 545

STDERR

I had to kill and restart supervisor. I guess the upgrade didn't really properly shutdown/restart supervisor here ... not sure how that was intended. Normally we want upgrades to be as quiet as possible and I guess a full restart isn't always needed anyway ...

Research potential for use of mitogen instead of execnet

Consider a good workflow for creating (batou) appenv lock files

@sweh started an implementation that works for him but isn't obviously the right choice for inclusion within batou.

The original discussion started in #69 and should continue here.

Provide a component for scripts

It would be handy to have a component which is

set to be executable by default
executed on changes (e.g. on itself or parts of parent component changed)

Improve exception reporting: only show deployment-specific traceback-part

We could rewrite exceptions so that framework stacks are removed and only project-specific code-paths are show. E.g. that if you trigger a FileNotFoundError you only get the part in your component but not 12 levels of traceback from batou internals ...

File: improve breadcrumbs

As sketched with @BarbarossaTM yesterday:

only show path if our parent is not the File component itself, otherwise just show basename
show path as absolute (and maybe even 'realpath') or, if beneath the workdir, with the workdir prefix removed

Warn if value is overriden both in secrets and the environment

Scenario:

A overwrite is done inside environments/myenv.cfg:

[component:example]
myvar = True

as well as in secrets secrets/myenv.cfg:


[component:example]
myvar = False

Obviously the overwrites do conflict.

Expectation

Batou should discover this scenario and warn the user

Improve (secret) data handling and separating model computation

Environments are evaluated on the controller
the model computation can happen on individual nodes (assets/agents) to leverage their network environment
we could even perform the dependency discovery in a distributed fashion this way
data/secrets are only fed into the agents as needed

backoff/retry connection failures

We started connecting to all target hosts at once to speed up overall deployment runs. This is fine if we connect to the remote hosts directly, but can be a problem with jump hosts/proxies/bastions if they rate limit new connections, like SSHD does with MaxStartup.

The situation results currently also in confusion because execnet raises HostNotFound even though SSH did give more specific errors (conn reset by peer). People expect DNS problems due to that message. Also, DNS might actually be a problem (isn't it always) in real situations that we don't handle well.

So, the idea would be to a) try to connect as usual b) handle failed connections by backing off and retrying using (CSMA/CD) assuming we have a 'shared medium access' problem. The algorithm for that already is coded in fc.qemu, so we can 'steal' from there.

Switch cwd to component/xyz during configure phase

Provide a better error message if gpg is missing

On a new MacOS Cataline, batou fails decrypting secrets with the message

        File "/Users/fc/develop/claimx_deployment/.batou/lib/python2.7/site-packages/batou/secrets/encryption.py", line 110, in _decrypt
          shell=True)
        File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 216, in check_output
          process = Popen(stdout=PIPE, *popenargs, **kwargs)
        File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 394, in __init__
          errread, errwrite)
        File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1047, in _execute_child
          raise child_exception
      TypeError: execv() arg 2 must contain only strings

Can batou check if gpg is installed and provide a better error message?

Smooth consistency check workflow for avoiding commit/push cycles for typos while setting up production

Usually having

update_method = git-bundle

for a deployment environment is a good idea to avoid uncommitted changes.

In some cases, like manually rolling out bigger changes, setting it to rsync

update_method = rsync

is more handy.

However, this might leads to checkins like that ...

-update_method = git-bundle
+update_method = rsync

To avoid this some kind of unwanted changes CLI argument for manually overwriting update_method would be nice.

pytest integration for deployment code

batou test ?

Design better data modelling / API

key/value style or more structured?
access control to which context can read/write which data?
types that help coalescing unset values to empty types (e.g. to avoid null handling being special when lists are expected: if 'x' in bar ..: for foo in bar['x']. or for foo in bar.get('x', [])
secrets
overlaying
integrating this with dependency modelling? who provides values, who reads values?

Inspiration can be taken from:

batou 1/2 with overrides, secrets, provide/require
puppet/hiera
chef/pillar
consul

This would help pave the road to make the modelling more explicit tool-based python and leverage a clean data API that is easy to access in Python and templates.

Implement fine-grained parallelization

The current state of parallelisation can only really leverage this over multiple hosts (because the execnet connection will block). Fine-grained parallelization would mean we can also trigger multiple components on the same host at the same time.

I have drafted a way to do this with execnet over here: https://gist.github.com/ctheune/eda98d8667f65c962c03c0ce2b92a11a

However, the existing code is quite involved and requires detailed cleanup of both the remote core and the controller and will benefit from revamping the output management (and maybe switch over to structlog).

switch output to structlog, separate output data collection from terminal/log writing
implement pattern from the gist
upgrade remote_core protocol to multiplex commands

Per-host secret overrides

The data-* sections aren't currently included in the secret overrides, which is a sorely missed feature for me right now.

Disable % variable expansion in configparser for environments and secrets

Not sure whether this still happens on Python 3, but we have an internal ticket that shows this error when someone accidentally adds a % to a secret file (i.e. when a randomly generated password contains this):

ERROR: Unexpected exception
      Traceback (most recent call last):
        File "/Users/.../sources/.../.batou/lib/python2.7/site-packages/batou/deploy.py", line 140, in main
          deployment.load()
        File "/Users/.../sources/.../.batou/lib/python2.7/site-packages/batou/deploy.py", line 60, in load
          self.environment.load_secrets()
        File "/Users/.../sources/.../.batou/lib/python2.7/site-packages/batou/environment.py", line 152, in load_secrets
          add_secrets_to_environment_override(self)
        File "/Users/.../sources/.../.batou/lib/python2.7/site-packages/batou/secrets/__init__.py", line 42, in add_secrets_to_environment_override
          o.update(f.config.items(section_))
        File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 655, in items
          for option in options]
        File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 666, in _interpolate
          value = value % vars
      ValueError: incomplete format key

Deployment hangs on `Loading secrets ...`

... if secrets file is opened in some other terminal.

Reproduce
batou v1.12.0

terminal one: ./batou secrets edit some_env

terminal two: ./batou deploy some_env

terminal output hangs at:

./batou deploy some_env
Installing git+git://github.com/flyingcircusio/batou_ext.git@5f658851cfa521c5b439e54e398a719004707775#egg=batou_ext
batou/1.12.0 (CPython 2.7.15-candidate1, Linux 4.15.0-1064-oem x86_64)
================================================================ Preparing =================================================================
main: Loading environment `some_env`...
main: Verifying repository ...
     You are using rsync. This is a non-verifying repository -- continuing on your own risk!
main: Loading secrets ...

expected (wished) behavior
show some message like Deployment cannot proceed as secrets file is currently in use. Please close it. Deployment will resume automatically.

On the plus side... deployment already continues when you close the secrets file.

provide `lib/` directory to place importable modules with reusable code.

Support bootstrapping/provisioning from within a batou model

bootstrapping can include needing to use temporary addresses/connections that are needed to even get proper users installed
may need more types to describe installed packages, users, system config for various platforms
provisioning may also need to talk to APIs to create VMs first and waiting for those to appear
warn if temporary/insecure connections are made ...

Make batou ssh call more flexible

Especially in deployment pipelines it needs to be defined in the pipeline which user and/or key should be used to connect.

I suggest using an environment variable BATOU_SSH_EXTRA_ARGS

AppEnv: non-reproducable EnvironmentError

Got this error once, next deploy run succeeded:

<host> > SMTPBackend > AppEnv('40ee344c') > LockedRequirements
ERROR: .appenv/40ee344cc81b5ed5384b00b10831686bcf5fa36925edd0a560c2dc9cdf99ca3e/bin/python -m pip install --no-deps -r .appenv/40ee344cc81b5ed5384b00b10831686bcf5fa36925edd0a560c2dc9cdf99ca3e/requirements.lock
Return code: 1
STDOUT
Looking in indexes: https://pypi.org/simple, https://pypi:****@<custompypi>/simple
Collecting Babel==2.7.0
  Using cached Babel-2.7.0-py2.py3-none-any.whl (8.4 MB)
Collecting Beaker
  Using cached Beaker-1.11.0.tar.gz (40 kB)
Collecting Chameleon==3.4
  Using cached Chameleon-3.4.tar.gz (116 kB)
Collecting IP2Location
  Using cached IP2Location-8.4.1-py3-none-any.whl (6.9 kB)
Collecting Mako==1.1.2
  Using cached Mako-1.1.2-py2.py3-none-any.whl (75 kB)
Collecting MarkupSafe==1.1.1
  Using cached MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl (27 kB)
Collecting Paste==3.2.5
  Using cached Paste-3.2.5-py2.py3-none-any.whl (592 kB)
Collecting PasteDeploy==2.0.1
  Using cached PasteDeploy-2.0.1-py2.py3-none-any.whl (17 kB)
Collecting Pillow
  Using cached Pillow-7.1.2-cp36-cp36m-manylinux1_x86_64.whl (2.1 MB)

STDERR
ERROR: Could not install packages due to an EnvironmentError: Could not find a suitable TLS CA certificate bundle, invalid path: /path/to/.appenv/40ee344cc81b5ed5384b00b10831686bcf5fa36925edd0a560c2dc9cdf99ca3e/lib/python3.6/site-packages/pip/_vendor/certifi/cacert.pem

Connecting with vagrant fails?

@sweh reported me this via chat:

default: Connecting via vagrant (1/1)
======================================================================================================================================= Configuring model ... =======================================================================================================================================
ERROR: Unexpected exception
      Traceback (most recent call last):
        File "/Users/sweh/dev/clxmci_deployment/.batou/2ed829d2e83a1c08d0384cba9b37cc22e8c3df118448c40b3b536e584923910e/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 50, in bootstrap_exec
          assert s == "1".encode("ascii")
      AssertionError

Extended component ordering

I have a case where between a component's verify and update, another component needs to run. Specifically that is a nixos UserEnv (batou_ext) which requires files to be prefetched. The flow would look like this:

userenv.verify()
prefetch.verify()  # <- checks if userenv.verify() requires update
prefetch.update()
userenv.update()

Test failures on master

Running the tests of current master (aka 297349b) leads locally to 8 failures. CI tests did not run for the last couple of commits. Locally I get:

====================================================================================================== FAILURES ======================================================================================================
____________________________________________________________________________________________________ FLAKE8-check ____________________________________________________________________________________________________
/.../batou/src/batou/deploy.py:154:80: E501 line too long (80 > 79 characters)

____________________________________________________________________________________________________ FLAKE8-check ____________________________________________________________________________________________________
/.../batou/src/batou/main.py:128:8: E713 test for membership should be 'not in'

________________________________________________________________________________________________ test_downloads_file _________________________________________________________________________________________________
Traceback (most recent call last):
  File "/.../batou/src/batou/component.py", line 301, in deploy
    self.verify()
  File "/.../batou/src/batou/lib/download.py", line 33, in verify
    raise batou.UpdateNeeded()
batou.UpdateNeeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.../batou/src/batou/lib/tests/test_download.py", line 53, in test_downloads_file
    root.component.deploy()
  File "/.../batou/src/batou/component.py", line 290, in deploy
    sub_component.deploy(predict_only)
  File "/.../batou/src/batou/component.py", line 312, in deploy
    self.update()
  File "/.../batou/src/batou/lib/download.py", line 41, in update
    self._update_requests()
  File "/.../batou/src/batou/lib/download.py", line 54, in _update_requests
    **(self.requests_kwargs if self.requests_kwargs else {}))
  File "/.../eggs/requests-2.22.0-py3.7.egg/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/.../eggs/requests-2.22.0-py3.7.egg/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/.../eggs/requests-2.22.0-py3.7.egg/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/.../eggs/requests-2.22.0-py3.7.egg/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/.../eggs/requests-2.22.0-py3.7.egg/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/.../eggs/urllib3-1.25.3-py3.7.egg/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/.../eggs/urllib3-1.25.3-py3.7.egg/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/.../eggs/urllib3-1.25.3-py3.7.egg/urllib3/connection.py", line 183, in connect
    conn = self._new_conn()
  File "/.../eggs/urllib3-1.25.3-py3.7.egg/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/.../eggs/urllib3-1.25.3-py3.7.egg/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
  File "/.../eggs/pytest_timeout-1.3.3-py3.7.egg/pytest_timeout.py", line 140, in handler
    timeout_sigalrm(item, params.timeout)
  File "/.../eggs/pytest_timeout-1.3.3-py3.7.egg/pytest_timeout.py", line 313, in timeout_sigalrm
    pytest.fail('Timeout >%ss' % timeout)
  File "/.../eggs/pytest-5.2.1-py3.7.egg/_pytest/outcomes.py", line 128, in fail
    raise Failed(msg=msg, pytrace=pytrace)
Failed: Timeout >45.0s
_________________________________________________________________________________________ test_runs_svn_to_clone_repository __________________________________________________________________________________________
Traceback (most recent call last):
  File "/.../batou/src/batou/lib/tests/test_svn.py", line 10, in test_runs_svn_to_clone_repository
    cmd('svnadmin create ' + repos_path)
  File "/.../batou/src/batou/utils.py", line 344, in cmd
    cmd, process.returncode, stdout, stderr)
batou.utils.CmdExecutionError: ('svnadmin create /private/var/folders/_2/5m3t4zvn29gb2f2_4phtk19r0000gn/T/pytest-of-mac/pytest-17/test_runs_svn_to_clone_reposit0/work/repos', 127, '', '/bin/sh: svnadmin: command not found\n')
________________________________________________________________________________________________ test_example_errors _________________________________________________________________________________________________
Traceback (most recent call last):
  File "/.../batou/src/batou/tests/test_endtoend.py", line 20, in test_example_errors
    assert out == Ellipsis("""\
AssertionError: assert - batou/2... (cpython 3...)
  +
  - ================================== Preparing ===================================
  - main: Loading environment `errors`...
  - main: Verifying repository ...
  - main: Loading secrets ...
  - ================================ Connecting ... ================================
  - localhost: Connecting via local (1/1)...

  ...Full output truncated (45 lines hidden), use '-vv' to show
______________________________________________________________________________________ test_example_errors_missing_environment _______________________________________________________________________________________
Traceback (most recent call last):
  File "/.../batou/src/batou/tests/test_endtoend.py", line 82, in test_example_errors_missing_environment
    assert out == Ellipsis("""\
AssertionError: assert - batou/2... (cpython 3...)
  +
  - ================================== Preparing ===================================
  - main: Loading environment `production`...
  - ERROR: Missing environment
  -      Environment: production
  - ============================== DEPLOYMENT FAILED ===============================
  -
________________________________________________________________________________________________ test_example_ignores ________________________________________________________________________________________________
Traceback (most recent call last):
  File "/.../batou/src/batou/tests/test_endtoend.py", line 96, in test_example_ignores
    out, _ = cmd('./batou deploy ignores')
  File "/.../batou/src/batou/utils.py", line 344, in cmd
    cmd, process.returncode, stdout, stderr)
batou.utils.CmdExecutionError: ('./batou deploy ignores', 1, '', 'Traceback (most recent call last):\n  File "<string>", line 1, in <module>\nModuleNotFoundError: No module named \'batou.bootstrap\'\n')
________________________________________________________________________________________ test_remote_deployment_initializable ________________________________________________________________________________________
Traceback (most recent call last):
  File "/.../batou/src/batou/tests/test_remote.py", line 17, in test_remote_deployment_initializable
    Deployment(env, platform='', timeout=30, fast=False, dirty=False)
TypeError: __init__() got an unexpected keyword argument 'fast'
------------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------------
Exception ignored in: <_io.FileIO name='/private/var/folders/_2/5m3t4zvn29gb2f2_4phtk19r0000gn/T/pytest-of-mac/pytest-17/test_remote_deployment_initial0/sample_service/components/hello/component.py' mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/private/var/folders/_2/5m3t4zvn29gb2f2_4phtk19r0000gn/T/pytest-of-mac/pytest-17/test_remote_deployment_initial0/sample_service/components/hello/component.py' mode='r' encoding='UTF-8'>
----------------------------------------------------------------------------- generated xml file: /.../batou/report.xml -----------------------------------------------------------------------------

---------- coverage: platform darwin, python 3.7.7-final-0 -----------
Coverage HTML written to dir htmlcov

=============================================================================== 8 failed, 298 passed, 79 skipped in 177.15s (0:02:57) ================================================================================

batou2: TypeError when sorting exception errors

Traceback (most recent call last):
  File "/Users/sweh/.pyenv/versions/3.6.10/lib/python3.6/threading.py", line 1294, in _shutdown
    t.join()
  File "/Users/sweh/dev/risclog.kravag.deployment/.batou/2758f56d9458119c7d627cf7dc268aa2c5c30c690c63454bbf57a1fc6fd5cf6c/src/batou/src/batou/deploy.py", line 30, in join
    raise exc_value.with_traceback(exc_tb)
  File "/Users/sweh/dev/risclog.kravag.deployment/.batou/2758f56d9458119c7d627cf7dc268aa2c5c30c690c63454bbf57a1fc6fd5cf6c/src/batou/src/batou/deploy.py", line 22, in run
    self.host.start()
  File "/Users/sweh/dev/risclog.kravag.deployment/.batou/2758f56d9458119c7d627cf7dc268aa2c5c30c690c63454bbf57a1fc6fd5cf6c/src/batou/src/batou/host.py", line 215, in start
    env.overrides, env.deployment.timeout, env.deployment.platform)
  File "/Users/sweh/dev/risclog.kravag.deployment/.batou/2758f56d9458119c7d627cf7dc268aa2c5c30c690c63454bbf57a1fc6fd5cf6c/src/batou/src/batou/host.py", line 64, in call
    message = self.host.channel.receive()
  File "/Users/sweh/dev/risclog.kravag.deployment/.batou/2758f56d9458119c7d627cf7dc268aa2c5c30c690c63454bbf57a1fc6fd5cf6c/lib/python3.6/site-packages/execnet/gateway_base.py", line 749, in receive
    raise self._getremoteerror() or EOFError()
execnet.gateway_base.RemoteError: Traceback (most recent call last):
  File "/Users/sweh/dev/risclog.kravag.deployment/.batou/2758f56d9458119c7d627cf7dc268aa2c5c30c690c63454bbf57a1fc6fd5cf6c/src/batou/src/batou/remote_core.py", line 343, in <module>
  File "/Users/sweh/dev/risclog.kravag.deployment/.batou/2758f56d9458119c7d627cf7dc268aa2c5c30c690c63454bbf57a1fc6fd5cf6c/src/batou/src/batou/remote_core.py", line 301, in setup_deployment
  File "/Users/sweh/dev/risclog.kravag.deployment/.batou/2758f56d9458119c7d627cf7dc268aa2c5c30c690c63454bbf57a1fc6fd5cf6c/src/batou/src/batou/remote_core.py", line 118, in load
  File "/srv/s-kravag/deployment/.batou/01380d5e047b578d89bfe542da556e2c9881250c0034c6e908a9dcb3307f169c/src/batou/src/batou/environment.py", line 395, in configure
    raise self.exceptions[0]
batou.UnknownComponentConfigurationError: (<RootComponent "smtpbackend" object at 140505307135000>, SilentConfigurationError(), <traceback object at 0x7fc9f0ba02c8>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1084, in executetask
  File "/Users/sweh/dev/risclog.kravag.deployment/.batou/2758f56d9458119c7d627cf7dc268aa2c5c30c690c63454bbf57a1fc6fd5cf6c/src/batou/src/batou/remote_core.py", line 349, in <module>
TypeError: '<' not supported between instances of 'int' and 'str'```

A simple approach to multi-phasic deployments?

E.g. we could easily bootstrap an empty cumulus switch if we

could have different credentials for different phases
work around password-based sudo
allow reconnecting after doing destructive actions like performing a system upgrade

Environment independent secrets

Secrets are currently only per environment.

There are certain secrets which are the same for all envs. For non secrets one just puts that on the component class or some file.

Alternative way to express component assignments to hosts, use Pythonic API to construct environments?

TypeError on usage of batou.lib.logrotate.Logrotate

batou git 895960f

Error:

ERROR: TypeError('can only concatenate str (not "bytes") to str')
     This /might/ be a batou bug. Please consider reporting it.
     
           Host: xxx
      Component: logrotate
     
     Traceback (simplified, most recent call last):
       File "/srv/xxx/deployment/.batou/lib/python3.7/site-packages/batou/lib/logrotate.py", line 33, in configure
         config = self.common_config + self.logrotate_template

I think it's is caused by

logrotate_template = pkg_resources.resource_string(
        __name__, 'resources/logrotate.in')

is returning byte, whereas common_config = '' is a str.

Improve lib.git with branch

The current behaviour is a bit odd. git.Clone with branch makes sure the right version is there, but it resets the master.

@gforcada wrote:

One could do git clean -dfxq to get rid of all untracked files and then follow it with a regular git checkout BRANCH/REVISION.

Move lib/ into top-level namespace and internal stuff to _batou?

The batou namespace is a big part of the user API. We could simplify this quite a bit by making it explicit that 'batou' is the namespace for the API of the standard distribution and allow:

from batou import File, Component, ...

Encouraging star imports probably isn't a good idea, though, but this makes things much much easier to bootstrap.

Clean up output formatting and ordering for diffs component status

Provide nicer error message when service user does not exist

When you try to deploy to the fcio infrastructure and the service user does not yet exist, you get something like...

jugmac00@jugmac00-XPS-13-9370:~/Projects/iq-reader.batou$ ./batou deploy production
Installing hg+https://bitbucket.org/flyingcircus/batou_ext@ad30396#egg=batou_ext
batou/1.11.0 (CPython 2.7.15-final0, Linux 4.15.0-1050-oem x86_64)
====================================================== Preparing =======================================================
main: Loading environment `production`...
main: Verifying repository ...
main: Loading secrets ...
================================================ Configuring first host ================================================
apisweb01: Connecting via ssh (1/1)
sudo: unknown user: s-iqreader
sudo: unable to initialize policy plugin
ERROR: Unexpected exception
      Traceback (most recent call last):
        File "/home/jugmac00/Projects/iq-reader.batou/.batou/local/lib/python2.7/site-packages/batou/deploy.py", line 141, in main
          deployment.configure()
        File "/home/jugmac00/Projects/iq-reader.batou/.batou/local/lib/python2.7/site-packages/batou/deploy.py", line 65, in configure
          self.connections.next().join()
        File "/home/jugmac00/Projects/iq-reader.batou/.batou/local/lib/python2.7/site-packages/batou/deploy.py", line 19, in run
          self.host.connect()
        File "/home/jugmac00/Projects/iq-reader.batou/.batou/local/lib/python2.7/site-packages/batou/host.py", line 181, in connect
          self.channel = self.gateway.remote_exec(remote_core)
        File "/home/jugmac00/Projects/iq-reader.batou/.batou/local/lib/python2.7/site-packages/execnet/gateway.py", line 126, in remote_exec
          channel = self.newchannel()
        File "/home/jugmac00/Projects/iq-reader.batou/.batou/local/lib/python2.7/site-packages/execnet/gateway_base.py", line 1002, in newchannel
          return self._channelfactory.new()
        File "/home/jugmac00/Projects/iq-reader.batou/.batou/local/lib/python2.7/site-packages/execnet/gateway_base.py", line 779, in new
          raise IOError("connexion already closed: %s" % (self.gateway,))
      IOError: connexion already closed: <Gateway id='gw1' not-receiving, thread model, 0 active channels>
      
================================================== DEPLOYMENT FAILED ===================================================

Instead, I'd like to see a similar excellent exception message like for errors in the File component:

                raise ValueError(
                    "Missing implicit template file {}. Or did you want "
                    "to create an empty file? Then use File('{}', content='')."
                    .format(guess_source, self._unmapped_path))

That means - what possible went wrong, and some suggestions what you could do to fix the problem yourself.

Acquire a lock on each remote server

batou should actually acquire a lock on each server, so that deployments from two people on different locations do not interfere. batou currently does only lock multiple runs from one location.

The implementation seems to be prepared in src/batou/remote_core.py:

def lock():
    # XXX implement!
    pass

Cannot install batou on Python 2.7 any more

Creating a fresh deployment using Python 2.7 fails because virtualenv tries to install the newest version of setuptools (aka 45.1.0) which is no longer compatible with Python 2.

Trying to pin setuptools using requirements.txt did not help.

Any ideas how to solve this issue?

AppEnv cleanup too early?

Cleaning up old appenvs is too early in my setup: Between AppEnvs configure (with cleanup) and the restart of my instances, errors like ValueError: Missing template asset: pkg:templates/login.pt (/srv/user/deployment/work/portal/.appenv/898d917c87d9d9207360432cbf5cfed35ef2bcc087b3d8a459d26bcfcfab7705/lib/python3.6/site-packages/pkg/templates/login.pt) or Can't connect to server (Could not find a suitable TLS CA certificate bundle, invalid path: /srv/user/deployment/work/portal/.appenv/344b3ce7e373d2de5cc51a45e018e57243fe8597139d85c1471509f9081fdf9e/lib/python3.6/site-packages/certifi/cacert.pem) might occur.

We should either find a solution that the resolved path uses the current link instead of the hash (preferred) or kind of mark an appenv as deleted and remove it some time later.

readable mode argument

I'd really like to be able to:

    self += File('check_ptm.py', mode='rwxr-xr-x')

Prohibit calling Component constructors with unknown attributes as arguments.

Allow marking changes as 'critical' or 'downtime' so prediction runs can tell whether we'll cause downtime

Things like restarting switchd will cause downtimes. It would be nice if a component could mark itself as causing downtime when running update() so that predictions can actually subsume this and give a report of "yellow" instead of "green" for a deployment including a list of things that will cause downtime (and maybe how long?)

batou.lib.git.Clone() might delete component folder when target is left empty

When not explicit giving a target for the Clone-component, it might happen that the folder below deployment/work/ gets deleted.

The default for target is set to the workdir of the component (self.map('.')). Later on it's checked whether there is a folder .git below this structure. If this folder is not found, it happen that Clone() is calling ensure_path_nonexistent(self.target) which results in deletion of this directory.

Code to reproduce:

Environment:

[environment]
connect_method = local

[host:localhost]
components =
        testclone1
        testclone2

Dummy component

import batou.component
import batou.lib.git


class Testclone1(batou.component.Component):

    def configure(self):
        self += batou.lib.git.Clone(
            '[email protected]:flyingcircusio/batou.git',
            branch='master')


class Testclone2(batou.component.Component):

    def configure(self):
        self += batou.lib.git.Clone(
            '[email protected]:flyingcircusio/batou.git',
            target='checkout',
            branch='master')

requirements.txt direct references are not updated properly

direct references (https://www.python.org/dev/peps/pep-0440/#direct-references) are not updated on change, e.g. batou_ext @ https://github.com/flyingcircusio/batou_ext/archive/aa154a0d9f7cf8e434f69e36db79586b91dadbb4.zip#sha256=a1b6a655d672bf488f0a9e773da097ce4a28919ad1c50b74ba4829d48690fa2f is being installed once but not updated if changed.

Support for ad-hoc actions

Examples:

invalidate the varnish cache (pre-defined in the deployment)
show me the output of running shell command 'x' (completely adhoc)
maybe even: preprocess and transfer data from the following files to a local (or remote) place

Catch up documentation

review changes
install/bootstrap/upgrade
assert in verify()
defdir during configure
others?

servicesstag70 > InstanceAdmin > Instance > Program('admin-instance')
ERROR: /srv/s-directory/deployment/work/supervisor/bin/supervisorctl status admin-instance
Return code: 3
STDOUT
admin-instance                   FATAL     Exited too quickly (process log may have details)

STDERR```

./batou appenv-update-lockfile causes wrong requirements.lock in combination with batou_ext

Having a minimal requirements.txt at the project:

batou==2.0b12
batou_ext @ https://github.com/flyingcircusio/batou_ext/archive/5b3a8de035d8b68e98aa10483568dd5fa524a149.zip#sha256=972260e6186ee4211f63f18d598db697169c2e1151189fe88d45b85e1028b2a2

After running

./batou appenv-update-lockfile                  
Updating lockfile
Creating venv ...
Ensuring pip ...
Installing packages ...

the created requirements.lock is not reflecting the batou_ext-dependency correctly

% cat requirements.lock 
apipkg==1.5
batou==2.0b12
batou-ext==0.1.dev0
certifi==2020.4.5.1
chardet==3.0.4
execnet==1.7.1
idna==2.9
Jinja2==2.11.2
MarkupSafe==1.1.1
py==1.8.1
pyaml==20.4.0
PyYAML==5.3.1
requests==2.23.0
six==1.15.0
urllib3==1.25.9