huntdatacenter/jujuna

Jujuna - Continuous deployment, upgrade and testing for Juju.

Jujuna

Continuous deployment, upgrade and testing for Juju.

At HUNT Cloud we run our scientific services based on OpenStack orchestrated by Juju. Such cloud deployments rely on a large set of collaborative softwares, and upgrades can sometimes cause considerable pain. We have therefore introduced Jujuna - a tool to simplify the validation of Juju-based OpenStack upgrades.

New to Juju? Juju is a cool controller and agent based tool from Canonical to easily deploy and manage applications (called Charms) on different clouds and environments (see how it works for more details).

Installation

Easy to install using:

pip3 install jujuna

May require installation of libssl-dev package or equivalent if not present in the system.

Run command jujuna --help to get the help menu. You can also check the usage in the documentation.

Usage

Running Jujuna requires a working deployment of Juju controller. Juju configs ~/.local/share/juju/ have to be present or credentials have to be specified using params.

# Deploy Ceph bundle into ceph model
jujuna deploy openstack/bundle.yaml -m test-cloud -w

# Upgrade apps in ceph model_name
jujuna upgrade -m test-cloud

# Test apps in the model after upgrade
jujuna test tests/openstack-ocata.yaml -t 1800 -m test-cloud

# Destroy apps within a model, without destroying the model
jujuna clean -m test-cloud -w -f -t 1800

If you don't have any bundle or just need to try jujuna with some simple example, you can follow our example guide.

Testing Jujuna

How to make sure jujuna and your feature works before pushing out new version.

Testing jujuna with python environments:

tox -e lint
tox -e py35
tox -e py36
tox -e py37

Testing specific feature:

py.test -k test_feature

Deploy to docker registry:

Build image:

docker build -t registry.example.com/group/jujuna:0.2.1 -t registry.example.com/group/jujuna:latest .

Push to registry:

docker push registry.example.com/group/jujuna

jujuna's Issues

UnboundLocalError: local variable 'resources' referenced before assignment when deploying bundle on Ubuntu Bionic 18.04

Deploying an example bundle with a local charm on Ubuntu Bionic 18.04 host fails while it works on Ubuntu Xenial 16.04 host. Obvious differences I see are the host system, system packaged Python version and Python package versions (see further down in the issue)

See GitHub actions logs for failed run and error:

tox -e test-xenial
test-xenial create: /home/runner/work/charm-megaraid-check/charm-megaraid-check/.tox/test-xenial
test-xenial installdeps: jujuna
test-xenial installed: argcomplete==1.10.0,async-timeout==3.0.1,bcrypt==3.1.7,certifi==2019.11.28,cffi==1.13.2,chardet==3.0.4,cryptography==2.8,idna==2.8,juju==2.6.3,jujubundlelib==0.5.6,jujuna==0.2.0,macaroonbakery==1.2.3,paramiko==2.7.0,protobuf==3.11.1,pyasn1==0.4.8,pycparser==2.19,pymacaroons==0.13.0,PyNaCl==1.3.0,pyRFC3339==1.1,pytz==2019.3,PyYAML==5.2,requests==2.22.0,six==1.13.0,theblues==0.5.2,toposort==1.5,urllib3==1.25.7,websockets==7.0
test-xenial run-test-pre: PYTHONHASHSEED='2133000312'
test-xenial run-test: commands[0] | jujuna deploy --wait tests/bundles/xenial.yaml
2019-12-04 18:56:58,863 - jujuna.deploy - INFO - Reading bundle: tests/bundles/xenial.yaml
2019-12-04 18:56:58,941 - jujuna.deploy - INFO - Deploy: local:tests/bundles/xenial.yaml
ERROR:root:Traceback (most recent call last):
ERROR:root:  File "/home/runner/work/charm-megaraid-check/charm-megaraid-check/.tox/test-xenial/lib/python3.7/site-packages/jujuna/__main__.py", line 203, in main
    run_action(action, timeout, args)
ERROR:root:  File "/home/runner/work/charm-megaraid-check/charm-megaraid-check/.tox/test-xenial/lib/python3.7/site-packages/juju/loop.py", line 38, in run
    raise task.exception()
ERROR:root:  File "/home/runner/work/charm-megaraid-check/charm-megaraid-check/.tox/test-xenial/lib/python3.7/site-packages/jujuna/__main__.py", line 145, in run_action
    return await selected_action(**args)
ERROR:root:  File "/home/runner/work/charm-megaraid-check/charm-megaraid-check/.tox/test-xenial/lib/python3.7/site-packages/jujuna/deploy.py", line 58, in deploy
    entity_url
ERROR:root:  File "/home/runner/work/charm-megaraid-check/charm-megaraid-check/.tox/test-xenial/lib/python3.7/site-packages/juju/model.py", line 1375, in deploy
    await handler.execute_plan()
ERROR:root:  File "/home/runner/work/charm-megaraid-check/charm-megaraid-check/.tox/test-xenial/lib/python3.7/site-packages/juju/bundle.py", line 153, in execute_plan
    self.references[step.id_] = await change.run(self)
ERROR:root:  File "/home/runner/work/charm-megaraid-check/charm-megaraid-check/.tox/test-xenial/lib/python3.7/site-packages/juju/bundle.py", line 331, in run
    resources=resources,
ERROR:root:UnboundLocalError: local variable 'resources' referenced before assignment
ERROR: InvocationError for command /home/runner/work/charm-megaraid-check/charm-megaraid-check/.tox/test-xenial/bin/jujuna deploy --wait tests/bundles/xenial.yaml (exited with code 1)

Working log from Ubuntu Xenial 16.04:

zeestrat@vagrant:~/dev/charms/charm-megaraid-check$ make test-xenial
tox -e test-xenial
test-xenial recreate: /home/zeestrat/dev/charms/charm-megaraid-check/.tox/test-xenial
test-xenial installdeps: jujuna
test-xenial installed: argcomplete==1.10.0,async-timeout==2.0.1,bcrypt==3.1.7,certifi==2019.11.28,cffi==1.13.2,chardet==3.0.4,cryptography==2.8,idna==2.8,juju==0.11.7,jujubundlelib==0.5.6,jujuna==0.2.0,macaroonbakery==1.2.3,paramiko==2.7.0,pkg-resources==0.0.0,protobuf==3.11.1,pyasn1==0.4.8,pycparser==2.19,pymacaroons==0.13.0,PyNaCl==1.3.0,pyRFC3339==1.1,pytz==2019.3,PyYAML==3.13,requests==2.22.0,six==1.13.0,theblues==0.5.2,urllib3==1.25.7,websockets==7.0
test-xenial runtests: PYTHONHASHSEED='1668498150'
test-xenial runtests: commands[0] | jujuna deploy --wait tests/bundles/xenial.yaml
2019-12-06 09:40:20,660 - jujuna.deploy - INFO - Reading bundle: tests/bundles/xenial.yaml
2019-12-06 09:40:20,833 - jujuna.deploy - INFO - Deploy: local:tests/bundles/xenial.yaml
2019-12-06 09:40:46,914 - jujuna.deploy - INFO - PROGRESS - Machines: 2 Apps: 4 Stats: {'waiting': 4} Workload: {'waiting': 2}

Python packages:

Bionic

test-xenial installed: argcomplete==1.10.0
async-timeout==2.0.1
bcrypt==3.1.7
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
cryptography==2.8
idna==2.8
juju==0.11.7
jujubundlelib==0.5.6
jujuna==0.2.0
macaroonbakery==1.2.3
paramiko==2.7.0
pkg-resources==0.0.0
protobuf==3.11.1
pyasn1==0.4.8
pycparser==2.19
pymacaroons==0.13.0
PyNaCl==1.3.0
pyRFC3339==1.1
pytz==2019.3
PyYAML==3.13
requests==2.22.0
six==1.13.0
theblues==0.5.2
urllib3==1.25.7
websockets==7.0

Xenial

test-xenial installed: argcomplete==1.10.0
async-timeout==3.0.1
bcrypt==3.1.7
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
cryptography==2.8
idna==2.8
juju==2.6.3
jujubundlelib==0.5.6
jujuna==0.2.0
macaroonbakery==1.2.3
paramiko==2.7.0
protobuf==3.11.1
pyasn1==0.4.8
pycparser==2.19
pymacaroons==0.13.0
PyNaCl==1.3.0
pyRFC3339==1.1
pytz==2019.3
PyYAML==5.2
requests==2.22.0
six==1.13.0
theblues==0.5.2
toposort==1.5
urllib3==1.25.7
websockets==7.0

Diff Python packages

diff bionic xenial
2c2
< async-timeout==3.0.1
---
> async-timeout==2.0.1
9c9
< juju==2.6.3
---
> juju==0.11.7
13a14
> pkg-resources==0.0.0
21c22
< PyYAML==5.2
---
> PyYAML==3.13
25d25
< toposort==1.5

Upgrade refactor

Intended upgrade refactor to decouple charm revision upgrade and service upgrade

Highlight default services and order of charms that will be upgraded

We should highlight which default services we attempt to upgrade and in which order from https://github.com/huntdatacenter/jujuna/blob/master/jujuna/settings.py so it is easier to understand what is going on and compare with the suggested order from https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-upgrade-openstack.html

Can we generate the list from the settings.py above in docs or something?

Service versions on upgrade

list service versions when upgrading

model.applications['nova-compute'].safe_data.get('workload-version', '')

utilize subordinate info from libjuju

This was not working before, should now be available in libjuju 2.7

Setup CI

Setup previously used testing in TravisCI

Go away from deployment specific charm names

Somewhat related to #34.

Many of the names of charm services listed in https://github.com/huntdatacenter/jujuna/blob/master/jujuna/settings.py are specific to our deployment which will probably break for other deployments/bundles.

Could quick ideas:

• Use/track actual charm names from the charm store and not the deployed service names
• Split up SERVICES list somehow so we can have one track only upstream charms from https://jujucharms.com/u/openstack-charmers/ and another for any other supplemental charms such as ntp, nrpe and etc?

Add example bundle and test

Create a small easy to deploy bundle (preferably some openstack service) with an example test suite

juju 2.5.0 bootstrap tasks fails in example Vagrant box

We're hitting an issue where the juju bootstrap lxd task with Juju 2.5.0 in the example Vagrant box. Juju 2.4.7 stable works so should switch to that for now.

Publish builds to Snap Store

Now that #41 is in place, we should add a deploy step that publishes builds to the Snap Store.

https://docs.travis-ci.com/user/deployment/snaps/

Failing to get charmstore info on upgrade not displaying revision

When upgrading and the charmstore doesn't respond, jujuna will continue upgrading without display the new revision.

Perhaps a couple of retries would be handy?

vagrant@vagrant:~$ jujuna upgrade -o cloud:xenial-ocata -p -a keystone
2019-01-11 21:24:12,504 - jujuna.upgrade - INFO - Applications present in the current model: keystone, percona-cluster
2019-01-11 21:24:12,505 - jujuna.upgrade - INFO - Upgrading charms
2019-01-11 21:24:12,839 - jujuna.upgrade - WARNING - Failed loading information from charmstore: cs:keystone-264
2019-01-11 21:24:12,840 - jujuna.upgrade - INFO - Upgrade keystone from: 264 to: 0
2019-01-11 21:24:50,561 - jujuna.upgrade - INFO - Upgraded: 1 charms

Add documentation for test writing and modules

We are missing documentation for test writing.