juju/charm-helpers

Juju is an open source application orchestration engine that enables any application operation (deployment, integration, lifecycle management) on any infrastructure (Kubernetes or otherwise) at any scale (development or production) in the same easy way (typically, one line of code), through special operators called ‘charms’.

👉	Juju	Learn how to quickly deploy, integrate, and manage charms on any cloud with Juju. It's as simple as juju deploy foo, juju integrate foo bar, ..., on any cloud.
	Charmhub	Sample our existing charms on Charmhub. A charm can be a cluster (OpenStack, Kubernetes), a data platform (PostgreSQL, MongoDB, etc.), an observability stack (Canonical Observability Stack), an MLOps solution (Kubeflow), and so much more.
	Charm SDK	Write your own charm! Juju is written in Go, but our SDK supports easy charm development in Python.

Give it a try!

Let's use Juju to deploy, configure, and integrate some Kubernetes charms:

Set up

You will need a cloud and Juju. The quickest way is to use a Multipass VM launched with the charm-dev blueprint.

Install Multipass: Linux | macOS | Windows. On Linux:

sudo snap install multipass

Use Multipass to launch an Ubuntu VM with the charm-dev blueprint:

multipass launch --cpus 4 --memory 8G --disk 30G --name tutorial-vm charm-dev 

Open a shell into the VM:

multipass shell tutorial-vm

Verify that you have Juju and two localhost clouds:

juju clouds

Bootstrap a Juju controller into the MicroK8s cloud:

juju bootstrap microk8s tutorial-controller

Add a workspace, or 'model':

juju add-model tutorial-model

Deploy, configure, and integrate a few things

Deploy Mattermost:

juju deploy mattermost-k8s

See more: Charmhub | mattermost-k8s

Deploy PostgreSQL:

juju deploy postgresql-k8s --channel 14/stable --trust

See more: Charmhub | postgresql-k8s

Enable security in your PostgreSQL deployment:

juju deploy tls-certificates-operator
juju config tls-certificates-operator generate-self-signed-certificates="true" ca-common-name="Test CA"
juju integrate postgresql-k8s tls-certificates-operator

Integrate Mattermost with PostgreSQL:

juju integrate mattermost-k8s postgresql-k8s:db

Watch your deployment come to life:

juju status --watch 1s

(Press Ctrl-C to quit. Drop the --watch 1s flag to get the status statically. Use the --relations flag to view more information about your integrations.)

Test your deployment

When everything is in active or idle status, note the IP address and port of Mattermost and pass them to curl:

curl <IP address>:<port>/api/v4/system/ping

You should see the output below:

{"AndroidLatestVersion":"","AndroidMinVersion":"","IosLatestVersion":"","IosMinVersion":"","status":"OK"}

Congratulations!

You now have a Kubernetes deployment consisting of a Mattermost backed by PosgreSQL with TLS-encrypted traffic!

Clean up

Delete your Multipass VM:

multipass delete --purge tutorial-vm

Uninstall Multipass: Linux | macOS | Windows. On Linux:

snap remove multipass

Next steps

Learn more

• Read our user docs
• Read our developer docs

Chat with us

Read our Code of conduct and:

• Join our chat: Matrix
• Join our forum: Discourse

File an issue

• Report a Juju bug on Launchpad
• Raise a general https://juju.is/docs documentation issue on Github | juju/docs

Make your mark

• Read our documentation contributor guidelines and help improve a doc
• Read our codebase contributor guidelines and help improve the codebase

charm-helpers's Issues

ansible-playbook can be more verbose

charm-helpers/charmhelpers/contrib/ansible/__init__.py

Lines 150 to 155 in 295afab

	call = [
	'ansible-playbook',
	'-c',
	'local',
	playbook,
	]

Currently there is no way to tell ansible-playbook to add more verbosity such as ansible-playbook -vv. It would be nice to have a switch to enable verbose mode to troubleshoot the playbook.

very easy to shoot yourself in the foot with hookenv.config()

As I discovered in canonical/layer-beats-base#18, it's very easy to shoot yourself in the foot with config().

If you call my_config = hookenv.config() and then modify an item of my_config, the mangled config will get written to disk, which will make the basic layer set the config.changed flag on the next hook run.

Unfortunately, I don't have a sensible solution to propose in order to fix this.

Does config('prefer-ipv6') retrieve option from model config?

I'd like to access the model configuration prefer-ipv6 from our charms. contrib.network.ip uses this, but I'm not sure if it works: https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/network/ip.py#L582

Setting model-config to true or false either gets me None, or when accessing the config dict, a KeyError. And if I have understood correctly, hookenv.config() should read the config.yaml of the charm.

Is there some kind of mix up with the OpenStack charms, as they seem to have a separate prefer-ipv6 option, or is my Juju version too old to support this?

https://github.com/openstack/charm-openstack-dashboard/blob/master/config.yaml#L238

Currently at 2.0 still.

keystone_wait_for_propagation has key error when relation key does not exist

In a circular twist of missing keys, the attempt to raise an exception for missing relation data results in a key error while trying to log details into the exception message.

DEBUG:runner:Traceback (most recent call last):
DEBUG:runner:  File "/tmp/bundletester-7ZZqdN/keystone/tests/gate-basic-trusty-kilo", line 24, in <module>
DEBUG:runner:    source='cloud:trusty-updates/kilo')
DEBUG:runner:  File "/tmp/bundletester-7ZZqdN/keystone/tests/basic_deployment.py", line 66, in __init__
DEBUG:runner:    self._initialize_test_differences()
DEBUG:runner:  File "/tmp/bundletester-7ZZqdN/keystone/tests/basic_deployment.py", line 331, in _initialize_test_differences
DEBUG:runner:    self.set_api_version(2)
DEBUG:runner:  File "/tmp/bundletester-7ZZqdN/keystone/tests/basic_deployment.py", line 197, in set_api_version
DEBUG:runner:    u.keystone_configure_api_version(se_rels, self, api_version)
DEBUG:runner:  File "/tmp/bundletester-7ZZqdN/keystone/tests/charmhelpers/contrib/openstack/amulet/utils.py", line 351, in keystone_configure_api_version
DEBUG:runner:    self.keystone_wait_for_propagation(sentry_relation_pairs, api_version)
DEBUG:runner:  File "/tmp/bundletester-7ZZqdN/keystone/tests/charmhelpers/contrib/openstack/amulet/utils.py", line 331, in keystone_wait_for_propagation
DEBUG:runner:    "".format(rel['api_version'], api_version))
DEBUG:runner:KeyError: 'api_version'
DEBUG:runner:Exit Code: 1

    def keystone_wait_for_propagation(self, sentry_relation_pairs,
                                      api_version):
        """Iterate over list of sentry and relation tuples and verify that
           api_version has the expected value.

        :param sentry_relation_pairs: list of sentry, relation name tuples used
                                      for monitoring propagation of relation
                                      data
        :param api_version: api_version to expect in relation data
        :returns: None if successful.  Raise on error.
        """
        for (sentry, relation_name) in sentry_relation_pairs:
            rel = sentry.relation('identity-service',
                                  relation_name)
            self.log.debug('keystone relation data: {}'.format(rel))
            if rel.get('api_version') != str(api_version):
                raise Exception("api_version not propagated through relation"
                                " data yet ('{}' != '{}')."
                                "".format(rel['api_version'], api_version))

• https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-keystone/557680/5/1317/consoleText.test_charm_amulet_full_2504.txt

• https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-keystone/557680/3/1316/consoleText.test_charm_amulet_full_2503.txt

copy_nrpe_checks doesn't verify the directory to copy from.

Due to LP:#1796830 it turns out that the copy_nrpe_checks() function assumes that charmhelpers will always be installed in the hooks directory (which was probably a reasonable assumption). However, certain issues around converting PY2 charms to PY3 charms has meant, that for some charms, charmhelpers is better placed in the CHARMDIR. This has the unfortunate effect of breaking copy_nrpe_checks.

Support for SSL configuration hardening in line with best practices

It would be great to support hardening SSL configuration for charms which make use of charm-helpers Apache configuration. This is a feature request from an Ubuntu OpenStack customer.

The options requested are

1. Setting SSLProtocol (original request is for TLSv1.2)
2. Setting SSLCipherSuite (original request is for a custom list)
3. Add a HSTS header

In terms of charms and opinionated configuration, I think ideally would be the ability to opt-in to a more secure set of defaults, in line perhaps with recommendations from https://mozilla.github.io/server-side-tls/ssl-config-generator/ or another similar source. To that end, I suggest we add an option such as ssl-security-level=old/intermediate/modern reflecting the configuration recommendations of the Mozilla Generator.

OpenStack also has some more simplistic recommendations for a very limited cipher suite for TLSv1.2
https://docs.openstack.org/security-guide/secure-communication/introduction-to-ssl-and-tls.html

I'd like to get input on on the best way to approach this in order to submit a Pull Request to add support. I welcome any input, but generally my thoughts are

1. What should we name the configuration option and values, should we use ssl-security-level=old/intermediate/modern in reflection of the mozilla set or use some other set (such as ssl-security-level=compatible/secure)
2. Should we support an override to specifically configure the SSLCipherSuite (as requested) instead of the default set we recommend. A better approach to avoid people 'over-optimising' the configuration might be to add support instead of a generic apache vhost configuration override (similar to 'extra-options' in some charms)
3. I would suggest in order to be most compatible with future software versions we should set SSLProtocol to "-SSLv3" or "-SSLv3 -TLSv1 -TLSv1.1" instead of "TLSv1.2" specifically - this allows for future protocol upgrades such as TLSv1.2 to be used with necessarily upgrading the config. This is perhaps less of a concern given we're likely to update the charm for later releases but it is better to be specifically future compatible by default instead of backwards breaking.

I plan to test OpenStack client versions from 12.04 through 18.04 with each of the settings so that we can fully understand the compatibility matrix before pushing the PR.

Contributing docs still refer to the LP site

https://github.com/juju/charm-helpers/blob/master/docs/contributing.rst

@cached mechanism broken in bionic (python 3.6)

Hi,

The @cached mechanism in core/hookenv.py is broken. The key for the cache dict is created as follows (from https://github.com/juju/charm-helpers/blob/master/charmhelpers/core/hookenv.py#L70) :

key = str((func, args, kwargs))

But there's a subtle difference in how str() called on a dict (kwargs, here) behaves between xenial and bionic.

On xenial :

 >>> a = {'rid': 'general-info:2', 'unit': 'ubuntu/0'}
>>> str(a)
"{'rid': 'general-info:2', 'unit': 'ubuntu/0'}"
>>> b = {'unit': 'ubuntu/0', 'rid': 'general-info:2'}
>>> str(b)
"{'rid': 'general-info:2', 'unit': 'ubuntu/0'}"
>>> str(a) == str(b)
True

On bionic :

>>> a = {'rid': 'general-info:2', 'unit': 'ubuntu/0'}
>>> str(a)
"{'rid': 'general-info:2', 'unit': 'ubuntu/0'}"
>>> b = {'unit': 'ubuntu/0', 'rid': 'general-info:2'}
>>> str(b)
"{'unit': 'ubuntu/0', 'rid': 'general-info:2'}"
>>> str(a) == str(b)
False

In other words, on xenial, the orders of the keys in the dict doesn't matter, but on bionic it does.

So for example, the relation_get() method being cached, if I call relation_get(rid=foo, unit=bar) and relation_get(unit=bar, rid=foo), on xenial they will have the same cache key, but on bionic they won't.

This shouldn't result in bugs however, just in slower calls. There is a twist however : some functions modify the object returned by the cached function call, which modify the object in the cache itself.

For example :

@cached
def relation_for_unit(unit=None, rid=None):
    """Get the json represenation of a unit's relation"""
    unit = unit or remote_unit()
    relation = relation_get(unit=unit, rid=rid)
    for key in relation:
        if key.endswith('-list'):
            relation[key] = relation[key].split()
    relation['__unit__'] = unit
return relation

(taken from https://github.com/juju/charm-helpers/blob/master/charmhelpers/core/hookenv.py#L466)

This will add a __unit___ key to the dict that is stored in the cache.

Now consider the following call, taken from https://github.com/juju/charm-helpers/blob/master/charmhelpers/core/services/helpers.py#L119 :

reldata = hookenv.relation_get(rid=rid, unit=unit)

On xenial, because str(a) == str(b) (see the first block of code), this call and relation_get(unit=unit, rid=rid) (kwargs in different order) will have the same key in the cache, and so will use the same cached object, containing __unit__. However on bionic, because str(a)!=str(b), these 2 calls will not have the same key, and so will not use the same object from the cache. So the reldata object above will never have a __unit__ key in it.

The short term fix I'm going to propose here is to have all calls to relation_get() in charmhelpers have the same order for kwargs.

For the long term, this caching technique must be changed. Perhaps using https://docs.python.org/dev/library/functools.html#functools.lru_cache ? I'll leave that up to you.

empty .juju-persistent-config breaks charm hooks

Realised I should have raised bug here instead of LP.

If for whatever reason you have an empty .juju-persistent-config state file (normally contains values of previously set config opts calls to Config.config() will fail with e.g.

2018-04-27 14:32:14 INFO config-changed Traceback (most recent call last):
2018-04-27 14:32:14 INFO config-changed File "/var/lib/juju/agents/unit-hacluster-rados-1/charm/hooks/config-changed", line 481, in <module>
2018-04-27 14:32:14 INFO config-changed hooks.execute(sys.argv)
2018-04-27 14:32:14 INFO config-changed File "/var/lib/juju/agents/unit-hacluster-rados-1/charm/hooks/charmhelpers/core/hookenv.py", line 805, in execute
2018-04-27 14:32:14 INFO config-changed self._hooks[hook_name]()
2018-04-27 14:32:14 INFO config-changed File "/var/lib/juju/agents/unit-hacluster-rados-1/charm/hooks/config-changed", line 157, in config_changed
2018-04-27 14:32:14 INFO config-changed update_nrpe_config()
2018-04-27 14:32:14 INFO config-changed File "/var/lib/juju/agents/unit-hacluster-rados-1/charm/hooks/config-changed", line 450, in update_nrpe_config
2018-04-27 14:32:14 INFO config-changed nrpe_setup = nrpe.NRPE(hostname=hostname)
2018-04-27 14:32:14 INFO config-changed File "/var/lib/juju/agents/unit-hacluster-rados-1/charm/hooks/charmhelpers/contrib/charmsupport/nrpe.py", line 244, in __init__
2018-04-27 14:32:14 INFO config-changed self.nagios_context = self.config['nagios_context']
2018-04-27 14:32:14 INFO config-changed TypeError: 'NoneType' object has no attribute '__getitem__'

Speedup assess_status() indirectly by changing pausing code to note use HookData()

Unfortunately, HookData() iterates through all of the relations which makes it very slow when it is used by the pause/unpause functions, and the assess_status() function template tests to see if the unit is supposed to be paused.

This can be speeded up by just using kv() directly. Issues are potentially around upgrades (although I can't think of one).

getting started needs update

Charm-helpers getting started doc seems to be a bit outdated. Can we can a refresh?

charmhelpers harden for suid_guid uses mlocate which may not always be installed

This is from: https://bugs.launchpad.net/charm-ceph-osd/+bug/1759145

Essentially, charmhelpers ought to check if it is available and complain/log/try to install it? (who knows what the package would be called if it's not an ubuntu minimal).

Relevant line:

charmhelpers/contrib/hardening/host/checks/suid_sgid.py
57:             '/usr/bin/mlocate',

Reactive charm no longer deploys with 0.18.3

My reactive charm built with layer:basic no longer installs. We've isolated the regression to the upgrade from charmhelpers 0.18.2 to 0.18.3.

2017-12-13 17:42:07 DEBUG install Traceback (most recent call last):
2017-12-13 17:42:07 DEBUG install   File "/var/lib/juju/agents/unit-livepatch-health-0/charm/hooks/install", line 8, in <module>
2017-12-13 17:42:07 DEBUG install     basic.bootstrap_charm_deps()
2017-12-13 17:42:07 DEBUG install   File "lib/charms/layer/basic.py", line 58, in bootstrap_charm_deps
2017-12-13 17:42:07 DEBUG install     cfg = layer.options('basic')
2017-12-13 17:42:07 DEBUG install AttributeError: module 'charms.layer' has no attribute 'options'
2017-12-13 17:42:07 ERROR juju.worker.uniter.operation runhook.go:113 hook "install" failed: exit status 1

charmsupport nrpe

https://github.com/juju/charm-helpers/blob/v0.18.2/charmhelpers/contrib/charmsupport/nrpe.py#L164-L177

What is the purpose of this function? This is just weird.

what this does, is to force the user to add all files to one of the directories in search_path.
If the file does not exist, check_cmd will be empty, nrpe will restart and break nrpe on this unit.

you cannot for example use 'sudo /usr/lib/nagios/plugins/check_something' as check_cmd, since sudo is not in the correct path. a workaround for this, is to create a wrapper which only function is to call 'sudo /usr/lib/nagios/plugins/check_something'

if one still need to micromanage the user, maybe extend the search_path to os.environ['PATH']
and if the command is not found, dont create the file with an empty check_cmd, which breaks nrpe. and be sure to log this action

ApacheSSLContext.enable_modules produces unnecessary output

This context function (which is run everytime the class instance is used) generates the following output:

Considering dependency setenvif for ssl:
Module setenvif already enabled
Considering dependency mime for ssl:
Module mime already enabled
Considering dependency socache_shmcb for ssl:
Module socache_shmcb already enabled
Module ssl already enabled
Module proxy already enabled
Considering dependency proxy for proxy_http:
Module proxy already enabled
Module proxy_http already enabled
Module headers already enabled

The function responsible is:

class ApacheSSLContext(OSContextGenerator):                                              

    ...

                                                                                  
     def enable_modules(self):                                                    
         cmd = ['a2enmod', 'ssl', 'proxy', 'proxy_http', 'headers']               
         check_call(cmd) 

If this were changed to:

     def enable_modules(self):                                                    
         cmd = ['a2enmod', 'ssl', 'proxy', 'proxy_http', 'headers', '1>/dev/null']               
         check_call(cmd, shell=True)

then the standard cruft would go, but we'd still get error messages. Thoughts?

hookenv.core.related_units fails when no relations exist

https://github.com/juju/charm-helpers/blob/master/charmhelpers/core/hookenv.py#L461

When removing a unit of the cs:graylog charm, I found that on my juju 2.2.4 environment that the "relation-list --format=json" command fails out with: ERROR no relation id specified and causes a Traceback failing the stop hook when disconnected from it's mongodb. I believe this is potentially a race condition in that code, however, I think charmhelpers should be updated to be aware that relation-list appears to return a bad result causing subprocess.check_output to dump.

Style on RTD docs is broken

There's some kind of issue with the formatting of https://charm-helpers.readthedocs.io/en/latest/ I don't see any theme-related error in the build log, nor do I see any file load errors when viewing the page, so I'm not sure what's going on. I tried rebuilding a different project to see if it was a systemic error, but that project still renders fine.

stack trace when writing rabbit_ssl_ca to the neutron-api config

2017-11-18 11:04:41 DEBUG juju-log amqp:46: Generating template context for amqp
2017-11-18 11:04:41 DEBUG amqp-relation-changed Traceback (most recent call last):
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/amqp-relation-changed", line 732, in
2017-11-18 11:04:41 DEBUG amqp-relation-changed main()
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/amqp-relation-changed", line 725, in main
2017-11-18 11:04:41 DEBUG amqp-relation-changed hooks.execute(sys.argv)
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/core/hookenv.py", line 797, in execute
2017-11-18 11:04:41 DEBUG amqp-relation-changed self._hookshook_name
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1890, in wrapped_f
2017-11-18 11:04:41 DEBUG amqp-relation-changed restart_functions)
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/core/host.py", line 728, in restart_on_change_helper
2017-11-18 11:04:41 DEBUG amqp-relation-changed r = lambda_f()
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1889, in
2017-11-18 11:04:41 DEBUG amqp-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/amqp-relation-changed", line 319, in amqp_changed
2017-11-18 11:04:41 DEBUG amqp-relation-changed if 'amqp' not in CONFIGS.complete_contexts():
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 304, in complete_contexts
2017-11-18 11:04:41 DEBUG amqp-relation-changed for i in six.itervalues(self.templates)]
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 304, in
2017-11-18 11:04:41 DEBUG amqp-relation-changed for i in six.itervalues(self.templates)]
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 124, in complete_contexts
2017-11-18 11:04:41 DEBUG amqp-relation-changed self.context()
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 109, in context
2017-11-18 11:04:41 DEBUG amqp-relation-changed _ctxt = context()
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/neutron_api_context.py", line 708, in call
2017-11-18 11:04:41 DEBUG amqp-relation-changed context = super(NeutronAMQPContext, self).call()
2017-11-18 11:04:41 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-0/charm/hooks/charmhelpers/contrib/openstack/context.py", line 463, in call
2017-11-18 11:04:41 DEBUG amqp-relation-changed fh.write(b64decode(ctxt['rabbit_ssl_ca']))
2017-11-18 11:04:41 DEBUG amqp-relation-changed TypeError: write() argument must be str, not bytes

charm-helpers assumes that metadata.yaml is ASCII

Hi,

I got the following traceback on a charm :

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/site-packages/charms/reactive/__init__.py", line 73, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/site-packages/charms/reactive/bus.py", line 382, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/site-packages/charms/reactive/bus.py", line 358, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/site-packages/charms/reactive/bus.py", line 180, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-foo-5/charm/reactive/telegraf.py", line 239, in configure_telegraf
    if get_remote_unit_name() is None:
  File "/var/lib/juju/agents/unit-foo-5/charm/reactive/telegraf.py", line 112, in get_remote_unit_name
    unit = hookenv.principal_unit()
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/site-packages/charmhelpers/core/hookenv.py", line 223, in principal_unit
    md = _metadata_unit(unit)
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/site-packages/charmhelpers/core/hookenv.py", line 546, in _metadata_unit
    return yaml.safe_load(md)
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/site-packages/yaml/__init__.py", line 94, in safe_load
    return load(stream, SafeLoader)
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/site-packages/yaml/__init__.py", line 70, in load
    loader = Loader(stream)
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/site-packages/yaml/loader.py", line 24, in __init__
    Reader.__init__(self, stream)
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/site-packages/yaml/reader.py", line 85, in __init__
    self.determine_encoding()
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/site-packages/yaml/reader.py", line 124, in determine_encoding
    self.update_raw()
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/site-packages/yaml/reader.py", line 178, in update_raw
    data = self.stream.read(size)
  File "/var/lib/juju/agents/unit-foo-5/.venv/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 694: ordinal not in range(128)

This is because the metadata file of a charm installed in the machine (elasticsearch, in this case) has a metadata.yaml encoded in UTF-8.

My charm isn't using the latest charm-helpers, but I can see that yaml.safe_load() is called the same way today : https://github.com/juju/charm-helpers/blob/master/charmhelpers/core/hookenv.py#L553

Could we please handle this better ?

Thanks

assess_status: hook tool failure result in Traceback and error state

Not sure how common this situation is, but it might be worth to probe if improved error handling is in order.

2018-08-22 04:16:53 DEBUG juju.worker.uniter.remotestate watcher.go:530 update status timer triggered
2018-08-22 04:16:53 DEBUG juju.worker.uniter resolver.go:133 no operations in progress; waiting for changes
2018-08-22 04:16:53 DEBUG juju.worker.uniter.operation executor.go:59 running operation run update-status hook
2018-08-22 04:16:53 DEBUG juju.machinelock machinelock.go:156 acquire machine lock for uniter (run update-status hook)
2018-08-22 04:16:53 DEBUG juju.machinelock machinelock.go:166 machine lock acquired for uniter (run update-status hook)
2018-08-22 04:16:53 DEBUG juju.worker.uniter.operation executor.go:90 preparing operation "run update-status hook"
2018-08-22 04:16:53 DEBUG juju.worker.uniter.operation executor.go:90 executing operation "run update-status hook"
2018-08-22 04:16:53 DEBUG worker.uniter.jujuc server.go:181 running hook tool "config-get"
2018-08-22 04:16:54 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:54 INFO juju-log Registered config file: /etc/keystone/keystone.conf
2018-08-22 04:16:54 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:54 INFO juju-log Registered config file: /etc/keystone/logging.conf
2018-08-22 04:16:54 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:54 INFO juju-log Registered config file: /etc/haproxy/haproxy.cfg
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 INFO juju-log Registered config file: /etc/apache2/sites-available/openstack_https_frontend.conf
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 INFO juju-log Registered config file: /etc/keystone/policy.json
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 INFO juju-log Registered config file: /etc/cron.d/keystone-token-flush
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 INFO juju-log Registered config file: /etc/cron.d/keystone-fernet-rotate-sync
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 INFO juju-log Registered config file: /etc/apache2/sites-enabled/wsgi-openstack-api.conf
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 INFO juju-log Registered config file: /etc/memcached.conf
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 DEBUG juju-log Hardening function 'install'
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 DEBUG juju-log Hardening function 'config_changed'
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 DEBUG juju-log Hardening function 'config_changed_postupgrade'
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 DEBUG juju-log Hardening function 'upgrade_charm'
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 DEBUG juju-log Hardening function 'update_status'
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 DEBUG juju-log No hardening applied to 'update_status'
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:55 INFO juju-log Updating status.
2018-08-22 04:16:55 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-ids"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-ids"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-ids"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-ids"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-ids"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-ids"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-ids"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-ids"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-ids"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-get"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-list"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-get"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-ids"
2018-08-22 04:16:56 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-ids"
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-ids"
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-get"
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-list"
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:57 INFO juju-log Configuring Keystone to use a random admin token.
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:57 INFO juju-log Loading a previously generated admin token from /var/lib/keystone/keystone.token
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "leader-get"
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "leader-get"
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "leader-get"
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "unit-get"
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "network-get"
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "unit-get"
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "network-get"
2018-08-22 04:16:57 DEBUG worker.uniter.jujuc server.go:181 running hook tool "relation-get"
2018-08-22 04:16:57 DEBUG update-status ERROR no relation id specified
2018-08-22 04:16:58 DEBUG worker.uniter.jujuc server.go:181 running hook tool "network-get"
2018-08-22 04:16:58 DEBUG worker.uniter.jujuc server.go:181 running hook tool "network-get"
2018-08-22 04:16:58 DEBUG worker.uniter.jujuc server.go:181 running hook tool "network-get"
2018-08-22 04:16:58 DEBUG worker.uniter.jujuc server.go:181 running hook tool "network-get"
2018-08-22 04:16:58 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:58 INFO juju-log HAProxy context is incomplete, this unit has no peers.
2018-08-22 04:16:58 DEBUG update-status none
2018-08-22 04:16:58 DEBUG update-status none
2018-08-22 04:16:58 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:58 INFO juju-log Configuring Keystone to use a random admin token.
2018-08-22 04:16:58 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:58 INFO juju-log Loading a previously generated admin token from /var/lib/keystone/keystone.token
2018-08-22 04:16:58 DEBUG worker.uniter.jujuc server.go:181 running hook tool "leader-get"
2018-08-22 04:16:58 DEBUG worker.uniter.jujuc server.go:181 running hook tool "leader-get"
2018-08-22 04:16:58 DEBUG worker.uniter.jujuc server.go:181 running hook tool "leader-get"
2018-08-22 04:16:58 DEBUG worker.uniter.jujuc server.go:181 running hook tool "network-get"
2018-08-22 04:16:58 DEBUG worker.uniter.jujuc server.go:181 running hook tool "network-get"
2018-08-22 04:16:59 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:59 INFO juju-log Configuring Keystone to use a random admin token.
2018-08-22 04:16:59 DEBUG worker.uniter.jujuc server.go:181 running hook tool "juju-log"
2018-08-22 04:16:59 INFO juju-log Loading a previously generated admin token from /var/lib/keystone/keystone.token
2018-08-22 04:16:59 DEBUG worker.uniter.jujuc server.go:181 running hook tool "leader-get"
2018-08-22 04:16:59 DEBUG worker.uniter.jujuc server.go:181 running hook tool "leader-get"
2018-08-22 04:16:59 DEBUG worker.uniter.jujuc server.go:181 running hook tool "leader-get"
2018-08-22 04:16:59 DEBUG worker.uniter.jujuc server.go:181 running hook tool "network-get"
2018-08-22 04:16:59 DEBUG worker.uniter.jujuc server.go:181 running hook tool "network-get"
2018-08-22 04:17:00 DEBUG juju.api monitor.go:35 RPC connection died
2018-08-22 04:17:00 DEBUG juju.rpc server.go:325 error closing codec: write tcp 252.0.17.1:39720->252.0.9.1:17070: write: broken pipe
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "log-sender" manifold worker stopped: cannot send log message: tls: use of closed connection
2018-08-22 04:17:00 ERROR juju.worker.dependency engine.go:548 "log-sender" manifold worker returned unexpected error: cannot send log message: tls: use of closed connection
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:550 stack trace:
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "upgrader" manifold worker stopped: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 ERROR juju.worker.dependency engine.go:548 "upgrader" manifold worker returned unexpected error: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:550 stack trace:
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "migration-inactive-flag" manifold worker stopped: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 ERROR juju.worker.dependency engine.go:548 "migration-inactive-flag" manifold worker returned unexpected error: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:550 stack trace:
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "api-caller" manifold worker stopped: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 ERROR juju.worker.dependency engine.go:548 "api-caller" manifold worker returned unexpected error: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:550 stack trace:
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "logging-config-updater" manifold worker stopped: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "meter-status" manifold worker stopped: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "proxy-config-updater" manifold worker stopped: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "migration-minion" manifold worker stopped: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 DEBUG juju.worker.uniter runlistener.go:120 juju-run listener stopping
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "hook-retry-strategy" manifold worker stopped: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "api-address-updater" manifold worker stopped: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "upgrader" manifold worker stopped: <nil>
2018-08-22 04:17:00 DEBUG juju.worker.uniter runlistener.go:139 juju-run listener stopped
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "log-sender" manifold worker stopped: <nil>
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "migration-inactive-flag" manifold worker stopped: <nil>
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "metric-collect" manifold worker stopped: <nil>
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "leadership-tracker" manifold worker stopped: <nil>
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "charm-dir" manifold worker stopped: <nil>
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "metric-sender" manifold worker stopped: <nil>
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "metric-spool" manifold worker stopped: <nil>
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "metric-collect" manifold worker stopped: <nil>
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "metric-sender" manifold worker stopped: <nil>
2018-08-22 04:17:00 DEBUG update-status Traceback (most recent call last):
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/update-status", line 819, in <module>
2018-08-22 04:17:00 DEBUG update-status     main()
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/update-status", line 815, in main
2018-08-22 04:17:00 DEBUG update-status     assess_status(CONFIGS)
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1824, in assess_status
2018-08-22 04:17:00 DEBUG update-status     assess_status_func(configs)()
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1409, in _assess_status_func
2018-08-22 04:17:00 DEBUG update-status     state, message = _determine_os_workload_status(*args, **kwargs)
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 812, in _determine_os_workload_status
2018-08-22 04:17:00 DEBUG update-status     configs, required_interfaces)
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 866, in _ows_check_generic_interfaces
2018-08-22 04:17:00 DEBUG update-status     required_interfaces)
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1147, in incomplete_relation_data
2018-08-22 04:17:00 DEBUG update-status     complete_ctxts = configs.complete_contexts()
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 351, in complete_contexts
2018-08-22 04:17:00 DEBUG update-status     for i in six.itervalues(self.templates)]
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 127, in complete_contexts
2018-08-22 04:17:00 DEBUG update-status     self.context()
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 112, in context
2018-08-22 04:17:00 DEBUG update-status     _ctxt = context()
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_context.py", line 204, in __call__
2018-08-22 04:17:00 DEBUG update-status     resolve_address(ADMIN),
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/ip.py", line 177, in resolve_address
2018-08-22 04:17:00 DEBUG update-status     resolved_address = network_get_primary_address(binding)
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 952, in inner_translate_exc2
2018-08-22 04:17:00 DEBUG update-status     return f(*args, **kwargs)
2018-08-22 04:17:00 DEBUG update-status   File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 1147, in network_get_primary_address
2018-08-22 04:17:00 DEBUG update-status     stderr=subprocess.STDOUT).decode('UTF-8').strip()
2018-08-22 04:17:00 DEBUG update-status   File "/usr/lib/python2.7/subprocess.py", line 223, in check_output
2018-08-22 04:17:00 DEBUG update-status     raise CalledProcessError(retcode, cmd, output=output)
2018-08-22 04:17:00 DEBUG update-status subprocess.CalledProcessError: Command '['network-get', '--primary-address', 'admin']' returned non-zero exit status 1
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "upgrader" manifold worker stopped: "api-caller" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "proxy-config-updater" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "metric-spool" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "hook-retry-strategy" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "metric-sender" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "log-sender" manifold worker stopped: "api-caller" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "api-address-updater" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "logging-config-updater" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "charm-dir" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "upgrade-steps-runner" manifold worker stopped: "api-caller" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "leadership-tracker" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "migration-inactive-flag" manifold worker stopped: "api-caller" not running: dependency not available
2018-08-22 04:17:00 ERROR juju.worker.uniter.operation runhook.go:115 hook "update-status" failed: exit status 1
2018-08-22 04:17:00 DEBUG juju.machinelock machinelock.go:180 machine lock released for uniter (run update-status hook)
2018-08-22 04:17:00 DEBUG juju.worker.uniter.operation executor.go:74 lock released
2018-08-22 04:17:00 INFO juju.worker.uniter uniter.go:397 unit "keystone/0" shutting down: (re)starting watcher: connection is shut down
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "uniter" manifold worker stopped: codec.ReadHeader error: error receiving message: read tcp 252.0.17.1:39720->252.0.9.1:17070: read: connection reset by peer
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "metric-collect" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "migration-minion" manifold worker stopped: "api-caller" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "meter-status" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:00 DEBUG juju.worker.dependency engine.go:501 "uniter" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:03 DEBUG juju.worker.apicaller connect.go:112 connecting with current password
2018-08-22 04:17:03 DEBUG juju.api apiclient.go:872 successfully dialed "wss://10.5.0.9:17070/model/beb7b09c-f7ac-40f8-83bf-8129af6225c4/api"
2018-08-22 04:17:03 INFO juju.api apiclient.go:599 connection established to "wss://10.5.0.9:17070/model/beb7b09c-f7ac-40f8-83bf-8129af6225c4/api"
2018-08-22 04:17:03 DEBUG juju.api apiclient.go:872 successfully dialed "wss://252.0.9.1:17070/model/beb7b09c-f7ac-40f8-83bf-8129af6225c4/api"
2018-08-22 04:17:03 DEBUG juju.worker.apicaller connect.go:155 connected
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "api-caller" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "migration-minion" manifold worker started
2018-08-22 04:17:03 INFO juju.worker.migrationminion worker.go:116 migration phase is now: NONE
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "metric-sender" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "hook-retry-strategy" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "upgrader" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "log-sender" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "proxy-config-updater" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "logging-config-updater" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "api-address-updater" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "leadership-tracker" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "uniter" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "meter-status" manifold worker stopped: "migration-inactive-flag" not running: dependency not available
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "upgrade-steps-runner" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "upgrade-steps-runner" manifold worker stopped: <nil>
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "migration-inactive-flag" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "meter-status" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "proxy-config-updater" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "api-address-updater" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "charm-dir" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "uniter" manifold worker stopped: <nil>
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "metric-collect" manifold worker stopped: <nil>
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "metric-spool" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "metric-sender" manifold worker stopped: <nil>
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "metric-collect" manifold worker stopped: <nil>
2018-08-22 04:17:03 DEBUG juju.worker.logger logger.go:43 initial log config: "<root>=DEBUG;unit=DEBUG"
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "logging-config-updater" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.logger logger.go:92 logger setup
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "leadership-tracker" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "uniter" manifold worker stopped: <nil>
2018-08-22 04:17:03 DEBUG juju.worker.leadership tracker.go:130 keystone/0 making initial claim for keystone leadership
2018-08-22 04:17:03 DEBUG juju.worker.proxyupdater proxyupdater.go:168 applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.5.0.9,127.0.0.1,252.0.9.1,::1,localhost", AutoNoProxy:""}
2018-08-22 04:17:03 DEBUG juju.worker.proxyupdater proxyupdater.go:188 saving new legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.5.0.9,127.0.0.1,252.0.9.1,::1,localhost", AutoNoProxy:""}
2018-08-22 04:17:03 DEBUG juju.worker.proxyupdater proxyupdater.go:252 new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
2018-08-22 04:17:03 INFO juju.worker.upgrader upgrader.go:139 abort check blocked until version event received
2018-08-22 04:17:03 INFO juju.worker.upgrader upgrader.go:145 unblocking abort check
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "metric-sender" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "hook-retry-strategy" manifold worker started
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:501 "uniter" manifold worker stopped: <nil>
2018-08-22 04:17:03 DEBUG juju.worker.meterstatus connected.go:88 got meter status change signal from watcher
2018-08-22 04:17:03 DEBUG juju.network network.go:507 no lxc bridge addresses to filter for machine
2018-08-22 04:17:03 DEBUG juju.network network.go:543 cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
2018-08-22 04:17:03 DEBUG juju.network network.go:543 cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
2018-08-22 04:17:03 DEBUG juju.network network.go:492 including address local-cloud:10.5.0.9 for machine
2018-08-22 04:17:03 DEBUG juju.network network.go:492 including address local-fan:252.0.9.1 for machine
2018-08-22 04:17:03 DEBUG juju.network network.go:492 including address local-machine:127.0.0.1 for machine
2018-08-22 04:17:03 DEBUG juju.network network.go:492 including address local-machine:::1 for machine
2018-08-22 04:17:03 DEBUG juju.network network.go:561 addresses after filtering: [local-cloud:10.5.0.9 local-fan:252.0.9.1 local-machine:127.0.0.1 local-machine:::1]
2018-08-22 04:17:03 DEBUG juju.worker.apiaddressupdater apiaddressupdater.go:88 updating API hostPorts to [[10.5.0.9:17070 252.0.9.1:17070 127.0.0.1:17070 [::1]:17070]]
2018-08-22 04:17:03 DEBUG juju.agent agent.go:578 API server address details [["10.5.0.9:17070" "252.0.9.1:17070" "127.0.0.1:17070" "[::1]:17070"]] written to agent config as ["10.5.0.9:17070" "252.0.9.1:17070"]
2018-08-22 04:17:03 DEBUG juju.worker.proxyupdater proxyupdater.go:168 applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.5.0.9,127.0.0.1,252.0.9.1,::1,localhost", AutoNoProxy:""}
2018-08-22 04:17:03 INFO juju.worker.leadership tracker.go:199 keystone/0 promoted to leadership of keystone
2018-08-22 04:17:03 INFO juju.worker.upgrader upgrader.go:178 desired agent binary version: 2.5-beta1
2018-08-22 04:17:03 DEBUG juju.worker.dependency engine.go:483 "uniter" manifold worker started
2018-08-22 04:17:04 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-keystone-0
2018-08-22 04:17:04 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/2.5-beta1-bionic-amd64
2018-08-22 04:17:04 DEBUG juju.agent.tools symlinks.go:44 jujud path /var/lib/juju/tools/2.5-beta1-bionic-amd64/jujud
2018-08-22 04:17:04 INFO juju.worker.uniter.relation relations.go:495 joining relation "keystone:cluster"
2018-08-22 04:17:04 INFO juju.worker.upgrader upgrader.go:178 desired agent binary version: 2.5-beta1
2018-08-22 04:17:04 INFO juju.worker.uniter.relation relations.go:531 joined relation "keystone:cluster"
2018-08-22 04:17:04 INFO juju.worker.upgrader upgrader.go:178 desired agent binary version: 2.5-beta1
2018-08-22 04:17:04 INFO juju.worker.uniter.relation relations.go:495 joining relation "keystone:shared-db mysql:shared-db"
2018-08-22 04:17:04 INFO juju.worker.uniter.relation relations.go:531 joined relation "keystone:shared-db mysql:shared-db"
2018-08-22 04:17:04 INFO juju.worker.upgrader upgrader.go:178 desired agent binary version: 2.5-beta1
2018-08-22 04:17:04 DEBUG juju.worker.uniter uniter.go:579 starting juju-run listener on unix:/var/lib/juju/agents/unit-keystone-0/run.socket
2018-08-22 04:17:04 DEBUG juju.worker.uniter runlistener.go:107 juju-run listener running
2018-08-22 04:17:04 INFO juju.worker.uniter uniter.go:197 unit "keystone/0" started
2018-08-22 04:17:04 INFO juju.worker.uniter uniter.go:236 hooks are retried false
2018-08-22 04:17:04 INFO juju.worker.upgrader upgrader.go:178 desired agent binary version: 2.5-beta1
2018-08-22 04:17:04 DEBUG juju.worker.uniter.remotestate watcher.go:431 got address change: ok=true
2018-08-22 04:17:04 DEBUG juju.worker.uniter.remotestate watcher.go:402 got application change
2018-08-22 04:17:04 DEBUG juju.worker.uniter.remotestate watcher.go:451 got action change: [] ok=true
2018-08-22 04:17:04 DEBUG juju.worker.uniter.remotestate watcher.go:471 got storage change: [] ok=true
2018-08-22 04:17:04 DEBUG juju.worker.uniter.remotestate watcher.go:422 got upgrade series change
2018-08-22 04:17:04 DEBUG juju.worker.uniter.remotestate watcher.go:441 got leader settings change: ok=true
2018-08-22 04:17:04 DEBUG juju.worker.uniter.remotestate watcher.go:375 got config change: ok=true
2018-08-22 04:17:04 DEBUG juju.worker.uniter.remotestate watcher.go:461 got relations change: ok=true
2018-08-22 04:17:04 DEBUG juju.worker.uniter.remotestate watcher.go:392 got unit change
2018-08-22 04:17:04 DEBUG juju.worker.uniter.remotestate watcher.go:481 got update status interval change: ok=true
2018-08-22 04:17:04 DEBUG juju.worker.uniter.remotestate watcher.go:375 got config change: ok=true
2018-08-22 04:17:04 INFO juju.worker.uniter resolver.go:112 awaiting error resolution for "update-status" hook
2018-08-22 04:17:04 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] error: hook failed: "update-status"
2018-08-22 04:17:04 DEBUG juju.worker.dependency engine.go:483 "metric-collect" manifold worker started

Should we drop "proposed" from fetch.ubuntu.add_source() ?

I discovered that during some testing of the nova-compute charm, that "proposed" and "distro-proposed" are identical from the perspective of fetch.ubuntu.add_source(...), but they are treated differently by contrib.openstack.utils.get_os_codename_install_source(...).

This means that if you use "proposed" as config('openstack-origin') it will fail, but if you use `distro-"proposed" the charm works as it should.

I think we should drop "proposed" from add_source but that may break charm configs during upgrades (usually when the config-changed hook is run). The alternative is to add "proposed" as an option to the get_os_codename_install_source(...) function.

Relevant code is here:

fetch.ubuntu.add_source(...)

    ...
    _mapping = OrderedDict([
        (r"^distro$", lambda: None),  # This is a NOP
        (r"^(?:proposed|distro-proposed)$", _add_proposed),
        (r"^cloud-archive:(.*)$", _add_apt_repository),
        (r"^((?:deb |http:|https:|ppa:).*)$", _add_apt_repository),
        (r"^cloud:(.*)-(.*)\/staging$", _add_cloud_staging),
        (r"^cloud:(.*)-(.*)$", _add_cloud_distro_check),
        (r"^cloud:(.*)$", _add_cloud_pocket),
        (r"^snap:.*-(.*)-(.*)$", _add_cloud_distro_check),
    ])
    ...

get_os_codename_install_source(...)

def get_os_codename_install_source(src):
    '''Derive OpenStack release codename from a given installation source.'''
    ubuntu_rel = lsb_release()['DISTRIB_CODENAME']
    rel = ''
    if src is None:
        return rel
    if src in ['distro', 'distro-proposed']:
        try:
    ...

need to provide volume api version for internal endpoint context

... in contrib.openstack.context.InternalEndpointContext

Ought to introduce a helper for "release = lsb_release()['DISTRIB_CODENAME'].lower()"

charm-helpers itself, and lots of charms, use this exact line:

release = lsb_release()['DISTRIB_CODENAME'].lower()

There ought to be a helper that does this, and do it through the code base.

Note, that this will also have to be done for the centos host_factory too.

Nrpe checks in files/nrpe-external-master checks will not be installed

The documentation in line

charm-helpers/charmhelpers/contrib/charmsupport/nrpe.py

Line 83 in 117dfa3

# 3. Add custom checks (Nagios plugins) to files/nrpe-external-master

seems to suggest that one should place his custom nagios plugins in that directory.

But copy_nrpe_checks() doesn't actually use this directory. It only copies stuff from contrib/openstack/files

I would find it very useful if there where code that will copy the files for me automatically.

The directory name suggested in the docu string is not great though. Since it doesn't really matter if it is local_monitors or nrpe_external_master

fetch.ubuntu.add_source should support {series}

originally reported here

When adding a source via a default config value that gets passed to configure_sources which requires the series to be specified, if the charm supports multiple series, it becomes difficult or impossible to use the correct series. Ideally, add_source would support a {series} placeholder which would get populated correctly.

ubuntu-cloud-keyring may not be installed if apt update is required

I am seeing an issue specifically in lxd containers on s390x arch where a manual apt-get update is required before ubuntu-cloud-keyring can be installed. Perhaps an apt update should be run before attempting to install any packages?

charm-helpers/charmhelpers/fetch/ubuntu.py

Line 452 in e1dc816

apt_install(filter_installed_packages(['ubuntu-cloud-keyring']),

CompareHostReleases needs cosmic series support

2018-09-18 12:47:49 DEBUG install none
2018-09-18 12:47:49 DEBUG install Traceback (most recent call last):
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/charmhelpers/core/strutils.py", line 92, in __init__
2018-09-18 12:47:49 DEBUG install     self.index = self._list.index(item)
2018-09-18 12:47:49 DEBUG install ValueError: tuple.index(x): x not in tuple
2018-09-18 12:47:49 DEBUG install 
2018-09-18 12:47:49 DEBUG install During handling of the above exception, another exception occurred:
2018-09-18 12:47:49 DEBUG install 
2018-09-18 12:47:49 DEBUG install Traceback (most recent call last):
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/hooks/install.real", line 644, in <module>
2018-09-18 12:47:49 DEBUG install     assess_status(CONFIGS)
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 440, in assess_status
2018-09-18 12:47:49 DEBUG install     assess_status_func(configs)()
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/charmhelpers/contrib/openstack/utils.py", line 1422, in _assess_status_func
2018-09-18 12:47:49 DEBUG install     state, message = _determine_os_workload_status(*args, **kwargs)
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/charmhelpers/contrib/openstack/utils.py", line 812, in _determine_os_workload_status
2018-09-18 12:47:49 DEBUG install     configs, required_interfaces)
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/charmhelpers/contrib/openstack/utils.py", line 879, in _ows_check_generic_interfaces
2018-09-18 12:47:49 DEBUG install     required_interfaces)
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/charmhelpers/contrib/openstack/utils.py", line 1160, in incomplete_relation_data
2018-09-18 12:47:49 DEBUG install     complete_ctxts = configs.complete_contexts()
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/charmhelpers/contrib/openstack/templating.py", line 351, in complete_contexts
2018-09-18 12:47:49 DEBUG install     for i in six.itervalues(self.templates)]
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/charmhelpers/contrib/openstack/templating.py", line 351, in <listcomp>
2018-09-18 12:47:49 DEBUG install     for i in six.itervalues(self.templates)]
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/charmhelpers/contrib/openstack/templating.py", line 127, in complete_contexts
2018-09-18 12:47:49 DEBUG install     self.context()
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/charmhelpers/contrib/openstack/templating.py", line 112, in context
2018-09-18 12:47:49 DEBUG install     _ctxt = context()
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/charmhelpers/contrib/openstack/context.py", line 1868, in __call__
2018-09-18 12:47:49 DEBUG install     if CompareHostReleases(release) > 'trusty':
2018-09-18 12:47:49 DEBUG install   File "/var/lib/juju/agents/unit-glance-0/charm/charmhelpers/core/strutils.py", line 95, in __init__
2018-09-18 12:47:49 DEBUG install     .format(item, self._list))
2018-09-18 12:47:49 DEBUG install KeyError: "Item 'cosmic' is not in list '('lucid', 'maverick', 'natty', 'oneiric', 'precise', 'quantal', 'raring', 'saucy', 'trusty', 'utopic', 'vivid', 'wily', 'xenial', 'yakkety', 'zesty', 'artful', 'bionic')'"
2018-09-18 12:47:49 ERROR juju.worker.uniter.operation runhook.go:115 hook "install" failed: exit status 1

distributed_wait function in hahelpers/cluster.py isn't optimal for non-sequential unit numbers

distributed_wait function in contrib/hahelpers/cluster.py relies on function modulo_distribution in core/host.py.

The modulo_distribution function assumes the unit numbers will be close together, say units 0,1,2.
However, if you add and then remove units, you may have say 3 nodes of units 0, 3, 6 which all give the same wait time when using a modulo of 3.
For tasks like service restarts this can cause some patterns of unit numbers to all restart at the same time which isn't the desired outcome!

A potential alternative solution would be for each charm to use a peer relation (existing or new) to work out its sequenced wait time based on the number of units.

rabbit_hosts and rabbit_host options are deprecated for removal: [DEFAULT]/transport_url is needed

See https://bugs.launchpad.net/layer-openstack/+bug/1733072

Valid for both classic and reactive openstack charms.

version-specific sections: renaming of "[DEFAULT]/notification_topics" to "[oslo_messaging_notifications]/topics" in section-oslo-notifications

This commit https://git.io/vFyPE deprecated "[DEFAULT]/notification_topics" in favor of [oslo_messaging_notifications]/topics.

-    cfg.ListOpt('notification_topics',
+    cfg.ListOpt('topics',
                 default=['notifications', ],
-                deprecated_name='topics',
-                deprecated_group='rpc_notifier2',
+                deprecated_opts=[
+                    cfg.DeprecatedOpt('topics',
+                                      group='rpc_notifier2'),
+                    cfg.DeprecatedOpt('notification_topics',
+                                      group='DEFAULT')
+                ],

Kilo uses oslo.messaging versions older than 3.1.0
https://releases.openstack.org/kilo/index.html#library-projects

Same with Icehouse
https://releases.openstack.org/icehouse/index.html#library-projects

git tag --contains 33c1010c3281804456a22b769c4bac5ac6a7cca1
3.1.0
4.0.0
...

Kilo looks to have extended support until mid-2018 and Icehouse until Q1-2019:
https://www.ubuntu.com/info/release-end-of-life

Mitaka supports oslo.messaging 4.6.1 as the latest version:
https://releases.openstack.org/mitaka/index.html#library-projects

[DEFAULT]/notification_topics is not deprecated for removal so we can still use it. It does pollute logs quite extensively though with deprecation messages.

I can create a PR for changing that but the question is Icehouse & Kilo support.

10periodic Extra junk at end of file Hangs Units

When ubuntu.py (correctly) finds junk in /etc/apt/apt.conf.d/10periodic this hangs the unit:

2018-02-16 04:18:03 INFO update-status * memcached is running
2018-02-16 04:18:03 INFO juju-log Unit is ready
2018-02-16 04:18:04 INFO juju-log Application Version: 6.1.5
2018-02-16 04:23:04 INFO update-status Traceback (most recent call last):
2018-02-16 04:23:04 INFO update-status File "/var/lib/juju/agents/unit-ceilometer-agent-27/charm/hooks/update-status", line 49, in
2018-02-16 04:23:04 INFO update-status CONFIGS = register_configs()
2018-02-16 04:23:04 INFO update-status File "/var/lib/juju/agents/unit-ceilometer-agent-27/charm/hooks/ceilometer_utils.py", line 98, in register_configs
2018-02-16 04:23:04 INFO update-status release = get_os_codename_package('ceilometer-common', fatal=False)
2018-02-16 04:23:04 INFO update-status File "/var/lib/juju/agents/unit-ceilometer-agent-27/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 339, in get_os_codename_package
2018-02-16 04:23:04 INFO update-status cache = apt_cache()
2018-02-16 04:23:04 INFO update-status File "/var/lib/juju/agents/unit-ceilometer-agent-27/charm/hooks/charmhelpers/fetch/ubuntu.py", line 140, in apt_cache
2018-02-16 04:23:04 INFO update-status from apt import apt_pkg
2018-02-16 04:23:04 INFO update-status File "/usr/lib/python2.7/dist-packages/apt/init.py", line 34, in
2018-02-16 04:23:04 INFO update-status apt_pkg.init_config()
2018-02-16 04:23:04 INFO update-status SystemError: E:Syntax error /etc/apt/apt.conf.d/10periodic:2: Extra junk at end of file
2018-02-16 04:23:04 ERROR juju.worker.uniter.operation runhook.go:107 hook "update-status" failed: exit status 1

In these cases (plural is correct), where the issue with 10periodic was transient, the unit does not run again until prompted manually, such as by "juju resolved -r ceilometer-agent/27" in the above example, and the error state is cleared and the unit returns to normal function without further intervention:

8-02-16 04:23:04 INFO update-status SystemError: E:Syntax error /etc/apt/apt.conf.d/10periodic:2: Extra junk at end of file
2018-02-16 04:23:04 ERROR juju.worker.uniter.operation runhook.go:107 hook "update-status" failed: exit status 1
2018-03-07 02:06:58 INFO juju-log Registered config file: /etc/ceilometer/ceilometer.conf
2018-03-07 02:06:58 INFO juju-log Registered config file: /etc/memcached.conf
2018-03-07 02:07:04 INFO juju-log Unit is ready
2018-03-07 02:07:06 INFO juju-log Application Version: 6.1.5
2018-03-07 02:12:13 INFO juju-log Registered config file: /etc/ceilometer/ceilometer.conf
2018-03-07 02:12:13 INFO juju-log Registered config file: /etc/memcached.conf
2018-03-07 02:12:20 INFO update-status * memcached is running
2018-03-07 02:12:20 INFO juju-log Unit is ready
2018-03-07 02:12:22 INFO juju-log Application Version: 6.1.5
2018-03-07 02:17:23 INFO juju-log Registered config file: /etc/ceilometer/ceilometer.conf
2018-03-07 02:17:23 INFO juju-log Registered config file: /etc/memcached.conf
2018-03-07 02:17:30 INFO update-status * memcached is running
2018-03-07 02:17:30 INFO juju-log Unit is ready
2018-03-07 02:17:32 INFO juju-log Application Version: 6.1.5

Happy to provide more information if required.

Thank you.

pip3 install dirties dist-packages

Description

After pip installing charm-helpers, my system's dist-packages directory has a 'tools' and 'tests' directory from charmhelpers. In doing so other python3 software that is trying to import "tests" or "tools" fails to execute because of this.

Expected Results

No tests or tools directories from pycharms in my dist-packages directory.

Actual Results

A tests and tools directory that override various namespaces, preventing the importing of those same directories by other tools.

Steps to Reproduce

1. lxc launch ubuntu-daily:x x
2. lxc exec x bash
3. apt update
4. apt install -y python3-pip
5. pip install charm-helpers
6. Notice tests and tools directory under /usr/local/lib/python3.5/dist-packages

Example: https://paste.ubuntu.com/26485423/

Support goal-state command

Juju 2.4 introduced the goal-state hook tool. We should support this in hookenv.

Addition of second stats socket line breaks on Trusty.

Commit 4abc9cd from #115 breaks on Trusty.

Haproxy fails to start

PR#58 breaks cert-file comparisons as one is binary and the other is text

Due to the PY3 changes in PR #58 the switch from text to binary has broken usage in charms.

We could easily to revert to just using unicode strings rather than writing in binary and then it ought to work fine. However, would need to check that it actually works in, say, neutron-api charm first.

os_release incorrectly detects openstack version by keystone-middleware package if openstack-origin is not present

For most of the OpenStack charms openstack-origin is present, however, subordinates may not always have this option because they rely on a primary charm's versions (e.g. keystone-ldap charm) given that they share a machine and debian packages (not sure how snaps will be handled in this case).

https://github.com/juju/charm-helpers/blob/v0.18.3/charmhelpers/contrib/openstack/utils.py#L514-L535

charms.openstack has default_select_release which uses python-keystonemiddleware as a package to be passed to os_release unconditionally:
https://github.com/juju/charm-helpers/blob/v0.18.3/charmhelpers/contrib/openstack/utils.py#L532
https://github.com/openstack/charms.openstack/blob/389d928ba50915560b2140e022a663035fa86a4b/charms_openstack/charm/defaults.py#L82-L101

There is no definition for python-keystonemiddleware in PACKAGE_CODENAMES
https://github.com/juju/charm-helpers/blob/v0.18.3/charmhelpers/contrib/openstack/utils.py#L197

which is why OPENSTACK_CODENAMES are used:

https://github.com/juju/charm-helpers/blob/v0.18.3/charmhelpers/contrib/openstack/utils.py#L466-L480
https://github.com/juju/charm-helpers/blob/v0.18.3/charmhelpers/contrib/openstack/utils.py#L147-L162

For python-keystonemiddleware this results in KeyError because it uses different versions from the ones in OPENSTACK_CODENAMES (. versioning has no relevance):
https://github.com/juju/charm-helpers/blob/v0.18.3/charmhelpers/contrib/openstack/utils.py#L476-L480
A sample unfiltered trace (search for "KeyError: '4.14'"):
https://gist.github.com/dshcherb/4789ea8c16db45617b68bdd6bb59e656

And, finally, _os_rel == base == 'essex' is returned from os_release:
https://github.com/juju/charm-helpers/blob/v0.18.3/charmhelpers/contrib/openstack/utils.py#L534

This is quite problematic as default_select_release release selector from charms.openstack doesn't take release_pkg into account even if it is present in a charm class definition (because a handler is not aware of the charm class in use): https://bugs.launchpad.net/charms.openstack/+bug/1741628

hookenv.network_get does not handle exceptions on Juju 1.25.*

Using hookenv.network_get fails on Juju 1.25.* with a FileNotFoundError exception. This is not handled by the error handling, leading to failed hooks when this function is used on juju 1.25.

2018-02-08 03:48:29 INFO upgrade-charm Traceback (most recent call last):
2018-02-08 03:48:29 INFO upgrade-charm File "/var/lib/juju/agents/unit-nrpe-lxc-46/charm/hooks/charmhelpers/core/hookenv.py", line 929, in inner_translate_exc2
2018-02-08 03:48:29 INFO upgrade-charm return f(*args, **kwargs)
2018-02-08 03:48:29 INFO upgrade-charm File "/var/lib/juju/agents/unit-nrpe-lxc-46/charm/hooks/charmhelpers/core/hookenv.py", line 1116, in network_get_primary_address
2018-02-08 03:48:29 INFO upgrade-charm stderr=subprocess.STDOUT).decode('UTF-8').strip()
2018-02-08 03:48:29 INFO upgrade-charm File "/usr/lib/python3.4/subprocess.py", line 607, in check_output
2018-02-08 03:48:29 INFO upgrade-charm with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
2018-02-08 03:48:29 INFO upgrade-charm File "/usr/lib/python3.4/subprocess.py", line 859, in init
2018-02-08 03:48:29 INFO upgrade-charm restore_signals, start_new_session)
2018-02-08 03:48:29 INFO upgrade-charm File "/usr/lib/python3.4/subprocess.py", line 1457, in _execute_child
2018-02-08 03:48:29 INFO upgrade-charm raise child_exception_type(errno_num, err_msg)
2018-02-08 03:48:29 INFO upgrade-charm FileNotFoundError: [Errno 2] No such file or directory: 'network-get'

IP addresses are used in AMQPContext instead of hostnames

See https://bugs.launchpad.net/charm-rabbitmq-server/+bug/1758470

For client-side TLS connectivity to rabbitmq nodes it is important to use hostnames rather than IP addresses to validate server TLS certificates.

bad handling of full filesystems

Hi,

I just had a bunch of charms on a unit fail because / ran out of space, so the .juju-persistent-config and READY-SERVICES.json files in /var/lib/agents/unit-$unit/charm were 0 sized.

charmhelpers does not cope well with this and expects all existing files to be properly formatted, so it errors out when they are 0 sized.

Can we please have charmhelpers handling this correctly ?

This is present (at least) in :

• charmhelpers.core.hookenv.config()
• charmhelpers.core.services.base._load_ready_file()
• charmhelpers.contrib.templating.contexts.juju_state_to_yaml()

Thanks

Suggestion to check if port is already open before the port is opened?

charm-helpers/charmhelpers/core/hookenv.py

Line 665 in 6d72df2

def open_port(port, protocol="TCP"):

I was testing one of my subordinate charms and When I wanted to open a port that was already opened with charmhelpers.core.hookenv.open_port I got an error

subprocess.CalledProcessError: Command '['open-port', '8080/TCP']' returned non-zero exit status 1

Now I noticed that the method opened_ports is not in the API documentation and not yet available.
AttributeError: module 'charmhelpers.core.hookenv' has no attribute 'opened_ports'

but this method is not able to see the opened ports on the machine so this would not completely fix my issue. Is there any other possibility to check which other ports have been opened on the machine using charmhelpers?

etcd shows Errored with 0 known peers

I brought a 9 node Kubernetes cluster using below commands in aws

juju update-clouds # This command ensures all the latest regions are up to date on your client

juju bootstrap aws/us-east-2

watch -c juju status --color

Launched a Kubernetes cluster

juju deploy canonical-kubernetes

Then issus is why etcd shows Errored with 0 known peers

Regression in haproxy check since LP#1713165

The original bug reported in LP#1713165 [0] suggested that the following part of the haproxy check was a problem when there were backups of the config file present, as it recursively greps the haproxy config directory.

AUTH=$(grep -r "stats auth" /etc/haproxy | awk 'NR=1{print $4}')

The suggestion fix was -
AUTH=$(egrep "stats auth" /etc/haproxy/haproxy.cfg | awk 'NR=1{print $3}')

The implemented fix was -
AUTH=$(egrep "stats auth" /etc/haproxy/haproxy.cfg | awk 'NR=1{print $4}')
Note the awk token was not correct, and per comments on the original bug [1] and GitHub PR [2]

Despite this, this change has also made it into the OpenStack charms via a charmhelpers sync, which means we also have a regression in the Charms.

In addition to this, the fix also needs to be applied to 'check_haproxy_queue_depth.sh' in the same directory.

So far, I have found this has been merged into the following OpenStack charms, but I suspect that there are likely many more charms which would have synced the latest charmhelpers -

• ceilometer
• ceilometer-agent
• ceph-radosgw
• cinder
• cinder-backup
• cinder-ceph
• glance
• heat
• neutron-api
• neutron-gateway
• neutron-openvswitch
• nova-cloud-controller
• nova-compute
• openstack-dashboard
• rabbitmq-server
• swift-proxy
• swift-storage

The impact is that all of these charms install NRPE checks for haproxy, which report all backends as being down (which is a critical level alert) falsely due to not being able to grab status API credentials correctly.

I have also filed over at [3] but please kill with fire as appropriate, I know things have moved/are moving to GitHub.

[0] https://bugs.launchpad.net/charm-helpers/+bug/1713165
[1] https://bugs.launchpad.net/charm-helpers/+bug/1713165/comments/3
[2] #6 (comment)
[3] https://bugs.launchpad.net/charm-helpers/+bug/1743287

contrib/network: get_address_in_network stack traces if an interface is deleted

Also reported in lp:1746908.
https://bugs.launchpad.net/charm-helpers/+bug/1746908

On an s390x compute node, neutron-openvswitch fails at the update-status hook. (this cloud is running ocata, with 17.11 charms)

The error is:

2018-02-02 08:19:32 DEBUG update-status Traceback (most recent call last):
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/update-status", line 187, in <module>
2018-02-02 08:19:32 DEBUG update-status main()
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/update-status", line 183, in main
2018-02-02 08:19:32 DEBUG update-status assess_status(CONFIGS)
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/neutron_ovs_utils.py", line 770, in assess_status
2018-02-02 08:19:32 DEBUG update-status assess_status_func(configs)()
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1852, in _assess_status_func
2018-02-02 08:19:32 DEBUG update-status state, message = _determine_os_workload_status(*args, **kwargs)
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1252, in _determine_os_workload_status
2018-02-02 08:19:32 DEBUG update-status configs, required_interfaces)
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1306, in _ows_check_generic_interfaces
2018-02-02 08:19:32 DEBUG update-status required_interfaces)
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1587, in incomplete_relation_data
2018-02-02 08:19:32 DEBUG update-status complete_ctxts = configs.complete_contexts()
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 304, in complete_contexts
2018-02-02 08:19:32 DEBUG update-status for i in six.itervalues(self.templates)]
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 304, in <listcomp>
2018-02-02 08:19:32 DEBUG update-status for i in six.itervalues(self.templates)]
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 124, in complete_contexts
2018-02-02 08:19:32 DEBUG update-status self.context()
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 109, in context
2018-02-02 08:19:32 DEBUG update-status _ctxt = context()
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/context.py", line 981, in __call__
2018-02-02 08:19:32 DEBUG update-status ctxt.update(self.ovs_ctxt())
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/neutron_ovs_context.py", line 108, in ovs_ctxt
2018-02-02 08:19:32 DEBUG update-status fallback)
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/network/ip.py", line 112, in get_address_in_network
2018-02-02 08:19:32 DEBUG update-status addresses = netifaces.ifaddresses(iface)
2018-02-02 08:19:32 DEBUG update-status ValueError: You must specify a valid interface name.
2018-02-02 08:19:32 ERROR juju.worker.uniter.operation runhook.go:107 hook "update-status" failed: exit status 1

As the stack trace says, it happens in ip.py", line 112:

        for iface in netifaces.interfaces():
            addresses = netifaces.ifaddresses(iface)

By running such a loop manually, I could see that it fails 50% of the time, either on a tap device or on a qbr* one:

tapc3c1d498-70
Traceback (most recent call last):
  File "./debug_netifaces.py", line 9, in <module>
    print(netifaces.ifaddresses(iface))
ValueError: You must specify a valid interface name.

My understanding is that this is a race condition, if an instance is present when netifaces.interfaces() but has been deleted when the loop reaches its netifaces.ifaddresses(iface). (or just if a port was deleted)
At the moment, netifaces.interfaces() returns 198 interfaces.

This results in the neutron-openvswitch unit being in error 50% of the time.

unitdata requires explicit flush

The examples for using unitdata.kv() should be updated to reflect that flush() should be explicitly called.

It's mentioned once in the docs for charmhelpers.core.unitdata.Storage, below all of the examples:

Modifications are not persisted unless flush() is called.

haproxy.conf file inconsistently rendered causing service restarts

The haproxy service is restarted automatically when the haproxy.conf file changes on disk. This is by design due to the usage of restart_on_change function. However, the template rendered using the charmhelpers.contrib.openstack.context.HAProxyContext does not consistently render across hook invocations due to traversing a primitive python dict.

The inconsistent rendering of the haproxy.conf file causes the pacemaker resource checks to fail, which in turn causes alerts to be raised in monitoring solutions such as nagios. The checks fail because the haproxy service is restarted at the same time the pacemaker resource probe checks the status. Since the status is down (due to stop/start behavior of restart), the check fails and leads to failed actions such as:

Failed Actions:
* res_aodh_haproxy_monitor_5000 on juju-71f1e0-default-0 'not running' (7): call=20, status=complete, exitreason='none',
    last-rc-change='Tue Nov  6 23:00:30 2018', queued=0ms, exec=0ms
* res_aodh_haproxy_monitor_5000 on juju-71f1e0-default-1 'not running' (7): call=20, status=complete, exitreason='none',
    last-rc-change='Tue Nov  6 23:00:37 2018', queued=0ms, exec=0ms

The logs show the line "DEBUG juju-log Writing file /etc/haproxy/haproxy.cfg root:designate 640" whenever the file is written to disk. The file is only written to disk if the contents of the file on disk differs from the contents pending to be written to disk. All input and variables are in a static state, so this must be something along the lines of order changing for values in the file.

Upon studying the code, I'm fairly positive this has to do with inconsistent dict traversal across hook invocations. If we look at charmhelpers.contrib.openstack.context.HAProxyContext [0] the cluster_hosts is a standard python dict:

        cluster_hosts = {}

which has host information added to it based on the relations and networks it retrieves. The data added to the cluster_hosts dict is populated consistently due to self.address_types being a list (consistent traversal ordering) and consistent traversal of related_units (its sorted AND there's only a single cluster relation).

After population, the primitive python dict is supplied to the jinja template for building out the haproxy.conf file. This is passed to the jinja template as the frontends variable:

        ctxt = {
            'frontends': cluster_hosts,
            'default_backend': addr
        }

The frontends var is iterated on in the jinja template when filling out frontend and backend details:

    {% for frontend in frontends -%}
    acl net_{{ frontend }} dst {{ frontends[frontend]['network'] }}
    use_backend {{ service }}_{{ frontend }} if net_{{ frontend }}
    {% endfor -%}

and

    {% for unit, address in frontends[frontend]['backends'].items() -%}
    server {{ unit }} {{ address }}:{{ ports[1] }} check
    {% endfor %}

Since the jinja templates are using the dict as read only, its likely that the ordering is consistent within the rendering of the template (as the iteration occurs within the same hook invocation and python interpreter runtime).

The change to fix this should be simple enough, simply instantiate the cluster_hosts as a collections.OrderedDict rather than a primitive python dict.

Docs on pythonhosted.org are out of date, stuck on v0.12.0

The docs on https://pythonhosted.org/charmhelpers/ seem to be pretty out of date with the title displaying v0.12.0.

config() function should log message from JSON parser when parsing fails

Debugging charm issues with reading config from juju config-get is currently hard as there is no log message for the failure event.

Example Traceback from stable/18.02 hacluster charm:

2018-04-24 12:03:01 INFO config-changed Traceback (most recent call last):
2018-04-24 12:03:01 INFO config-changed   File "/var/lib/juju/agents/unit-hacluster-rados-1/charm/hooks/config-changed", line 481, in <module>
2018-04-24 12:03:01 INFO config-changed     hooks.execute(sys.argv)
2018-04-24 12:03:01 INFO config-changed   File "/var/lib/juju/agents/unit-hacluster-rados-1/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-04-24 12:03:01 INFO config-changed     self._hooks[hook_name]()
2018-04-24 12:03:01 INFO config-changed   File "/var/lib/juju/agents/unit-hacluster-rados-1/charm/hooks/config-changed", line 157, in config_changed
2018-04-24 12:03:01 INFO config-changed     update_nrpe_config()
2018-04-24 12:03:01 INFO config-changed   File "/var/lib/juju/agents/unit-hacluster-rados-1/charm/hooks/config-changed", line 450, in update_nrpe_config
2018-04-24 12:03:01 INFO config-changed     nrpe_setup = nrpe.NRPE(hostname=hostname)
2018-04-24 12:03:01 INFO config-changed   File "/var/lib/juju/agents/unit-hacluster-rados-1/charm/hooks/charmhelpers/contrib/charmsupport/nrpe.py", line 244, in __init__
2018-04-24 12:03:01 INFO config-changed     self.nagios_context = self.config['nagios_context']
2018-04-24 12:03:01 INFO config-changed TypeError: 'NoneType' object has no attribute '__getitem__'

It is here pretty clear that the following events occur:

• config-get produces invalid output
• This causes a subsequent json.loads() call to raise a ValueError
• This causes the config() function to return None
• Which results in the Traceback since the code in charmhelpers/contrib/charmsupport/nrpe.py attempts to use None as a Dictionary.

However when the user ran the config-get manually the output looks just fine. Having a log call sharing the details of why the ValueError was raised from json.loads() would help immensely in these types of issues.

pypi description still links to the LP site.

https://pypi.python.org/pypi/charmhelpers/0.18.1

`bzr` is hardcoded in `scripts/update-revno`

#!/bin/bash
VERSION=$(cat VERSION)
REVNO=$(bzr revno)
bzr di &>/dev/null
if [ $? ]; then
    REVNO="${REVNO}+"
fi
cat << EOF > charmhelpers/version.py
CHARMHELPERS_VERSION = '${VERSION}'
CHARMHELPERS_BZRREVNO = '${REVNO}'
EOF

@cached decorator allows cached objects to be modified

Hi,

As mentioned in #126, the @cached mechanism in hookenv.py allows the cached results to be modified. This shouldn't be possible.

Fixing this could potentially lead to charm breaking (it would certainly break the current cs:nrpe), so we should be careful.

Thanks