charmed-lma/charm-k8s-grafana

Kubernetes Operator for Grafana

charm-k8s-grafana's Issues

Consider replacing `kubectl port-forward` with `juju expose`

From this discourse comment

The kubectl ... port-forward is also interesting, as I would have thought that juju expose grafana would be the way to interact with it. Certainly a goal would be to not have to know the exact ports that the charm uses, but have things like juju expose to do the work for you. Also hopefully expose would mean that you don’t have to rerun the command when a new unit comes up.

The metadata.yaml file should not be executable

But it is:

$ ll metadata.yaml
-rwxrwxr-x 1 facundo facundo 431 jun 24 10:44 metadata.yaml

Reconsider the "FrameworkAdapter" infrastructure

It's a layer above many things. A quite lightweight one, though, so there's not a lot of work hidden in that layer, but as it's a layer above several other things, to where each property belongs is not easily discoverable.

Most of the FrameworkAdapter attributes are from the Model, so they can always be accessed doing self.framework.model in the charm. But note that the charm already has shortcuts like app or unit (check the docs here).

So, for example you're currently doing...

juju_app = fw_adapter.get_app_name()

...and it's shorter and more accurate to do...

juju_app = self.framework.app.name

It's even absurd to have the separation layer when doing exactly the same thing, forcing you to write...

            self.fw_adapter.observe(event, delegator)

...when it's just doing exactly the same that you would be doing by...

            self.framework.observe(event, delegator)

This in general lowers a lot the learnability of this code. People getting here needs to learn this new abstraction layer instead of just using the Framework, as they are used to do.

An example of that is the developer needing to learn/understand what you're doing in...

    if not fw_adapter.am_i_leader():

...when it's simple (as long you know the Framework, which is a knowledge it's expected to have) to understand what the code does in...

    if not self.unit.is_leader():

It's fine IMO to have some layers/adapters as you have (for building the build_juju_pod_spec, for example).

But in the case of the Framework, which is by design tightly coupled with the Charm (and the Model), I would totally recommend to not have this adapter.

Please add docstring to the classes and methods in charm.py and other Python files

Should use charmcraft to build the charm

This way you could stop fixing the Python path to include lib in src/charm.py, have dependencies in a requirements.txt file, etc...

Allow supporting multiple http-based datasources

Regarding this line:

https://github.com/relaxdiego/charm-k8s-grafana/blob/b7938f5e5d3b0f751aac1aa8cafc7f0821d7f74a/src/http_interface.py#L111

As per this feedback, Grafana will not be able support multiple Prometheus (or http-based) datasources. Modify it such that Grafana gets to decides what to do with multiple endpoints.

Remove `charm-k8s-` in application name

From this discourse comment

It would be more normal to just call the charm by the application it is deploying rather than ‘charm-k8s-*’. So you would do:
juju deploy grafana
If you look at microk8s.kubectl -n lma get pods we already have grafana-operator vs grafana. The extra ‘charm-k8s’ is just noise in the deploy.

Handle the case where the unit is not (yet) the leader

2020-02-20 08:41:26 DEBUG start Traceback (most recent call last):
2020-02-20 08:41:26 DEBUG start   File "/var/lib/juju/agents/unit-charm-k8s-grafana-1/charm/hooks/start", line 67, in <module>
2020-02-20 08:41:26 DEBUG start     main(Charm)
2020-02-20 08:41:26 DEBUG start   File "lib/ops/main.py", line 187, in main
2020-02-20 08:41:26 DEBUG start     _emit_charm_event(charm, juju_event_name)
2020-02-20 08:41:26 DEBUG start   File "lib/ops/main.py", line 118, in _emit_charm_event
2020-02-20 08:41:26 DEBUG start     event_to_emit.emit(*args, **kwargs)
2020-02-20 08:41:26 DEBUG start   File "lib/ops/framework.py", line 192, in emit
2020-02-20 08:41:26 DEBUG start     framework._emit(event)
2020-02-20 08:41:26 DEBUG start   File "lib/ops/framework.py", line 602, in _emit
2020-02-20 08:41:26 DEBUG start     self._reemit(event_path)
2020-02-20 08:41:26 DEBUG start   File "lib/ops/framework.py", line 637, in _reemit
2020-02-20 08:41:26 DEBUG start     custom_handler(event)
2020-02-20 08:41:26 DEBUG start   File "/var/lib/juju/agents/unit-charm-k8s-grafana-1/charm/hooks/start", line 48, in on_start_delegator
2020-02-20 08:41:26 DEBUG start     self.adapter.set_pod_spec(output.spec)
2020-02-20 08:41:26 DEBUG start   File "/var/lib/juju/agents/application-charm-k8s-grafana/charm/src/adapters.py", line 28, in set_pod_spec
2020-02-20 08:41:26 DEBUG start     self._framework.model.pod.set_spec(spec_obj)
2020-02-20 08:41:26 DEBUG start   File "lib/ops/model.py", line 452, in set_spec
2020-02-20 08:41:26 DEBUG start     raise ModelError('cannot set a pod spec as this unit is not a leader')
2020-02-20 08:41:26 DEBUG start ops.model.ModelError: cannot set a pod spec as this unit is not a leader
2020-02-20 08:41:27 ERROR juju.worker.uniter.operation runhook.go:132 hook "start" failed: exit status 1
2020-02-20 08:41:37 DEBUG start Traceback (most recent call last):
2020-02-20 08:41:37 DEBUG start   File "/var/lib/juju/agents/unit-charm-k8s-grafana-1/charm/hooks/start", line 67, in <module>
2020-02-20 08:41:37 DEBUG start     main(Charm)
2020-02-20 08:41:37 DEBUG start   File "lib/ops/main.py", line 187, in main
2020-02-20 08:41:37 DEBUG start     _emit_charm_event(charm, juju_event_name)
2020-02-20 08:41:37 DEBUG start   File "lib/ops/main.py", line 118, in _emit_charm_event
2020-02-20 08:41:37 DEBUG start     event_to_emit.emit(*args, **kwargs)
2020-02-20 08:41:37 DEBUG start   File "lib/ops/framework.py", line 192, in emit
2020-02-20 08:41:37 DEBUG start     framework._emit(event)
2020-02-20 08:41:37 DEBUG start   File "lib/ops/framework.py", line 602, in _emit
2020-02-20 08:41:37 DEBUG start     self._reemit(event_path)
2020-02-20 08:41:37 DEBUG start   File "lib/ops/framework.py", line 637, in _reemit
2020-02-20 08:41:37 DEBUG start     custom_handler(event)
2020-02-20 08:41:37 DEBUG start   File "/var/lib/juju/agents/unit-charm-k8s-grafana-1/charm/hooks/start", line 48, in on_start_delegator
2020-02-20 08:41:37 DEBUG start     self.adapter.set_pod_spec(output.spec)
2020-02-20 08:41:37 DEBUG start   File "/var/lib/juju/agents/application-charm-k8s-grafana/charm/src/adapters.py", line 28, in set_pod_spec
2020-02-20 08:41:37 DEBUG start     self._framework.model.pod.set_spec(spec_obj)
2020-02-20 08:41:37 DEBUG start   File "lib/ops/model.py", line 452, in set_spec
2020-02-20 08:41:37 DEBUG start     raise ModelError('cannot set a pod spec as this unit is not a leader')
2020-02-20 08:41:37 DEBUG start ops.model.ModelError: cannot set a pod spec as this unit is not a leader
2020-02-20 08:41:37 ERROR juju.worker.uniter.operation runhook.go:132 hook "start" failed: exit status 1

Grafana GPG key rotation

Grafana recently rotated their GPG key. As a result, the charm fails to install Grafana:

unit-grafana-1: 14:30:06 INFO juju.worker.uniter found queued "install" hook
unit-grafana-1: 14:30:29 WARNING unit.grafana/1.install ERROR: launchpadlib 1.10.13 requires testresources, which is not installed.
unit-grafana-1: 14:30:31 INFO unit.grafana/1.juju-log Reactive main running for hook install
unit-grafana-1: 14:30:31 ERROR unit.grafana/1.juju-log Unable to find implementation for relation: requires of register-application
unit-grafana-1: 14:30:31 INFO unit.grafana/1.juju-log Initializing Snap Layer
unit-grafana-1: 14:30:31 INFO unit.grafana/1.juju-log Invoking reactive handler: reactive/grafana.py:205:install_packages
unit-grafana-1: 14:30:37 WARNING unit.grafana/1.install W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
unit-grafana-1: 14:30:37 WARNING unit.grafana/1.install E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
unit-grafana-1: 14:30:37 INFO unit.grafana/1.juju-log Failed executing 'add-apt-repository --yes deb https://packages.grafana.com/oss/deb stable main'. Will retry in 10 seconds
unit-grafana-1: 14:30:50 WARNING unit.grafana/1.install W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
unit-grafana-1: 14:30:50 WARNING unit.grafana/1.install E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
unit-grafana-1: 14:30:50 INFO unit.grafana/1.juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
unit-grafana-1: 14:31:02 WARNING unit.grafana/1.install W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
unit-grafana-1: 14:31:02 WARNING unit.grafana/1.install E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
unit-grafana-1: 14:31:02 INFO unit.grafana/1.juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
unit-grafana-1: 14:31:14 WARNING unit.grafana/1.install W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
unit-grafana-1: 14:31:14 WARNING unit.grafana/1.install E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
unit-grafana-1: 14:31:14 INFO unit.grafana/1.juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
unit-grafana-1: 14:31:25 WARNING unit.grafana/1.install W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
unit-grafana-1: 14:31:25 WARNING unit.grafana/1.install E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
unit-grafana-1: 14:31:25 INFO unit.grafana/1.juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
unit-grafana-1: 14:31:37 WARNING unit.grafana/1.install W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
unit-grafana-1: 14:31:37 WARNING unit.grafana/1.install E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
unit-grafana-1: 14:31:37 INFO unit.grafana/1.juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
unit-grafana-1: 14:31:49 WARNING unit.grafana/1.install W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
unit-grafana-1: 14:31:49 WARNING unit.grafana/1.install E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
unit-grafana-1: 14:31:49 INFO unit.grafana/1.juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
unit-grafana-1: 14:32:01 WARNING unit.grafana/1.install W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
unit-grafana-1: 14:32:01 WARNING unit.grafana/1.install E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
unit-grafana-1: 14:32:01 INFO unit.grafana/1.juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
unit-grafana-1: 14:32:12 WARNING unit.grafana/1.install W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
unit-grafana-1: 14:32:12 WARNING unit.grafana/1.install E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
unit-grafana-1: 14:32:12 INFO unit.grafana/1.juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
unit-grafana-1: 14:32:24 WARNING unit.grafana/1.install W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
unit-grafana-1: 14:32:24 WARNING unit.grafana/1.install E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
unit-grafana-1: 14:32:24 INFO unit.grafana/1.juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
unit-grafana-1: 14:32:36 WARNING unit.grafana/1.install W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
unit-grafana-1: 14:32:36 WARNING unit.grafana/1.install E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
unit-grafana-1: 14:32:36 INFO unit.grafana/1.juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install W: GPG error: https://packages.grafana.com/oss/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9E439B102CF3C0C6
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install E: The repository 'https://packages.grafana.com/oss/deb stable InRelease' is not signed.
unit-grafana-1: 14:32:48 ERROR unit.grafana/1.juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-grafana-1/charm/reactive/grafana.py", line 258, in install_packages
    fetch.configure_sources(update=True)
  File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charmhelpers/fetch/__init__.py", line 149, in configure_sources
    _fetch_update(fatal=True)
  File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charmhelpers/fetch/ubuntu.py", line 400, in apt_update
    _run_apt_command(cmd, fatal)
  File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charmhelpers/fetch/ubuntu.py", line 945, in _run_apt_command
    _run_with_retries(
  File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charmhelpers/fetch/ubuntu.py", line 922, in _run_with_retries
    result = subprocess.check_call(cmd, env=env, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['apt-get', 'update']' returned non-zero exit status 100.

unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install Traceback (most recent call last):
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install   File "/var/lib/juju/agents/unit-grafana-1/charm/hooks/install", line 22, in <module>
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install     main()
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install   File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install     bus.dispatch(restricted=restricted_mode)
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install   File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install     _invoke(other_handlers)
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install   File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install     handler.invoke()
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install   File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install     self._action(*args)
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install   File "/var/lib/juju/agents/unit-grafana-1/charm/reactive/grafana.py", line 258, in install_packages
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install     fetch.configure_sources(update=True)
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install   File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charmhelpers/fetch/__init__.py", line 149, in configure_sources
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install     _fetch_update(fatal=True)
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install   File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charmhelpers/fetch/ubuntu.py", line 400, in apt_update
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install     _run_apt_command(cmd, fatal)
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install   File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charmhelpers/fetch/ubuntu.py", line 945, in _run_apt_command
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install     _run_with_retries(
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install   File "/var/lib/juju/agents/unit-grafana-1/.venv/lib/python3.8/site-packages/charmhelpers/fetch/ubuntu.py", line 922, in _run_with_retries
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install     result = subprocess.check_call(cmd, env=env, **kwargs)
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install     raise CalledProcessError(retcode, cmd)
unit-grafana-1: 14:32:48 WARNING unit.grafana/1.install subprocess.CalledProcessError: Command '['apt-get', 'update']' returned non-zero exit status 100.
unit-grafana-1: 14:32:48 ERROR juju.worker.uniter.operation hook "install" (via explicit, bespoke hook script) failed: exit status 1
unit-grafana-1: 14:32:48 INFO juju.worker.uniter awaiting error resolution for "install" hook

Simplify ImageMeta

It looks its only purpose is to provide a "holder" to move some data around, so you can do...

                'imagePath': image_meta.image_path,

...instead of...

                'imagePath': image_meta['registrypath'],

As it doesn't have other functionality, you could just use a namedtuple:

>>> from collections import namedtuple
>>> ImageMeta = namedtuple('ImageMeta', 'image_path repo_username repo_password')
>>> rd = dict(registrypath='foo', username='bar', password='baz')
>>> im = ImageMeta(image_path=rd['registrypath'], repo_username=rd['username'], repo_password=rd['password'])
>>> im.image_path
'foo'