claws/buildbot-prometheus

A Prometheus metrics exporter for Buildbot.

buildbot-prometheus's Issues

Prometheus stepsConsumer raising type error for step duration

I recently connected the Prometheus reporter to a master instance but am receiving an error related to the step duration returning an instance of datetime.timedelta when trying to set a metric value. Error included below:

buildbot_1 | 2021-02-15 19:34:11+0000 [-] while invoking <bound method Prometheus.stepsConsumer of <buildbot_prometheus.prometheus.Prometheus object at 0x7f86abf43b20>>
buildbot_1 | Traceback (most recent call last):
buildbot_1 | File "/usr/lib/python3.8/site-packages/twisted/internet/defer.py", line 460, in callback
buildbot_1 | self._startRunCallbacks(result)
buildbot_1 | File "/usr/lib/python3.8/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
buildbot_1 | self._runCallbacks()
buildbot_1 | File "/usr/lib/python3.8/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
buildbot_1 | current.result = callback(current.result, *args, **kw)
buildbot_1 | File "/usr/lib/python3.8/site-packages/twisted/internet/defer.py", line 1475, in gotResult
buildbot_1 | _inlineCallbacks(r, g, status)
buildbot_1 | --- ---
buildbot_1 | File "/usr/lib/python3.8/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
buildbot_1 | result = g.send(result)
buildbot_1 | File "/usr/lib/python3.8/site-packages/buildbot_prometheus/prometheus.py", line 496, in stepsConsumer
buildbot_1 | self.g_steps_duration.labels(**labels).set(duration_seconds)
buildbot_1 | File "/usr/lib/python3.8/site-packages/prometheus_client/metrics.py", line 359, in set
buildbot_1 | self._value.set(float(value))
buildbot_1 | builtins.TypeError: float() argument must be a string or a number, not 'datetime.timedelta'

I am running version 2.10 of the Buildbot master and 20.4.0 of the Prometheus reporter. I have resolved this in a subclass on my current master instance 👍

Installation with current pip fails

Using pip 22.0.3 to install buildbot-prometheus master locally fails with

pip install .
Processing c:\users\bla\buildbot-prometheus
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\bla\buildbot-prometheus\setup.py", line 5, in <module>
          from pip.req import parse_requirements
      ModuleNotFoundError: No module named 'pip.req'

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

It seems the present setup.py is outdated, using since-removed pip API.

Modifying it to this at least installs:

from pip._internal.req import parse_requirements
from pip._internal.network.session import PipSession

from setuptools import setup

install_reqs = parse_requirements("requirements.txt", session=PipSession())
requires = [ir.requirement for ir in install_reqs]

I don't know if it's "modern usage", though - using pip._internal feels strange. The setup.py claims that this package supports Python 2, I guess it should be reviewed anyway... ;-)

Tag releases

Hi @claws

can you provide tagged releases for buildbot-prometheus?

Thanks!

Newer Buildbot Version?

Is that possible to use in newer versions of buildbot? Or do you have a list what would be needed to be updated?

Python version check breaks on Debian testing

2017-12-28 16:08:02+0000 [-] Traceback (most recent call last):
2017-12-28 16:08:02+0000 [-]   File "/home/buildbot/venv/local/lib/python2.7/site-packages/buildbot/plugins/db.py", line 258, in _load_entry
2017-12-28 16:08:02+0000 [-]     result = entry.load()
2017-12-28 16:08:02+0000 [-]   File "/home/buildbot/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2291, in load
2017-12-28 16:08:02+0000 [-]     return self.resolve()
2017-12-28 16:08:02+0000 [-]   File "/home/buildbot/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2297, in resolve
2017-12-28 16:08:02+0000 [-]     module = __import__(self.module_name, fromlist=['__name__'], level=0)
2017-12-28 16:08:02+0000 [-]   File "/home/buildbot/venv/local/lib/python2.7/site-packages/buildbot_prometheus/__init__.py", line 2, in <module>
2017-12-28 16:08:02+0000 [-]     from .prometheus import Prometheus
2017-12-28 16:08:02+0000 [-]   File "/home/buildbot/venv/local/lib/python2.7/site-packages/buildbot_prometheus/prometheus.py", line 36, in <module>
2017-12-28 16:08:02+0000 [-]     from prometheus_client import Counter, Gauge, Histogram
2017-12-28 16:08:02+0000 [-]   File "/home/buildbot/venv/local/lib/python2.7/site-packages/prometheus_client/__init__.py", line 4, in <module>
2017-12-28 16:08:02+0000 [-]     from . import exposition
2017-12-28 16:08:02+0000 [-]   File "/home/buildbot/venv/local/lib/python2.7/site-packages/prometheus_client/exposition.py", line 31, in <module>
2017-12-28 16:08:02+0000 [-]     PYTHON26_OR_OLDER = tuple(int(val) for val in sys.version.split()[0].split('.')) < (2, 7, 0)
2017-12-28 16:08:02+0000 [-]   File "/home/buildbot/venv/local/lib/python2.7/site-packages/prometheus_client/exposition.py", line 31, in <genexpr>
2017-12-28 16:08:02+0000 [-]     PYTHON26_OR_OLDER = tuple(int(val) for val in sys.version.split()[0].split('.')) < (2, 7, 0)
2017-12-28 16:08:02+0000 [-] ValueError: invalid literal for int() with base 10: '14+'

Debian currently ships with a Python version "2.7.14+", which is arguably stupid but I can't do much about that.

FR: Pending builds per builder, elapsed time for current build per builder

[I may end up implementing that myself in the future, but filing a FR since other people might be interested and might know the codebase better than I do.]

On dolphin-emu's buildbot we keep hitting "stuck builds" blocking our builders until someone takes manual action. It would be great if buildmaster was more resilient to random failures in general, but in the meantime having monitoring+alerting for stuck builds would improve MTTR quite a lot.

The two main metrics I would suggest to be added are:

• Count of pending/queued builds, per builder.
• Elapsed time for current build, per builder (or 0 if no build in progress).

WDYT?