schaermu/changedetection.io-exporter

A prometheus-compatible exporter for your changedetection.io instance.

Changedetection.io Prometheus Exporter

This application exports the latest prices of watches configured inside a changedetection.io instance as prometheus-compatible metrics. Additionally, the exporter exposes metrics regarding scraping statistics and certain system information in order to enable system monitoring as well.

• Exposes price metrics for eligible watches (must be of type @Offer).
• Exposes scrape metrics to monitor performance of your watches.
• Monitor instance metrics like queue size, uptime or overdue watches.
• Visualize watches with the same name (but different sources) to compare price developments.

Right now, there are no plans for further development (maybe an example Grafana dashboard at some point in time). If you feel something is missing, feel free to open up an issue or (even better) a pull request!

Installation

The recommended way to run the exporter is as a part of your docker-compose stack or as a pod on your k8s cluster. Of course, you can also run it directly as a binary on your host, you have to manually compile this one yourself though.

Docker (with docker-compose)

services:
  changedection-exporter:
    image: ghcr.io/schaermu/changedetection.io-exporter:latest
    container_name: changedetection-exporter
    restart: unless-stopped
    environment:
      - CDIO_API_BASE_URL=http://changedetection:5000
      - CDIO_API_KEY=...
    depends_on:
      changedetection:
        condition: service_started

This example assumes that you have started an instance of changedetection.io within the same docker network reachable via hostname changedetection and didn't change the default port of 5000.

Kubernetes (tbd)

The exporter is configured using the following environment variables:

Environment Variable	Default value	Mandatory?
CDIO_API_BASE_URL	-	yes
CDIO_API_KEY	-	yes
PORT	9123	no
LOG_LEVEL	info	no

For all scenarios, setting both the CDIO_API_BASE_URL and a CDIO_API_KEY environment variable is mandatory, and the exporter will panic on startup if any of those is missing.

Usage

Metrics can be access by requesting the path /metrics using the exporter's hostname and its configured port (or the default one of 9123).

If you want to read those metrics into Prometheus or VictoriaMetrics, you have to configure a scraper like this:

scrape_configs:
  - job_name: "changedetection"
    static_configs:
      - targets: ["changedetection-exporter:9123"]

If you haven't got any watches registered on your changedetection.io instance, you will simply get the system metrics read in:

Metric name	Labels	Type
changedetectionio_system_uptime	version	Gauge
changedetectionio_system_watch_count	version	Gauge
changedetectionio_system_overdue_watch_count	version	Gauge
changedetectionio_system_queue_size	version	Gauge

The label version contains the running version of changedetection.io (i.e. 0.45.6).

If you have any watches registered, you will additionally get the following metrics for each of those:

Metric name	Labels	Type
changedetectionio_watch_check_count	title,source	Counter
changedetectionio_watch_fetch_time	title,source	Gauge
changedetectionio_watch_notification_alert_count	title,source	Counter
changedetectionio_watch_last_check_status	title,source	Gauge
changedetectionio_watch_price	title,source	Gauge

IMPORTANT: the metric changedetectionio_watch_price will ONLY be exposed for watches that return price information in the shape of a JSON object (with the attribute @type set to Offer).

The label title should be pretty self-explanatory, it simply contains the title from changedetection.io. In order to make sure all those metrics are unique, an additional label source is being exported. It contains the host-part of the monitored URL (i.e. www.foobar.org, so including the subdomain).

Contributing

There are two ways you can build and run the exporter locally: using the binary build or a docker image. For both options, there are Makefile targets:

# clean artifacts, run tests and build binary
$ make

# clean artifacts, compile and run binary
$ make run

# create a docker image (see note below)
$ make docker

IMPORTANT: before running the docker build, you have to bootstrap the multi-platform build system in docker by running docker buildx create --name multi-builder --bootstrap --use.

To run the tests, you can leverage Makefile targets as well:

# run tests
$ make tests

# run tests with code coverage output (coverage.html)
$ make cover

# run tests in watch mode to re-run on all code-changes
$ make watch

changedetection.io-exporter's Issues

Add unit tests

Implement unit tests (see https://github.com/prometheus/haproxy_exporter/blob/main/haproxy_exporter_test.go)

cicd: upgrade actions setup-go and checkout

Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/setup-go@v2, actions/checkout@v2. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.

Log entries created from user inputs

Tracking issue for:

• https://github.com/schaermu/changedetection.io-exporter/security/code-scanning/3 (internal/testutil/api_helpers.go:92)
• https://github.com/schaermu/changedetection.io-exporter/security/code-scanning/2 (internal/testutil/api_helpers.go:76)

Add documentation

Add more information to README.md:

• Description
• Features
• Installation/Setup
• Contributing
• ...

error while fetching watches: json: cannot unmarshal string into Go struct field WatchItem.last_error of type bool

changedetection-exporter | time="2024-04-19 03:49:54.484583" level=error msg="error while fetching watches: json: cannot unmarshal string into Go struct field WatchItem.last_error of type bool"****

Fix CollectAndCount bug when testing watch metrics

changedetection.io-exporter/pkg/collectors/watch_collector_test.go

Line 29 in d5f3303

//testutil.ExpectMetricCount(t, c, 2, expectedWatchMetrics...)

Extend CI/CD pipeline

• Add CodeQL
• Implement testing workflow (depends on #5)
• pre-commit hooks for linting (conventional-commits & golint)

Add more labels to price & scrape metrics

Beside title we should add more labels to guarantee uniqueness across metrics:

• uuid - unique identifier for each watch
• source - the host part of the watch's URL
• …

This issue depends on #2 to be implemented.

• Refactor label generation to be centralized and re-usable
• Add UUID as label
• Implement source extraction
• Add source as label
• Add test to verify labels on metrics
• Adopt metric fixtures

Adding Metric for last change data

Hey there,

mit would be cool to have data like what changed on each check. So basically what a Notification would include with changed parts.

I’m not familiar with the Changedetection API but maybe this Entrypoint would be fitting to receive the Data: https://changedetection.io/docs/api_v1/index.html#api-Watch_History-Get_single_snapshot_content

Add system metrics

Expose information from /api/v1/systeminfo as changedetectionio_system_xxx:

{
    'queue_size': 10 ,
    'overdue_watches': ["watch-uuid-list"],
    'uptime': 38344.55,
    'watch_count': 800,
    'version': "0.40.1"
}

Missing error check

Tracking issue for:

• https://github.com/schaermu/changedetection.io-exporter/security/code-scanning/1 (pkg/cdio/api.go:68)

Fix tests for automatic registration/deregistration of metrics

The exporter should automatically pick up newly created or removed watches within the monitored changedetection.io instance without having to restart.

This behavior does not seem to work properly (or at least the tests fail), this is why TestAutoUnregisterCollector and TestAutoregisterPriceCollector are commented out.

Add scraper metrics for all watches

Collect infomation from /api/v1/watch/{id}, relevant properties are:

• check_count
• fetch_time
• notification_alert_count
• last_check_status

Expose as changedetectionio_watch_xxx with label title. More labels are implemented with #15.

• Implement new watchCollector struct
• Implement watch details endpoint on ApiClient at /api/v1/watch/{id}.
• Expose relevant properties as metrics
• Extract test fixtures & implement tests for all metrics