Comments (12)
We discovered an issue that didn't activate the compress_chunk policies, thus leaving your data uncompressed even after it is eligible. Working on a fix
from tobs.
@jpigree yes, this is related to several small changes we did throughout the codebase, but also a bugfix in the latest TimescaleDB release regarding the background worker framework. I'll mark the issue as closed
from tobs.
I also have trouble finding the size of each metrics data in the database which would really help to filter out the unused metrics taking the most disk.
I am new to timescaledb but I tried to look around the DB with pgadmin4 without success. I also tried this but I got an empty result:
postgres=# \d cadvisor_version_info
View "prom_metric.cadvisor_version_info"
Column | Type | Collation | Nullable | Default
---------------------+--------------------------+-----------+----------+---------
time | timestamp with time zone | | |
value | double precision | | |
series_id | integer | | |
labels | label_array | | |
cadvisorRevision_id | integer | | |
cadvisorVersion_id | integer | | |
cluster_id | integer | | |
dockerVersion_id | integer | | |
instance_id | integer | | |
job_id | integer | | |
kernelVersion_id | integer | | |
osVersion_id | integer | | |
postgres=# SELECT table_bytes, index_bytes, toast_bytes, total_bytes
FROM hypertable_relation_size('cadvisor_version_info');
table_bytes | index_bytes | toast_bytes | total_bytes
-------------+-------------+-------------+-------------
| | |
(1 row)
from tobs.
Hello @jpigree thank you for trying out timescale-prometheus.
I'll reply with the compression related question first:
Compression is enabled by default for each metric when timescale-prometheus creates the schema. Why you might not be seeing your data get compressed is because the default chunk-size for each metric is 8h. Compression only works on chunks whose end-time is in the past, so when a new chunk is created for each metric, the old chunks will be eligible for compression.
Several things you can do here (I'll be referencing functions from our public_api: https://github.com/timescale/timescale-prometheus/blob/master/docs/sql_api.md):
- Modify default chunk-interval for each metric (
set_default_chunk_interval
). Reducing the chunk-interval will make timescale create smaller chunks, and older the chunks will become compressed sooner. But setting a too-small-chunk interval will mean a lot of chunks and that has adverse effects on query performance. - Modify the chunk-interval for a specific metric (
set_metric_chunk_interval
). Not all metrics receive data at the same rates, you can reduce the chunk_interval for metrics with high ingest rates, and increase it for sparser metrics. Depending on your scrape intervals you should have some hints which metrics should be adjusted.
For example: A clean database with a default metric chunk-interval of 4 hours where you're seeing a 1GB/h ingest rates. Data starts coming in at t=0, first chunks are created for each metric. At t=4h the db would grow up to ~4GB. at t=4h+1s new chunks get created for each metric, old chunks are available for compression.
from tobs.
Regarding the size of each metric table.cadvisor_version_info
is a view that we create for easier querying, but the metrics are stored in prom_data.cadvisor_version_info, a normalized table. You can see the size of all metrics by:
SELECT *
FROM timescaledb_information.hypertable
WHERE table_name IN (SELECT table_name FROM _prom_catalog.metric);
timescaledb_information.hypertable
is documented here https://docs.timescale.com/latest/api#timescaledb_information-hypertable_prom_catalog.metric
is a regular table that contains information about each metric
Also forgot to mention that you can set the retention period for all or specific metrics check out the API doc I linked in the previous comment
from tobs.
Thank you so much for the in-depth answers @blagojts. I understand better now how compression works and I am able to check the metrics size.
However, there isn't a mecanism to easily set the metric chunk intervals from the Helm chart. Is there any plan to make this configurable from the values.yaml
?
Otherwise, I will have to do it with a separate script. Thank you again!
from tobs.
There is no mechanism in place for setting this from the helm chart, or a flag of Timescale-Prometheus, since it's handled in the migration scripts. The idea is that you would use the SQL API for this. But we'll have it under consideration
from tobs.
Okay. If I find something myself I will share. Thanks again for your help!
from tobs.
Hi @blagojts.
I tested a bit the compression and I definitely felt the performance hit on queries when I decreased the chunks interval. However, when I am looking at the storage usage, I don't see the reduction I expected to see at each occurrence of "chunk interval".
Here is a screenshot from the persistent volume usage of my timescaledb instance:
During this time range, I reduced the chunks interval to 2 hours. I tried to zoom in to see the compression effect without success. The increase rate looks constant.
What I expected to see, was a graph like this:
This is the persistent volume usage of one of my prometheus instance using local storage.
When looking at the graph, the compression effect is much more evident.
Is this deemed normal behavior? My scrape interval is 1 minute so I know I can miss fluctuations. Another possibility is that, the storage gained is not reclaimed.
Is there a way to improve the compression further?
Thanks for your help!
from tobs.
Can you paste your PromQL for the disk usage graph here? I want to run some tests
from tobs.
hi @blagojts. I use the prometheus-operator so the PromQL is from the kubernetes-mixins repo.
I managed to extract it:
(
sum without(instance, node) (kubelet_volume_stats_capacity_bytes{job="kubelet", metrics_path="/metrics", namespace="$namespace", persistentvolumeclaim="$volume"})
-
sum without(instance, node) (kubelet_volume_stats_available_bytes{job="kubelet", metrics_path="/metrics", namespace="$namespace", persistentvolumeclaim="$volume"})
)
from tobs.
Hi @blagojts.
I recreated my whole setup a while ago with the new version of the chart "timescale-observability-0.1.0-alpha.4.1".
And compression started to work magically. The only parameter I changed recently was the dropChunk interval I reduced to every hour.
Is this a fix from the team?
from tobs.
Related Issues (20)
- Prometheus PVC name too long HOT 6
- Add test to check for *.build_info
- testing sometimes will fail or timeout HOT 6
- Incorrect port-forwarding instructions
- Include Promscale recommended Prometheus config by default
- Error on installing Tobs on GKE 1.24 HOT 9
- Flux V2 Support HOT 4
- Outdated comment
- Flag to disable grafana secret HOT 2
- Helm doesn't uninstall open telemetry collector HOT 3
- Extend Postgres Overview dashboard
- Cleanup output from notes once tobs is installed
- Production tobs: PG backups HOT 3
- Install fails due to "enableHttp2" in com.coreos.monitoring.v1.ServiceMonitor.spec.endpoints HOT 1
- Tolerations/Affinity default keys
- service accounts rights are missing for tobs-timescaledb role HOT 10
- postgres_exporter takes >5mins to generate metrics on large database HOT 6
- prometheus-stack-prometheus-0/1 pods are stuck in status "init:0/1" HOT 2
- Re-enable gathering database statistics
- shellcheck reports SC2086 in test-metrics.sh HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tobs.