Any chance there will be a support of counters?

[badoo]

Any chance there will be a support of counters?

[anton-povarov]

Reading that discussion - i would assume that the request is to have counters that live effectively forever (survive restarts, etc.)?

This is a non-goal for pinba imo.
Pinba's role in the stats pipeline is aggregation and convenient query-ing, but it's not a time-series storage engine.

There are plenty of those on the market anyway. It might be a nice idea to have easy integration with those tools, though.

---

if i got the idea wrong - feel free to elaborate.

[neanton]

We are trying to implement an exporter from Pinba to a real time series database.

Other tools, eg. prometheus, handle counter resets, so surviving restart is not a big deal.
The main reason for counters is to get an absolute value. That way Prometheus can calculate rate during the scrape interval window.

Right now we can emulate counters using timers with a tag report so that we have a rolling 2 minute window of "counter-requests-hits". But this window can not be queried for a time interval of data (something like SELECT timer_name, hits WHERE time BETWEEN last_scrape_time AND current_time). That is why there is no ability to get absolute timer hits delta between scrapes and the only way is to export all data from Pinba as GAUGE type.

I hoped we can use Pinba as an alternative to Statsd (for internal application metrics), as it does it's job extremely well for collecting PHP metrics.

[anton-povarov]

Hm, i think i get it now.
Basically you want a way to figure out the physical 'time window' for the data returned in selects ?

This is a nice idea, but i'm kinda not sure how to do it interface-wise.
SQL tables are quite limited in what can be expressed with them.

Internally the data is stored in 1-second wide 'timeslices' (that are all merged on select) - just need to find a way to expose that info with sql interface.

Any ideas ?

[anton-povarov]

I was thinking of maybe making pinba internal stuff into a library that can be used to build other access interfaces (like a go daemon with it's own query language or whatever).
But that might be out of scope for this particular issue :)

[anton-povarov]

Another interpretation, suggested by @Sannis is:
You want a report that effectively has infinite time window (pinba_start_time, +inf).

This would mean that report never deletes anything.
This would require a different internal implementation, but should be pretty straightforward, and as a bonus - would actually perform better on selects.

[neanton]

That seems a good idea and will provide enough information to integrate Pinba with other monitoring solutions. Not sure how much use it will make for Badoo team though :)

I think for now we’ll evaluate another alternative solutions for custom application metrics (seems like https://github.com/prometheus/statsd_exporter looks exactly what we need) and leave Pinba for general metrics and timers.

Thank you for your time on researching on this topic!

[anton-povarov]

Ok, i guess this is a "nice to have" feature from now on.