Introduce a config property to disable exemplars for Metrics of counter type

[xak2000]

Introduce a config property to disable exemplars for Metrics of counter type

[jonatan-ivanov]

I'm a bit puzzled here since exemplars for Counter was supported by Prometheus since the introduction of that feature. On the other hand exemplars on _count for Summary and Histogram was not, please see the description of the issue you linked:

Prometheus did not support exemplars for Summaries and it did not support exemplars for anything else than buckets on Histograms. Since Prometheus now supports exemplars on every time series, we can add exemplars to Summary/Histogram _count, see: prometheus/prometheus#11982

So you should not have any problems with Counters whatsoever, but you can run into issues with _count on Summary and Histogram.

Because of @ConditionalOnMissingBean(ExemplarSampler.class) if you create any ExemplarSampler @Bean Boot will back off and not create one so you don't need the BeanPostProcessor:

@Bean
public IgnoreCountersExemplarSampler exemplarSampler(SpanContextSupplier spanContextSupplier) { 
 	return new IgnoreCountersExemplarSampler(new DefaultExemplarSampler(spanContextSupplier)); 
 }

or if IgnoreCountersExemplarSampler would extend DefaultExemplarSampler, you would only need to override one method and create the instance like this:

new IgnoreCountersExemplarSampler(spanContextSupplier);

If you want to disable all exemplars, you can create a SpanContextSupplier that always says isSampled false. Also, your solution not just disables _count but it also disables exemplars on Prometheus Counters. Could you please help us with your request a bit?

• Is it ok for you to disable exemplars for everything?
• Or you want to disable exemplars only for _count (the one that prevents you to upgrade) and use the other Exemplars?
• Disabling exemplars on _count and Counter does not make a lot of sense to me but let us know if this is what you want.

Has anyone asked GKE, why they are using a Prometheus version this old? 2.50.1 is the current Prometheus Server version.

[mhalbritter]

Thanks for your input, @jonatan-ivanov .

[xak2000]

I'm a bit puzzled here since exemplars for Counter was supported by Prometheus since the introduction of that feature. On the other hand exemplars on _count for Summary and Histogram was not

Please forgive me my ignorance, I didn't know that Counter and _count for Summary and Histogram are different things. To be honest, the terminology of metric types and other stuff in Prometheus is very confusing to me and despite the fact I use Prometheus several years, I still newbie in it.

Because of @ConditionalOnMissingBean(ExemplarSampler.class) if you create any ExemplarSampler @Bean Boot will back off and not create one so you don't need the BeanPostProcessor:

Yes, I know, please see my answer in the SO question. I just think that BeanPostProcessor is a better solution for a workaround, as it is not relying on the DefaultExemplarSampler class. If Spring Boot will change the default implementation class in the AutoConfiguration, the BeanPostProcessor will still wrap this new implementation, while custom bean will use the hardcoded DefaultExemplarSampler. But, anyway, it is just a matter of preference. Both solutions work identically.

If you want to disable all exemplars, you can create a SpanContextSupplier that always says isSampled false.

Yes, I know. You mentioned it in the SO answer. But this will disable exemplars for all metrics, right? I prefer to keep them for metrics where they were supported in Prometheus 2.41. A quote from GCP docs:

Google Cloud Managed Service for Prometheus can only ingest exemplars attached to histogram metrics. Exemplars attached to counter metrics can't be ingested. Prometheus histogram metrics are converted to the analogous Cloud Monitoring Distribution type, which supports exemplars. Non-distribution metrics in Cloud Monitoring do not support exemplars.

So, I would like to keep supported stuff in-place. Also, returning false from isSampled span that is really sampled looks like a hack to me. The method will lie just to fool the caller. What if isSampled will be used (or maybe it already is) by other callers that will use this information for some other means (not just to decide if exemplar should be added or not)...

Also, your solution not just disables _count but it also disables exemplars on Prometheus Counters.

I didn't know that. :(

Could you please help us with your request a bit?

To be honest, what I really want is to make sure that apps on Spring Boot 3.2+ will continue to be monitored on GCP. I already solved this problem for myself (see the workaround above). Probably not ideally, as you mentioned that my solution will disable exemplars on Counters that is unnecessary as it is supported by Prometheus 2.41, so by GCP. But at least my workaround allows our apps to be monitored again.

So, I wanted to create a solution that allows to solve this problem for anyone (not just me, and not only for GCP) without such workarounds. I think the problem of non-functional monitoring is important enough to justify the config property.

But, as you mentioned that this solution will not just disable exemplars for _count rows in the output, but for Counters too, then I have no idea how to disable exemplars just for _count rows...

Maybe it's not worth to do if it will require too many internal changes in Micrometer or Spring Boot. When these changes will finally be available in the next Spring Boot version, GCP (and I hope most other platforms) will already be upgraded to Prometheus 2.43+.

Has anyone asked GKE, why they are using a Prometheus version this old? 2.50.1 is the current Prometheus Server version.

This question is flying in the air, but I do not know anyone from GKE team to ask such technical question. Maybe it's just a principle: "If it ain’t broke, don’t fix it." :)

And thank you for you thoughtful input, @jonatan-ivanov!

P.S. As you mentioned that Counter and _count are different things, can you rename please the title of this issue? I don't know how to propertly call this _count thing. It's still a counter, but not Counter metric type.

[jonatan-ivanov]

Please forgive me my ignorance [...]

I wanted to make sure that we are on the same page and I'm not missing something.

Google Cloud Managed Service for Prometheus can only ingest exemplars attached to histogram metrics. Exemplars attached to counter metrics can't be ingested.

GCP ignoring exemplars on Counters surprises me a bit but I'm not sure I understand why is it a problem for GCP that other time series can also contain exemplars in this case. GCP could ignore those too. (Maybe they remove the exemplar after Prometheus parsed the data and the older version of Prometheus they use cannot parse _count?)

So, I would like to keep supported stuff in-place.

This complicates things since I feel in order to solve this in Micrometer/Boot, Micrometer also needs to be changed since it should make _count configurable. This can be a rabbit hole if we want to add extra exemplars later (we have/had requests for more, like _max).

Also, your solution not just disables _count but it also disables exemplars on Prometheus Counters.

I didn't know that. :(

If you only use it for GCP which seems to be ignoring exemplars on Counter, disabling it does not make a difference for you, so your workaround is ok for GCP.

Maybe it's not worth to do if it will require too many internal changes in Micrometer or Spring Boot. When these changes will finally be available in the next Spring Boot version, GCP (and I hope most other platforms) will already be upgraded to Prometheus 2.43+.

This is definitely something we should consider. I think a proper solution for this would be through content negotiation: releasing OpenMetrics 1.1 with the extra exemplars feature that Prometheus already supports, plus making the Prometheus Java Client return the extra exemplars if OpenMetrics 1.1 is requested and do not return them if OpenMetrics 1.0 is asked for.

[xak2000]

I think you are right and content negotiation is the right way to do it.

Looking back, I think that when the ability to parse _count metrics with exemplars was implemented in Prometheus 2.43, a new content type version should have been created, so if a scrapper from Prometheus 2.41 uses an older version in the Accept header, the output should just not include exemplars in _count. This way such change would not be a breaking change.

But such solution will probably require many ifs in the implementation of metrics data serializer..

Or, if older Prometheus would just ignore unknown parts of line (e.g. anything after # that could not be parsed is considered as a comment), then adding exemplars to _count would also not be a breaking change. But I don't know the format well, so it's a speculation from my side. Maybe there are good reasons why Prometheus doesn't do this.

[mhalbritter]

There's nothing Spring Boot can here, right? I'm closing this issue.

[jonatan-ivanov]

@xak2000

I think that when the ability to parse _count metrics with exemplars was implemented in Prometheus 2.43, a new content type version should have been created [...]

There is no such content type, Open Metrics 1.1 does not exist (this should change in the future though).

if older Prometheus would just ignore unknown parts [...]

Yeah, I don't know why Prometheus drops the whole scrape response instead of ignoring what is not supported.

@mhalbritter I think in this particular case nothing Spring Boot can do here since @xak2000 needs existing exemplars and only wants to ignore the new ones (this also needs a change in Micrometer). Though there might be users in the future who will ask an enable flag for the whole exemplars functionality.