Help on Latency SLO definition

[hdimitriou]

Help on Latency SLO definition

[tokheim]

The normal approach is to consider any request that doesn't meet latency target as a straight up error. This would mean you should use le approach, and tailor le to your desired targets. You can read underlying motivation here https://grafana.com/blog/2019/11/27/kubecon-recap-how-to-include-latency-in-slo-based-alerting/.

For your histogram_quantile approach it seems to calculate error rate as number of time slices with a 98q delay over 1s. It might help smoothing the graphs to make the time slices as small as possible, something like

errorQuery: -
  sum_over_time(
    count(
      histogram_quantile(0.98,
        sum by (le) (rate(<expr>[30s]))
      )
    > 1000)
  [{{.window}}:])

Still drawbacks I see with such an approach are

• rates each time period as equally important (not taking traffic into account)
• I wouldn't trust histogram_quantile(..) > 1000 to report accurately unless I also have a le="1000" bucket
• Seems like an expensive query to execute over large windows