Cant retrieve Custom Metrics

[zalando-incubator]

Cant retrieve Custom Metrics

[mikkeloscar]

Using something like $.measurements[:].value does work though, but I don't know if it represents the thing you want?

However this does not work with the kube-metrics-adapter because we expect a single value to be the result rather than a list. I think you need to do: $.measurements[0].value to get something that makes sense.

[balamuru]

You are correct. It should be $.measurements[0].value. Custom autoscaling now works
Thanks
The valid command / yaml is

cat <<EOF | kubectl apply -f -
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: springboot-custom-hpa
  namespace: default
  labels:
    application: custom-metrics-consumer
  annotations:
    # metric-config.<metricType>.<metricName>.<collectorName>/<configKey>
    metric-config.pods.load-per-min.json-path/json-key: "$.measurements[0].value"
    metric-config.pods.load-per-min.json-path/path: /actuator/metrics/system.load.average.1m
    metric-config.pods.load-per-min.json-path/port: "7070"
    metric-config.pods.load-per-min.json-path/scheme: "http"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: springboot-webapp
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metricName: load-per-min
      targetAverageValue: 1
EOF

Sample output

$ kubectl get hpa --watch
NAME                    REFERENCE                      TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
springboot-custom-hpa   Deployment/springboot-webapp   189m/1    1         10        1          44h
springboot-custom-hpa   Deployment/springboot-webapp   435m/1    1         10        1          44h
springboot-custom-hpa   Deployment/springboot-webapp   1351m/1   1         10        1          44h
springboot-custom-hpa   Deployment/springboot-webapp   1351m/1   1         10        2          44h
springboot-custom-hpa   Deployment/springboot-webapp   2388m/1   1         10        2          44h
springboot-custom-hpa   Deployment/springboot-webapp   2388m/1   1         10        4          44h
springboot-custom-hpa   Deployment/springboot-webapp   2388m/1   1         10        8          44h
springboot-custom-hpa   Deployment/springboot-webapp   2644m/1   1         10        8          44h
springboot-custom-hpa   Deployment/springboot-webapp   2644m/1   1         10        10         44h
`

[mikkeloscar]

Thanks for a detailed report, could you also show the logs of the kube-metrics-adapter pod. I would imagine it would log some kind of error when trying to pull the metrics.

[balamuru]

Here you go . Looks like some sort out of range error

sh-4.2$ kubectl get hpa
NAME                    REFERENCE                      TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
springboot-custom-hpa   Deployment/springboot-webapp   <unknown>/1   1         10        2          40h
sh-4.2$ kubectl get po -n kube-system |grep metrics
kube-metrics-adapter-55cbb64dc9-g22pb   1/1     Running   0          3d22h
metrics-server-9cb648b76-4j6tf          1/1     Running   2          10d
sh-4.2$ kubectl logs  kube-metrics-adapter-55cbb64dc9-g22pb -n kube-system
.
.
.
time="2019-08-23T14:50:06Z" level=error msg="Failed to get metrics from pod 'default/springboot-webapp-79475bc7c-gc6pm': index [to] out of range: len: 1, to: 1" Collector=Pod
time="2019-08-23T14:50:06Z" level=error msg="Failed to get metrics from pod 'default/springboot-webapp-79475bc7c-tdvkr': index [to] out of range: len: 1, to: 1" Collector=Pod
time="2019-08-23T14:50:06Z" level=info msg="Collected 0 new metric(s)" provider=hpa
time="2019-08-23T14:50:08Z" level=info msg="Looking for HPAs" provider=hpa
time="2019-08-23T14:50:08Z" level=info msg="Found 0 new/updated HPA(s)" provider=hpa
time="2019-08-23T14:50:38Z" level=info msg="Looking for HPAs" provider=hpa
time="2019-08-23T14:50:38Z" level=info msg="Found 0 new/updated HPA(s)" provider=hpa
time="2019-08-23T14:51:06Z" level=error msg="Failed to get metrics from pod 'default/springboot-webapp-79475bc7c-gc6pm': index [to] out of range: len: 1, to: 1" Collector=Pod
time="2019-08-23T14:51:06Z" level=error msg="Failed to get metrics from pod 'default/springboot-webapp-79475bc7c-tdvkr': index [to] out of range: len: 1, to: 1" Collector=Pod
time="2019-08-23T14:51:06Z" level=info msg="Collected 0 new metric(s)" provider=hpa
time="2019-08-23T14:51:08Z" level=info msg="Looking for HPAs" provider=hpa
time="2019-08-23T14:51:08Z" level=info msg="Found 0 new/updated HPA(s)" provider=hpa

[mikkeloscar]

Then the question is if this query $.measurements[:1].value is valid in https://github.com/oliveagle/jsonpath

At least it would not be so hard to test this with all the data you provided :)

[mikkeloscar]

I tested it like this and it is not valid at least with the json defined:

package main

import (
	"encoding/json"
	"fmt"

	"github.com/oliveagle/jsonpath"
)

const v = `{
  "name" : "system.load.average.1m",
  "description" : "The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time",
  "baseUnit" : null,
  "measurements" : [ {
    "statistic" : "VALUE",
    "value" : 0.0439453125
  } ],
  "availableTags" : [ ]
}`

func main() {
	var json_data interface{}
	json.Unmarshal([]byte(v), &json_data)

	res, err := jsonpath.JsonPathLookup(json_data, "$.measurements[:1].value")
	fmt.Println(err)
	fmt.Println(res)
}

Output is:

index [to] out of range: len: 1, to: 1
<nil>

Using something like $.measurements[:].value does work though, but I don't know if it represents the thing you want?