Possible to write HPA using nested prometheus metric ?

[zalando-incubator]

Possible to write HPA using nested prometheus metric ?

[karthikic]

I have tried this:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: custom-metrics-consumer
  namespace: default
  labels:
    application: custom-metrics-consumer
  annotations:
    # metric-config.<metricType>.<metricName>.<collectorName>/<configKey>
    metric-config.object.prometheus-target-kbasync-lenh-seca.prometheus/query: |
      scalar(sum(rate(collectd_statsd_derive_total{statsd="prometheus_target_kbasync_lenh_seca"}[20m])))
    metric-config.object.prometheus-target-kbasync-lenh-seca.prometheus/per-replica: "true"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: custom-metrics-consumer
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Object
    object:
      metricName: prometheus-target-kbasync-lenh-seca
      target:
        apiVersion: v1
        kind: .
        name: custom-metrics-consumer
      targetValue: 10 # this will be treated as targetAverageValue

My prometheus query is correct :

curl -g prometheus.monitoring.svc:9090/api/v1/query?query="scalar(sum(rate(collectd_statsd_derive_total{statsd='prometheus_target_kbasync_lenh_sec'}[5m])))"
{"status":"success","data":{"resultType":"scalar","result":[1544377450.443,"0.008333333333333333"]}}

Edit: Updated correct HPA

[karthikic]

was able to get it working with above HPA but kind is getting stored as "" for my metric
eg:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/""/*/prometheus-target-kbasync-lenh-sec" --kubeconfig=config-demo | jq
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default//%2A/prometheus-target-kbasync-lenh-sec"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Deployment",
        "namespace": "default",
        "name": "custom-metrics-consumer",
        "apiVersion": "v1"
      },
      "metricName": "prometheus-target-kbasync-lenh-sec",
      "timestamp": "2018-12-09T18:46:24Z",
      "value": "2m"
    }
  ]
}

how do i tell what kind to use so that HPA will fetch the metrics properly ?

[karthikic]

logs:

I1209 19:39:13.546788 1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/deployments/custom-metrics-consumer/prometheus-target-kbasync-lenh-seca: (1.950008ms) 404

but if replace deployment with "" or . or empty it works.

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/""/*/prometheus-target-kbasync-lenh-sec" --kubeconfig=config-demo | jq
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default//%2A/prometheus-target-kbasync-lenh-sec"
  },
  "items": [
    {

So prometheus collector is not storing the kind correctly ?.
HPA doesnt accept "", . or empty for kind in object.target field

[karthikic]

so i looked at the code and found

		groupResource = schema.GroupResource{
			Resource: "pods",
		}
	case "Ingress":
		groupResource = schema.GroupResource{
			Resource: "ingresses",
			Group:    "extensions",
		}
	}

There is no support for deployment and when i changed kind to pod it started working.

Closing this issue