Prometheus could not pull metric from node-gpu-expoter

[kubeflow]

Prometheus could not pull metric from node-gpu-expoter

[ldd91]

I wonder if you have the plane to open source node-gpu-expoter code

[xiaozhouX]

I wonder if you have the plane to open source node-gpu-expoter code

What your mean that node-gpu-expoter become down ? It pod still running or not ?
Can you print some detail for node-gpu-expoter, like exec
kubelet -n kube-system describe po node-gpu-expoter

[ldd91]

Thank you for timely reply,the pod is still running,when i exec
kubelet -n kube-system describe po node-gpu-expoter

Last Status: Terminated
Reason: OOMKilled
Exit Code: 137

[xiaozhouX]

Thanks for your feedback, the node-gpu-expoter memory limit is too small. You can adjust it by kubectl -n kube-system edit ds node-gpu-exporter , and modify resource.limits.memory to memory: 200Mi. We will change the default yaml template, and check whether there is memory leak.
The expoter will open source soon.

[ldd91]

Thanks you,i will have a try later.I have a problem with prometheus,i can not query date from prometheus UI.When I exec kubectl logs -f prometheus-core-5bf584fdf-h4nhq -n monitoring,it shows

time="2019-01-11T16:31:57Z" level=warning msg="Flag -storage.local.memory-chunks is deprecated. Its value 2000000 is used to override -storage.local.target-heap-size to 6144000000." source="config.go:317"
time="2019-01-11T16:31:57Z" level=info msg="Starting prometheus (version=1.7.0, branch=master, revision=bfa37c8ee39d11078662dce16c162a61dccf616c)" source="main.go:88"
time="2019-01-11T16:31:57Z" level=info msg="Build context (go=go1.8.3, user=root@7a6329cc02bb, date=20170607-09:43:48)" source="main.go:89"
time="2019-01-11T16:31:57Z" level=info msg="Host details (Linux 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 prometheus-core-5bf584fdf-h4nhq (none))" source="main.go:90"
time="2019-01-11T16:31:57Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yaml" source="main.go:252"
time="2019-01-11T16:31:57Z" level=info msg="Loading series map and head chunks..." source="storage.go:428"
time="2019-01-11T16:31:57Z" level=info msg="0 series loaded." source="storage.go:439"
time="2019-01-11T16:31:57Z" level=info msg="Starting target manager..." source="targetmanager.go:63"
time="2019-01-11T16:31:57Z" level=info msg="Listening on :9090" source="web.go:259"

[ldd91]

Here is my prometheus.yaml
args:
- '-storage.local.retention=24h'
- '-storage.local.memory-chunks=2000000'
- '-config.file=/etc/prometheus/prometheus.yaml'
- '-storage.local.path=/prometheus'
- '-alertmanager.url=http://alertmanager:9093/'
ports:
- name: webui
containerPort: 9090
resources:
requests:
cpu: 500m
memory: 1000M
limits:
cpu: 500m
memory: 1000M
volumeMounts:
- name: config-volume
mountPath: /etc/prometheus
- name: rules-volume
mountPath: /etc/prometheus-rules
- name: prometheus-data
mountPath: /prometheus
volumes:
- name: config-volume
configMap:
name: prometheus-core
- name: rules-volume
configMap:
name: prometheus-rules
- name: prometheus-data
#emptyDir: {}
hostPath:
path: /prometheus

[xiaozhouX]

Here is my prometheus.yaml
args:

• '-storage.local.retention=24h'
• '-storage.local.memory-chunks=2000000'
• '-config.file=/etc/prometheus/prometheus.yaml'
• '-storage.local.path=/prometheus'
• '-alertmanager.url=http://alertmanager:9093/'
  ports:
• name: webui
  containerPort: 9090
  resources:
  requests:
  cpu: 500m
  memory: 1000M
  limits:
  cpu: 500m
  memory: 1000M
  volumeMounts:
• name: config-volume
  mountPath: /etc/prometheus
• name: rules-volume
  mountPath: /etc/prometheus-rules
• name: prometheus-data
  mountPath: /prometheus
  volumes:
• name: config-volume
  configMap:
  name: prometheus-core
• name: rules-volume
  configMap:
  name: prometheus-rules
• name: prometheus-data
  #emptyDir: {}
  hostPath:
  path: /prometheus

It seems different from our yaml template. Did you deploy prometheus by prometheus operator? I don't know your image version and discovery config. If you can query other metrics, maybe you can check the discovery config.

[ldd91]

Thank you for replay,I have solved this problem,it was caused by ntp,in my cluster i didn't config the ntp server.