ohsawa0515/gcp-gpu-stackdriver-reporting

This repository provides a tool that sends metrics on GPU utilization on Google Compute Engine (GCE) to Stackdriver.

gcp-gpu-stackdriver-reporting

This repository provides a tool that sends metrics on GPU utilization on Google Compute Engine (GCE) to Stackdriver.

This tools is able to supports Linux only.

• Ubuntu 16.04/18.04

Installation

Download binary

Download it from releases page and extract it to /usr/local/bin.

$ curl -L -O https://github.com/ohsawa0515/gcp-gpu-stackdriver-reporting/releases/download/<version>/gcp-gpu-stackdriver-reporting_linux_amd64.tar.gz
$ tar zxf gcp-gpu-stackdriver-reporting_linux_amd64.tar.gz
$ mv ./gcp-gpu-stackdriver-reporting /usr/local/bin/
$ chmod +x /usr/local/bin/gcp-gpu-stackdriver-reporting

go get

$ go get github.com/ohsawa0515/gcp-gpu-stackdriver-reporting
$ mv $GOPATH/gcp-gpu-stackdriver-reporting /usr/local/bin/
$ chmod +x /usr/local/bin/gcp-gpu-stackdriver-reporting

Run as systemd

$ cat <<-EOH > /lib/systemd/system/gcp-gpu-stackdriver-reporting.service
[Unit]
Description=GPU Utilization Metric Reporting
[Service]
Type=simple
PIDFile=/run/gcp-gpu-stackdriver-reporting.pid
ExecStart=/usr/local/bin/gcp-gpu-stackdriver-reporting
User=root
Group=root
WorkingDirectory=/
Restart=always
[Install]
WantedBy=multi-user.target
EOH
$ systemctl daemon-reload
$ systemctl enable gcp-gpu-stackdriver-reporting.service
$ systemctl start gcp-gpu-stackdriver-reporting.service

Run as docker container

NVIDIA driver is required. Please install from here.

$ docker pull ohsawa0515/gcp-gpu-stackdriver-reporting:latest
$ docker run -d --runtime=nvidia --rm ohsawa0515/gcp-gpu-stackdriver-reporting:latest

Run as Google Kubernetes Engine(GKE)

Create GKE cluster and node pools with GPU

See document.

Installing NVIDIA GPU device drivers.

$ kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml

gcp-gpu-stackdriver-reporting apply into GKE as daemonset

$ kubectl apply -f daemonset-sample.yaml