tomix86/oneagent-nvbit-extension

A Dynatrace OneAgent extension for gathering NVIDIA GPU metrics using NVBit

⚠️ This project was a proof of concept prepared during the course of my doctoral studies. It was abandoned early in development phase and is not stable. Nevertheless, feel free to draw inspiration and build upon it. ⚠️

OneAgent NVBit Extension

Foreword

Created by Tomasz Gajger.

Notice: although the author is a Dynatrace employee, this is a private project. It is not maintained nor endorsed by the Dynatrace.

The project is released under the MIT License.

Overview

A Dynatrace OneAgent extension for gathering NVIDIA GPU metrics using NVIDIA Binary Instrumentation Tool (NVBit).

The extension consists of two parts:

• native module, which is injected into monitored applications, gathers and publishes measurements,
• Python extension responsible for providing configuration to the native module, retrieving the metrics, aggregating them and sending to Dynatrace cluster.

All metrics are process-specific and reported per-PGI. The extension is capable of monitoring multiple GPUs, the metrics coming from all the devices will be aggregated and sent as combined timeseries. There is no support for sending separate timeseries per device.

Requirements

• OneAgent version >= 1.203.
• See Python part README.
• Compiled native module.

Setup and configuration

For a list of available configuration options, see extension README.

All processes for which the metrics should be gathered need to be instrumented manually with native module.

Reported metrics

The table below outlines metrics collected by the extension. Figures 1 and 2 exemplify how metrics are presented on the WebUI.

Key	Metric description
instructions_per_second	Count of instuctions executed per second
gpu_occupancy	Average occupancy achieved by kernels
gmem_access_coalescence	Average global memory accesses coalescence factor achieve by kernels

If there are multiple GPUs present, the metrics will be displayed in a joint fashion, i.e:

• instructions_per_second will be a sum of instructions executed on all devices,
• gpu_occupancy and gmem_access_coalescence will be an average from per-device usage metrics.

Fig 1. Host metrics reported by the extension

Fig 2. PGI metrics reported by the extension

Planned metrics (future enhancements)

• Global memory access efficiency (coalescence factor)
• Branch divergence
• GPU time (the time it took for the computations on GPU to complete)

Alerting

There are no built-in alerts defined yet.