how it will work
How it fits into GPU.ai
GPU Observability is planned to chart the GPUs your work runs on. It will be built on open source tools, NVIDIA DCGM Exporter to read GPU metrics, Prometheus to store them, and Grafana to chart them. It is meant for the GPU Memory Scheduler and Kubernetes Orchestration, so you can see how jobs use the GPUs they run on.
- DCGM Exporterreads GPU metrics
- Prometheusstores them over time
- Grafanacharts them for you
Metrics from the GPU itself
DCGM Exporter reads utilization, memory, temperature, and power from each GPU.
Kept in Prometheus
Metrics are stored over time, so you can look back at a job after it ends.
Charted in Grafana
Dashboards per GPU, per job, and per cluster, set up for you.
NVIDIA DCGM, Prometheus, and Grafana belong to their respective owners. GPU.ai is not affiliated with or endorsed by them.