`gpustat`

Just less than nvidia-smi?

Screenshot: gpustat -cp

NOTE: This works with NVIDIA Graphics Devices only, no AMD support as of now. Contributions are welcome!

Self-Promotion: A web interface of gpustat is available (in alpha)! Check out gpustat-web.

Quick Installation

Install from PyPI:

pip install gpustat

If you don't have root (sudo) privilege, please try installing gpustat on user namespace: pip install --user gpustat.

To install the latest version (master branch) via pip:

pip install git+https://github.com/wookayin/gpustat.git@master

Important

DO NOT: pip install pynvml, nor include pynvml as a dependency in your python project. This will not work.

Instead: pip install nvidia-ml-py. nvidia-ml-py is NVIDIA's the official python binding for NVML.

gpustat 1.2+: Requires nvidia-ml-py >= 12.535.108 (#161)
gpustat 1.0+: Requires NVIDIA Driver 450.00 or higher and nvidia-ml-py >= 11.450.129.
If your NVIDIA driver is too old, you can use older gpustat versions (pip install gpustat<1.0). See #107 for more details.

$ gpustat

Options (Please see gpustat --help for more details):

--color : Force colored output (even when stdout is not a tty)
--no-color : Suppress colored output
-u, --show-user : Display username of the process owner
-c, --show-cmd : Display the process name
-f, --show-full-cmd : Display full command and cpu stats of running process
-p, --show-pid : Display PID of the process
-F, --show-fan : Display GPU fan speed
-e, --show-codec : Display encoder and/or decoder utilization
-P, --show-power : Display GPU power usage and/or limit (draw or draw,limit)
-a, --show-all : Display all gpu properties above
--id : Target and query specific GPUs only with the specified indices (e.g. --id 0,1,2)
--no-processes : Do not display process information (user, memory) (#133)
--watch, -i, --interval : Run in watch mode (equivalent to watch gpustat) if given. Denotes interval between updates.
--json : JSON Output (#10)
--print-completion (bash|zsh|tcsh) : Print a shell completion script. See #131 for usage.

Try gpustat --debug if something goes wrong.
To periodically watch, try gpustat --watch or gpustat -i (#41).
- For older versions, one may use watch --color -n1.0 gpustat --color.
Running nvidia-smi daemon (root privilege required) will make querying GPUs much faster and use less CPU (#54).
The GPU ID (index) shown by gpustat (and nvidia-smi) is PCI BUS ID, while CUDA uses a different ordering (assigns the fastest GPU with the lowest ID) by default. Therefore, in order to ensure CUDA and gpustat use same GPU index, configure the CUDA_DEVICE_ORDER environment variable to PCI_BUS_ID (before setting CUDA_VISIBLE_DEVICES for your CUDA program): export CUDA_DEVICE_ORDER=PCI_BUS_ID.

[0] GeForce GTX Titan X | 77°C,  96 % | 11848 / 12287 MB | python/52046(11821M)

[0]: GPU index (starts from 0) as PCI_BUS_ID
GeForce GTX Titan X: GPU name
77°C: GPU Temperature (in Celsius)
96 %: GPU Utilization
11848 / 12287 MB: GPU Memory Usage (Used / Total)
python/...: Running processes on GPU, owner/cmdline/PID (and their GPU memory usage)