How to use the NVIDIA MIG technology on GPU Instances
Reviewed on 04 January 2024 • Published on 31 August 2023
Note
Scaleway offers MIG compatible GPU Instances such as H100 PCIe GPU Instances
Nvidia uses the term GPU instance to designate a MIG partition of a GPU (MIG= Multi-Instance GPU)
To avoid confusion, we will use the term GPU Instance in this document to designate the Scaleway GPU Instance, and MIG partition in the context of the MIG feature.
NVIDIA Multi-Instance GPU (MIG) is a technology introduced by NVIDIA to enhance the utilization and flexibility of their data center GPUs, specifically designed for virtualization and multi-tenant environments.
It allows a single physical GPU to be partitioned into up to seven smaller Instances, each of which operates as an independent MIG partition with its own dedicated resources, such as memory, compute cores, and video outputs.
MIG ensures one client cannot impact the work or scheduling of other clients and provides enhanced isolation for customers.
With MIG, you are able to see and schedule jobs on virtual MIG partitions as if they were physical GPUs. MIG works with Linux operating systems and containers using Docker Engine, with support for Kubernetes and virtual machines using hypervisors such as Red Hat Virtualization and VMware vSphere.
MIG also provides you with the following benefits:
Resource sharing: MIG allows multiple users or workloads to share a single physical GPU, which can lead to better resource utilization and cost savings.
Complete isolation: Each MIG partition is isolated from the others, ensuring that workloads running on one Instance do not interfere with or impact the performance of workloads in other Instances. With MIG, each MIG partition’s processors have separate and isolated paths through the entire memory system - the on-chip crossbar ports, L2 cache banks, memory controllers, and DRAM address buses are all assigned uniquely to an individual Instance.
Guaranteed resources: MIG partitions can be configured with specific resource allocations, providing dedicated performance to different workloads. This is especially important in multi-tenant environments.
For more information about NVIDIA MIG, refer to the official MIG documentation.
The NVIDIA driver provides several predefined profiles you can choose from while setting up the MIG (Multi-Instance GPU) feature on the H100.
These profiles determine the sizes and functionalities available of the MIG partitions that users can generate. Additionally, the driver supplies details regarding placements, which specify the types and quantities of Instances that can be established.
Run the following command to list the possible placements available. The syntax of the placement is {<index>}:<GPU Slice Count> and shows the placement of the instances on the GPU.
Successfully created GPU instance ID 2 on GPU 0 using profile MIG 3g.40gb (ID 9)
Successfully created compute instance ID 0 on GPU 0 GPU instance ID 2 using profile MIG 3g.40gb (ID 2)
Successfully created GPU instance ID 7 on GPU 0 using profile MIG 1g.10gb (ID 19)
Successfully created compute instance ID 0 on GPU 0 GPU instance ID 7 using profile MIG 1g.10gb (ID 0)
Successfully created GPU instance ID 8 on GPU 0 using profile MIG 1g.10gb (ID 19)
Successfully created compute instance ID 0 on GPU 0 GPU instance ID 8 using profile MIG 1g.10gb (ID 0)
Successfully created GPU instance ID 9 on GPU 0 using profile MIG 1g.10gb (ID 19)
Successfully created compute instance ID 0 on GPU 0 GPU instance ID 9 using profile MIG 1g.10gb (ID 0)
The command nvidia-smi mig -cgi 9,19,19,19 -C is used to manage NVIDIA MIG partitions and their configurations using the NVIDIA System Management Interface (nvidia-smi) tool.
Here is a breakdown of the components of the command:
nvidia-smi: is the command-line tool provided by NVIDIA to interact with and manage NVIDIA GPUs.
mig: is a subcommand of nvidia-smi specifically used for managing MIG partitions. MIG (Multi-Instance GPU) allows a single NVIDIA GPU to be partitioned into multiple smaller instances, each with its own memory, compute, and other resources.
-cgi 9,19,19,19: this flag specifies the MIG partition configuration. The numbers following the flag represent the MIG partitions for each of the four MIG device slices. In this case, there are four slices with configurations 9, 19, 19, and 19 compute instances each. These numbers correspond to the profile IDs retrieved previously. Note that you can use either of the following:
Profile ID (e.g. 9, 14, 5)
Short name of the profile (e.g. 3g.40gb)
Full profile name of the instance (e.g. MIG 3g.40gb)
-C: this flag automatically creates the corresponding compute instances for the MIG partitions.
The command instructs the nvidia-smi tool to set up a MIG configuration where the GPU is divided into four slices, each containing different numbers of MIG partition configurations as specified: an MIG 3g.40gb (Profile ID 9) for the first slice, and an MIG 1g.10gb (Profile ID 19) for each of the remaining three slices.
Note
Running CUDA workloads on the GPU requires the creation of MIG partitions along with their corresponding compute instances. Just enabling MIG mode on the GPU is not enough to achieve this.
MIG devices are not retained after a system reboot, meaning that resetting your GPU or system requires you to set up MIG configurations again. You can use the NVIDIA MIG Partition Editor (mig-parted) tool for automated assistance in this process.
With it, you can create a systemd service, which can be used to reestablish the MIG geometry when the system starts up.
Run the following command to verify the MIG configuration of the GPU:
Launch a Jupyter Notebook within a Docker container on the MIG 3g.40gb MIG partition. Once initiated, you can reach it by connecting to port 8888 using the public IP of your GPU Instance.
root@my-h100-instance:~# sudo docker run --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=MIG-da06d78f-7534-56a0-a062-62fef012be9 -p 8888:8888 jupyter/minimal-notebook