GPU sharing
One of the biggest advantages of GPUs on Kubernetes is the ability to share them. Most workloads don't need the full memory and compute of an individual GPU, so sharing GPUs across multiple processes increases GPU utilization and cuts infrastructure cost.
GPU sharing currently works through 3 mechanisms: Time Slicing, Multi-Instance GPU (MIG), and Multi-Process Service (MPS).
Time Slicing
Time Slicing interleaves GPU workloads using a time-slicing scheduler that runs multiple CUDA processes concurrently through time-sharing. Once enabled, the GPU shares its resources fairly across processes by switching between them at regular intervals. This introduces overhead from constant process switching (frame stuttering and higher latency), and Time Slicing has no memory isolation or allocation limits between shared processes, which can cause frequent out-of-memory (OOM) errors.
Multi-Instance GPU (MIG)
MIG is an NVIDIA architecture that subdivides a GPU into separate, isolated partitions — isolated from each other in memory bandwidth, cache, and processing cores. Each instance's graphics processor is independent, with its own dedicated memory system: the on-chip port crossbar, L2 cache, memory controllers, and DRAM bus addresses are uniquely assigned per instance. This gives fault tolerance, with predictable throughput and latency per instance.
MIG partitions are named by their memory and compute resources — for example, 1g.6gb is a slice with 6 GB of memory. MIG doesn't support custom partition sizes or counts: each GPU model supports only a specific set of MIG configurations based on its hardware, which limits resource optimization, and MIG devices must follow specific arrangement rules, further limiting flexibility.
MIG gives the highest level of isolation between processes of the three mechanisms, but it's inflexible and only works on certain GPU architectures (Ampere and Hopper). On Kubernetes, NVIDIA's GPU Operator makes MIG easier to use, though limitations remain — you define a set of MIG configurations in a ConfigMap and apply them per node by labeling the node. Keeping GPU usage efficient means continuously adjusting that ConfigMap as workload resource needs change, which is inconvenient enough that MIG configurations are often left unchanged over time (or not applied at all), leading to inefficient and costly GPU usage. Automatic GPU partitioning (MPS) addresses this.
Multi-Process Service (MPS)
MPS uses the CUDA API's client-server model to run multiple processes on the same GPU simultaneously. The server manages GPU access across clients; clients connect to it through the client runtime, built into the CUDA Driver library and transparent to the application.
MPS is compatible with all current GPUs and offers the most flexibility — you can create GPU partitions with both memory and compute options. However, MPS doesn't fully isolate memory between processes, making it a middle ground between MIG and Time Slicing.
Compared to Time Slicing, MPS removes context-switching overhead by running processes in parallel through spatial sharing, improving performance, and gives each process its own GPU memory address space.
Processes under MPS still aren't fully isolated from each other: MPS can limit a client's memory and compute resources, but it doesn't provide error isolation or memory protection. A process that fails can trigger a GPU reset that affects every other process on that GPU. Kubernetes doesn't natively support MPS partitioning — using MPS on Kubernetes requires third-party software.