Saba Shahrukh September 13, 2026 0 If you want to keep track of your post-reading status, please register on the site.

Container Lifecycle Interception and Multi-Tenant GPU Isolation

Deploying hardware-accelerated workloads within containerized environments introduces a fundamental challenge: container runtimes (such as runc or crun) are designed to enforce strict process, mount, and network isolation using native Linux namespaces and cgroups. However, high-performance deep learning models require zero-overhead access to host PCIe devices and host-level user-space driver libraries.

Rather than running full virtual machines or rewriting container engines, modern AI infrastructure relies on OCI (Open Container Initiative) runtime hooks. The NVIDIA Container Toolkit intercepts container lifecycle events to dynamically inject host character device nodes, mount host-installed CUDA driver libraries, and enforce multi-tenant isolation at the Linux kernel boundary.

1. Intercepting the Container Lifecycle: --gpus Delegation

When an engineer executes docker run --gpus '"device=0,1"', the container runtime does not natively understand how to interact with physical silicon. Instead, the flag triggers a sequence of runtime delegations that alter the container’s OCI runtime specification before the entrypoint process executes.

+---------------------------------------------------------------------------------------+
| DOCKER CLI / KUBERNETES CONTAINERD                                                    |
|  - Parses flag: --gpus '"device=0,1"'                                                 |
|  - Sets environment variables: NVIDIA_VISIBLE_DEVICES=0,1                             |
+---------------------------------------------------------------------------------------+
                                           |
                                           v Passes OCI Spec (config.json)
+---------------------------------------------------------------------------------------+
| CONTAINER ENGINE (containerd / Docker Daemon)                                         |
|  - Delegates runtime execution to: nvidia-container-runtime                           |
+---------------------------------------------------------------------------------------+
                                           |
                                           v Modifies OCI spec
+---------------------------------------------------------------------------------------+
| NVIDIA CONTAINER RUNTIME                                                              |
|  - Injects prestart hook into OCI config.json: nvidia-container-runtime-hook          |
|  - Hands control to low-level runtime: runc                                           |
+---------------------------------------------------------------------------------------+
                                           |
                                           v Creates Container Namespaces & RootFS
+---------------------------------------------------------------------------------------+
| LOW-LEVEL CONTAINER RUNTIME (runc)                                                    |
|  1. Creates namespaces (pid, mnt, net, ipc, user)                                     |
|  2. Mounts container root filesystem (rootfs)                                         |
|  3. EXECUTES PRESTART HOOK -> nvidia-container-toolkit                                |
+---------------------------------------------------------------------------------------+
                                           |
                                           v Intercepts Lifecycle (Host Root Namespace)
+---------------------------------------------------------------------------------------+
| OCI PRESTART HOOK (nvidia-container-toolkit)                                          |
|  - Reads container PID and mounts rootfs namespace                                    |
|  - Injects character device nodes (/dev/nvidia0, /dev/nvidia-uvm) via mknod           |
|  - Bind-mounts host user-space driver libraries (libcuda.so)                          |
|  - Configures Linux cgroups (devices subsystem) to whitelist allocated GPUs          |
+---------------------------------------------------------------------------------------+
                                           |
                                           v Hook completes execution
+---------------------------------------------------------------------------------------+
| CONTAINER ENTRYPOINT PROCESS                                                          |
|  - Launches application (e.g., python train.py) inside isolated environment           |
|  - Transparently accesses physical GPUs with zero virtualization overhead             |
+---------------------------------------------------------------------------------------+

The OCI Hook Execution Sequence

  1. Spec Modification: The high-level runtime (nvidia-container-runtime) intercepts the OCI specification (config.json) and inserts an entry into the hooks.prestart array.
  2. Namespace Creation: The low-level runtime (runc) sets up the container’s Linux namespaces (pid, mnt, net, ipc) and mounts the container image’s root filesystem (rootfs).
  3. Prestart Hook Interception: Right before runc executes the container’s entrypoint command (e.g., python train.py), it executes the binaries listed in the prestart hook array.
  4. Host Context Execution: The nvidia-container-toolkit runs inside the host’s root namespace, allowing it to inspect host hardware state, query host driver paths via ldconfig, and manipulate the container’s rootfs namespace directly via nsenter / setns.

2. Hardware Passthrough Mechanics: Device Nodes & Library Injection

Because container images do not (and should not) ship with host hardware drivers, the OCI hook dynamically bridges the host operating system’s driver stack into the container’s isolated filesystem namespace during container startup.

+-----------------------------------------------------------------------------------+
| HOST FILESYSTEM & KERNEL SPACE (Ring 0 / Host RootFS)                             |
|                                                                                   |
|  Host Device Nodes:             Host Driver Libraries:                            |
|  - /dev/nvidia0 (Major 195:0)    - /usr/lib/x86_64-linux-gnu/libcuda.so.550.54     |
|  - /dev/nvidiactl (195:255)     - /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.550.54 |
|  - /dev/nvidia-uvm (236:0)                                                        |
+-----------------------------------------------------------------------------------+
           |                                             |
           | mknod (Inject character devices)            | bind-mount (Inject shared libraries)
           v                                             v
+-----------------------------------------------------------------------------------+
| CONTAINER MOUNT & DEVICE NAMESPACE (Isolated RootFS)                              |
|                                                                                   |
|  Injected Device Files:         Injected Driver Libraries:                        |
|  - /dev/nvidia0                 - /usr/lib/x86_64-linux-gnu/libcuda.so.1          |
|  - /dev/nvidiactl               - /usr/lib/x86_64-linux-gnu/libcuda.so           |
|  - /dev/nvidia-uvm             - /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1    |
+-----------------------------------------------------------------------------------+

1. Dynamic Character Device Node Injection

In Linux, hardware devices are exposed in user space as character device files identified by a Major Number (which identifies the kernel driver family) and a Minor Number (which identifies the specific physical device index or control channel).

During prestart execution, the toolkit hook performs the following operations:

  • Interrogates Host Hardware: It identifies the physical GPU target mapped to the container request.
  • Executes mknod in Container Space: The hook calls mknod() inside the container’s /dev path to build hardware control nodes on demand:
    • /dev/nvidia0: Physical GPU 0 (Character Device, Major 195, Minor 0)
    • /dev/nvidiactl: Global GPU Driver Control Channel (Character Device, Major 195, Minor 255)
    • /dev/nvidia-uvm: Unified Virtual Memory Management Node (Character Device, Major 236, Minor 0)

2. Host User-Space Library Bind-Mounting

To allow container applications to issue ioctl calls to these injected device nodes, the user-space CUDA driver libraries must match the exact host kernel module version (nvidia.ko).

The hook inspects the host’s dynamic linker cache (/etc/ld.so.cache) and bind-mounts host shared object libraries into the container’s library search paths:

  • libcuda.so.XXX.YY$\rightarrow$/usr/lib/x86_64-linux-gnu/libcuda.so.1
  • libnvidia-ml.so.XXX.YY$\rightarrow$/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1
  • libnvidia-ptxjitcompiler.so.XXX.YY$\rightarrow$/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.1

By bind-mounting host driver .so libraries dynamically at container startup, container images remain portable across different host driver versions.

3. Multi-Tenant GPU Isolation: Visibility Scoping & Cgroup Enforcement

In multi-tenant cloud infrastructure (such as Kubernetes clusters), physical GPUs must be partitioned safely among multiple isolated workloads. Hardware isolation is enforced through a combination of Environment Variable Scoping and Linux Control Groups (cgroups).

Environment Variable Control Interface

The NVIDIA Container Toolkit relies on two primary environment variables passed in the container specification:

  1. NVIDIA_VISIBLE_DEVICES: Controls hardware device exposure.
    • NVIDIA_VISIBLE_DEVICES=all: Exposes all host GPUs.
    • NVIDIA_VISIBLE_DEVICES=0,1: Exposes physical GPU index 0 and 1 only.
    • NVIDIA_VISIBLE_DEVICES=GPU-a1b2c3d4-...: Exposes specific GPUs matching unique UUIDs.
    • NVIDIA_VISIBLE_DEVICES=none: Suppresses all GPU device node injection.
  2. NVIDIA_DRIVER_CAPABILITIES: Controls library injection sub-components.
    • compute: Injects CUDA and OpenCL runtimes (libcuda.so).
    • utility: Injects NVML monitoring libraries (libnvidia-ml.so).
    • graphics / video: Injects OpenGL, Vulkan, and NVENC/NVDEC hardware video codecs.

Hardware Enforcement via Linux devices Cgroup

Environment variables alone do not prevent malicious processes inside a container from creating device nodes manually using mknod. Hardware access is securely restricted at the host kernel level using Linux cgroups (devices subsystem).

Let $D_{\text{host}}$ represent the complete set of physical GPU character device nodes on the host OS:

$$D_{\text{host}} = \{ \text{dev}_{195,0}, \text{dev}_{195,1}, \dots, \text{dev}_{195,N-1} \} \cup \{ \text{dev}_{195,255}, \text{dev}_{236,0} \}$$

When a container is granted access to a subset of devices $S \subset \{0, 1, \dots, N-1\}$, the toolkit updates the container’s cgroup configuration:

$$\text{Cgroup Rule Mask} = \{ \text{c } 195:i \text{ rwm} \mid i \in S \} \cup \{ \text{c } 195:255 \text{ rwm}, \text{c } 236:0 \text{ rwm} \}$$

Cgroup Rule Specification Mechanics

  • cgroup v1 (devices.allow / devices.deny): The toolkit issues a global deny rule (b *:* rwm and c *:* rwm), revoking access to all block and character devices. It then writes explicit permission rules to devices.allow:
    Plaintextc 195:0 rwm # Permit Read, Write, Mknod on Physical GPU 0 c 195:255 rwm # Permit Read, Write, Mknod on Global Control Node c 236:0 rwm # Permit Read, Write, Mknod on UVM Node
  • cgroup v2 (eBPF Device Filters): In cgroup v2, device filtering is implemented using attached BPF_PROG_TYPE_CGROUP_DEVICE eBPF programs. The kernel executes an inline eBPF filter on every open(), mknod(), or ioctl() system call issued by container processes, verifying device major/minor numbers before passing operations to physical hardware.

4. Lifecycle Execution & Isolation Matrix

Container Lifecycle PhaseExecuting EntitySystem ActionSecurity & Isolation State
1. Spec GenerationDocker / KubernetesParses --gpus flag and populates NVIDIA_VISIBLE_DEVICES in spec.User-space configuration validation.
2. Namespace Creationrunc / crunAllocates isolated PID, Mount, Network, and IPC namespaces.Process isolation active; no hardware visibility.
3. Hook Triggernvidia-container-toolkitIntercepts prestart phase inside host root namespace context.Inspects host hardware topology and cgroup boundaries.
4. Device Injectionmknod System CallCreates /dev/nvidia* character nodes inside container /dev.Hardware device nodes bound to physical major/minor numbers.
5. Library MountLinux bind MountMounts host libcuda.so.XXX libraries into container /usr/lib/.Guarantees exact binary match with host nvidia.ko.
6. Cgroup EnforcementLinux KernelApplies devices.allow rules or attaches eBPF cgroup filter.Kernel-enforced hardware isolation. Access to non-assigned GPUs blocked at syscall boundary.
7. Entrypoint ExecutionApplication ProcessRuns PyTorch/CUDA application.Direct bare-metal hardware performance.

5. End-to-End Inspection & Isolation Diagnostic Script

The following Python script inspects the container runtime environment. It queries active environment configuration variables, verifies character device node injection, checks assigned Linux cgroup device whitelists, and validates multi-tenant GPU isolation bounds using ctypes and system interfaces.

Python

import os
import sys
import ctypes

def inspect_environment_scoping():
    """Inspects OCI environment variables passed by container runtime."""
    print("=== 1. OCI Isolation Environment Scoping ===")
    visible_devices = os.environ.get("NVIDIA_VISIBLE_DEVICES", "Not Set")
    driver_caps = os.environ.get("NVIDIA_DRIVER_CAPABILITIES", "Not Set")
    
    print(f"NVIDIA_VISIBLE_DEVICES    : {visible_devices}")
    print(f"NVIDIA_DRIVER_CAPABILITIES : {driver_caps}\n")

def inspect_injected_device_nodes():
    """Scans container /dev filesystem for dynamically injected character devices."""
    print("=== 2. Injected Character Device Node Inspection ===")
    
    dev_dir = "/dev"
    nvidia_nodes = []
    
    if os.path.exists(dev_dir):
        for entry in os.listdir(dev_dir):
            if entry.startswith("nvidia"):
                full_path = os.path.join(dev_dir, entry)
                try:
                    stat_info = os.stat(full_path)
                    # Extract Major and Minor character device numbers
                    rdev = stat_info.st_rdev
                    major = os.major(rdev)
                    minor = os.minor(rdev)
                    nvidia_nodes.append((entry, major, minor))
                except OSError:
                    continue

    if nvidia_nodes:
        for node_name, major, minor in sorted(nvidia_nodes, key=lambda x: x[0]):
            print(f"Device Node: /dev/{node_name:<15} | Major: {major:<3} | Minor: {minor:<3}")
    else:
        print("[WARNING] No /dev/nvidia* device nodes detected inside container filesystem!")
    print()

def inspect_cgroup_device_rules():
    """Inspects Linux cgroup device rules to verify kernel-level hardware restriction."""
    print("=== 3. Linux Cgroup Device Isolation Inspection ===")
    
    cgroup_v1_path = "/sys/fs/cgroup/devices/devices.list"
    cgroup_v2_controllers = "/sys/fs/cgroup/cgroup.controllers"
    
    if os.path.exists(cgroup_v1_path):
        print(f"Cgroup Mode: cgroup v1 detected ({cgroup_v1_path})")
        try:
            with open(cgroup_v1_path, "r") as f:
                rules = f.read().strip().split("\n")
            print("Active Device Whitelist Rules:")
            for rule in rules:
                print(f"  -> {rule}")
        except Exception as e:
            print(f"  [ERROR] Reading cgroup v1 devices list failed: {e}")
            
    elif os.path.exists(cgroup_v2_controllers):
        print("Cgroup Mode: cgroup v2 unified hierarchy detected.")
        print("Device Filtering: Enforced via inline eBPF cgroup programs (BPF_PROG_TYPE_CGROUP_DEVICE).")
        
        # Check proc self cgroup status
        try:
            with open("/proc/self/cgroup", "r") as f:
                cgroup_entry = f.read().strip()
            print(f"Process Cgroup Scope: {cgroup_entry}")
        except Exception as e:
            print(f"  [ERROR] Reading /proc/self/cgroup failed: {e}")
    else:
        print("[UNKNOWN] Unable to locate standard Linux cgroup controller interfaces.")
    print()

def validate_cuda_device_enumeration():
    """Queries CUDA API via ctypes to verify visible hardware index scoping."""
    print("=== 4. Hardware Isolation & CUDA Enumeration Validation ===")
    
    try:
        cuda_driver = ctypes.CDLL("libcuda.so")
    except OSError:
        try:
            cuda_driver = ctypes.CDLL("libcuda.so.1")
        except OSError:
            print("[ERROR] Could not load libcuda.so! Injected user-space library missing.")
            return

    # Initialize CUDA Driver API
    if cuda_driver.cuInit(0) != 0:
        print("[ERROR] Failed to initialize CUDA Driver API via cuInit(0).")
        return

    # Query Visible Hardware Count
    device_count = ctypes.c_int()
    cuda_driver.cuDeviceGetCount(ctypes.byref(device_count))
    count = device_count.value

    print(f"Accessible Hardware Devices Count: {count}")
    
    for i in range(count):
        dev = ctypes.c_int()
        cuda_driver.cuDeviceGet(ctypes.byref(dev), i)
        
        # Query Device Name
        name_buf = ctypes.create_string_buffer(256)
        cuda_driver.cuDeviceGetName(name_buf, 256, dev.value)
        
        # Query PCI Bus ID to verify hardware indexing mapping
        pci_buf = ctypes.create_string_buffer(64)
        cuda_driver.cuDeviceGetPCIBusId(pci_buf, 64, dev.value)
        
        print(f"  Logical Index [{i}] -> CUDA Device Handle ({dev.value}) | Name: {name_buf.value.decode()} | Bus ID: {pci_buf.value.decode()}")

if __name__ == "__main__":
    inspect_environment_scoping()
    inspect_injected_device_nodes()
    inspect_cgroup_device_rules()
    validate_cuda_device_enumeration()
Category: 

Leave a Comment