NVIDIA Mirror
Sync Policy

The full NVIDIA CUDA mirror ships packages for every supported OS (Fedora, openSUSE, SLES, Ubuntu 12.04–24.04, RHEL, Debian, WSL…) and every CUDA and driver version going back years. A full mirror consumes 2–4 TB. Most of that serves zero machines here.

This policy narrows the sync to two Ubuntu LTS releases, seven recent CUDA versions, five driver branches, and the companion libraries GPU servers actually use. In production we run it in all-versions mode (every patch build kept, not just the latest), landing at ~500 GB on the 1 TB VM.

Verified against the mirror's Packages index (Aug 2026).

OS

Excluded: ubuntu2004, rhel*, fedora*, opensuse*, sles*, wsl*, ubuntu1*

CUDA — rolling window

Sub-packages synced per version: toolkit, libraries, compiler, runtime, cudart, nvcc, nvrtc, nvtx, cupti, gds-tools, cuBLAS, cuFFT, cuSolver, cuSparse, cuRAND, NPP, nvJPEG, cuFile, nvJitLink, nvFatbin

Excluded: cuda-samples, cuda-documentation, cuda-visual-tools, cuda-nvvp, nsight-systems, nsight-compute

Driver + Fabric Manager

Driver + Fabric Manager come from the NVIDIA CUDA repo alone — the Ubuntu archive is deliberately no longer consulted. The archive publishes only one build per branch, so the day 580.173.02 shipped, 580.159.03 became unobtainable and a node reinstalled with the identical command got a different driver. NVIDIA's repo keeps the full build history (12 builds of 580 alone), every one FM-matched, and every intra-set dependency is exact-versioned (= <ver>) — so pinning one build cascades the whole set instead of being dragged forward.

Every build listed under "Builds kept" stays installable by exact pin — that is the whole point of sourcing from NVIDIA. Both prop and -open kernel-module flavors are mirrored for every branch above.

# Install with exact version pin (FM must match driver)
sudo apt install nvidia-driver-580-server=580.159.03-* nvidia-fabricmanager-580=580.159.03-*

# Verify versions match
dpkg-query -W -f='${Package} ${Version}\n' nvidia-driver-580-server nvidia-fabricmanager-580

DCGM 4 — CUDA 12 and 13

datacenter-gpu-manager-4-cuda12
datacenter-gpu-manager-4-cuda13
datacenter-gpu-manager-4-core
datacenter-gpu-manager-4-multinode-cuda12/13
datacenter-gpu-manager-4-proprietary-cuda12/13
datacenter-gpu-manager-exporter

cuDNN — all (covers CUDA 11/12/13)

cudnn9-cuda-11/12/13
libcudnn9-cuda-11/12/13
libcudnn9-dev-cuda-11/12/13
libcudnn9-jit-cuda-11/12/13
libcudnn9-static-cuda-12/13

NCCL — CUDA 12 and 13 only

Filter by apt version string containing +cuda12 or +cuda13.

Latest in mirror: libnccl2 2.29.7-1+cuda12.9 / 2.29.7-1+cuda13.2

TensorRT — all versions in mirror

TensorRT packages have no CUDA suffix in the package name. All versions in mirror (10.4–10.15) build against CUDA 12.6–13.x — matches our range, no filtering needed.

libnvinfer10, libnvinfer-dev, libnvinfer-lean10, libnvinfer-dispatch10
libnvinfer-plugin10, libnvinfer-vc-plugin10
libnvonnxparsers10, tensorrt, tensorrt-libs, tensorrt-dev
python3-libnvinfer, python3-libnvinfer-dev

Excluded: libnvinfer-samples, libnvinfer-win-builder-resource10

DOCA-OFED — Ubuntu 22.04 and 24.04

Served from mirror at /mirror-repo/doca/<distro>/ (nginx: https://mirrors.siam.ai/doca/<distro>/).

Download source: developer.nvidia.com/doca-downloads
(Host-Server → DOCA-Host → Linux → x86_64 → doca-ofed → Ubuntu → 22.04 / 24.04)

# Download and store to mirror server (run on mirror server)
# ubuntu2404
wget "https://www.mellanox.com/downloads/DOCA/DOCA_v3.4.0/host/doca-host_3.4.0-085000-26.04-ubuntu2404_amd64.deb" \
  -O /mirror-repo/doca/ubuntu2404/doca-host_3.4.0-085000-26.04-ubuntu2404_amd64.deb

# ubuntu2204
wget "https://www.mellanox.com/downloads/DOCA/DOCA_v3.4.0/host/doca-host_3.4.0-085000-26.04-ubuntu2204_amd64.deb" \
  -O /mirror-repo/doca/ubuntu2204/doca-host_3.4.0-085000-26.04-ubuntu2204_amd64.deb

Install on GPU nodes via ./get-cuda-mirror.sh --doca=true (auto-detects distro, downloads from mirror).

Network install via mirror

sudo ./get-cuda-mirror.sh --nvidia-driver=<BRANCH> --cuda-toolkit=<VERSION> [OPTIONS]

# Full HGX install (DOCA + driver + CUDA in one run, one reboot)
sudo ./get-cuda-mirror.sh --doca=true --nvidia-driver=580 --cuda-toolkit=13-0 --nvitop=true
sudo reboot

# Driver + CUDA only
sudo ./get-cuda-mirror.sh --nvidia-driver=595 --cuda-toolkit=13-2 --nvitop=true

# VM without NVLink
sudo ./get-cuda-mirror.sh --nvidia-driver=590 --cuda-toolkit=12-9 --nvitop=true --no-nv=true

# Uninstall — pick interactively from what's installed
sudo ./get-cuda-mirror.sh --uninstall

# Uninstall specific components / everything
sudo ./get-cuda-mirror.sh --uninstall=doca,driver
sudo ./get-cuda-mirror.sh --uninstall=all

Offline / air-gapped node (--local-install)

For a node with no mirror access at install time: pre-stage the .deb files from the mirror, copy them to the node, then run the same installer against the local repo — no official NVIDIA egress.

# On a machine WITH mirror access: pull installer + the .deb set, copy to the node
wget https://mirrors.siam.ai/get-cuda-mirror.sh

# On the air-gapped node: install from the pre-staged local repo
sudo ./get-cuda-mirror.sh --local-install=true --nvidia-driver=580 --cuda-toolkit=13-0

Verify after reboot

nvidia-smi -L | wc -l
systemctl status nvidia-fabricmanager
lsmod | grep nvidia_peermem
dcgm.dcgmi discovery -l

Self-contained selective sync — downloads Packages.gz from NVIDIA, matches package names against policy.yaml, downloads only matching .deb files, and generates local repo metadata with apt-ftparchive. No external dependency on cuda-repo-management.

DRY_RUN=1 ./sync-mirror.sh

systemctl start cuda-mirror.service

sudo ./setup.sh --sync --doca --all-versions

journalctl -u cuda-mirror.service -f

DRY_RUN=1 ./cleanup-mirror.sh   # preview
DRY_RUN=0 ./cleanup-mirror.sh   # apply

Two retention modes, both driven by the same policy.yaml. Production uses all-versions on the 1 TB VM.

• FM version must match driver branch — mismatch causes service start failure.
• ubuntu2204 carries the same CUDA set as noble — CUDA 13.x is available for jammy too; running 13.x on jammy needs a driver ≥ 580 on that node.
• DOCA-OFED must be installed before the driver (same run) so nvidia_peermem links at DKMS build time — then one reboot at the end.
• --no-nv=true adds kernel blacklist — reboot required to take effect.
• DCGM takes ~10–20s after boot to report GPU state.