NVIDIA Mirror
Usage Guide

Pick your driver branch and CUDA version — the install command updates live. Copy and run on any GPU node.

# 1. Download installer from mirror
wget https://mirrors.siam.ai/get-cuda-mirror.sh
chmod +x get-cuda-mirror.sh

# 2. Install
sudo ./get-cuda-mirror.sh --nvidia-driver=595 --cuda-toolkit=13-2 --nvitop=true

Driver + Fabric Manager

How to install. Pass --nvidia-driver=<branch> (e.g. --nvidia-driver=590) to get that branch's newest build — the one tagged default above, resolved live from the mirror. To pin an exact build, pass the full version from that branch's row above (e.g. --nvidia-driver=<branch>.<build>). When a branch has more than one build on the mirror, every installable version is listed in its row so you can pin any of them. The matching fabricmanager (nvidia-fabricmanager-<branch> on older branches, plain nvidia-fabricmanager from 580 on) is installed automatically in the same apt transaction, at the same exact version.

Everything comes from NVIDIA. The mirror sources the driver stack from NVIDIA's CUDA repo only — never the Ubuntu archive. That is deliberate: Ubuntu publishes just the one current build per branch and rewrites it in place, so a build you installed last month is gone from upstream today and the fleet drifts apart. NVIDIA keeps every point release, which is what makes --nvidia-driver=<exact build> reproducible months later. Branches are auto-selected by an FM-gated rolling window: keep the newest 5 branches that have both a driver and a fabricmanager of the same exact version (floor 570), keep the newest 5 builds within each branch, and serve each dropped branch 30 days more before deletion. A branch whose FM has not shipped yet is skipped until it does. Driver .debs already on disk are folded back into each run, so a build stays installable even after upstream moves on.

Driver → max CUDA toolkit

Source: NVIDIA Data Center Driver release notes. Newer minor CUDA toolkits need a driver whose branch reaches that number — e.g. CUDA 13.2 needs driver ≥ 595, CUDA 13.1 needs ≥ 590. Older toolkits always work on newer drivers.

CUDA Toolkit — 7 versions

Companion libraries

Full HGX install (DOCA + driver + CUDA) — one run, one reboot

For HGX/NVSwitch nodes with ConnectX + GPUDirect RDMA. Install DOCA-OFED and the driver in a single command — reboot only once at the end.

# DOCA + driver + CUDA + nvitop in one command
wget https://mirrors.siam.ai/get-cuda-mirror.sh
chmod +x get-cuda-mirror.sh
sudo ./get-cuda-mirror.sh --doca=true --nvidia-driver=580 --cuda-toolkit=13-0 --nvitop=true

# Reboot once — loads DOCA ib_core + driver + nvidia_peermem together
sudo reboot

Driver + CUDA only (no DOCA)

Use when the GPU node has network access and doesn't need GPUDirect RDMA (e.g., VMs without IB).

# Download installer
wget https://mirrors.siam.ai/get-cuda-mirror.sh
chmod +x get-cuda-mirror.sh

# Latest driver + CUDA
sudo ./get-cuda-mirror.sh --nvidia-driver=595 --cuda-toolkit=13-2 --nvitop=true

# VM without NVLink
sudo ./get-cuda-mirror.sh --nvidia-driver=590 --cuda-toolkit=12-9 --nvitop=true --no-nv=true

Offline / air-gapped node (--local-install)

For a node that can't reach the mirror at install time, pre-stage the .deb files from the mirror, copy them to the node, then point the same installer at the local repo — no official NVIDIA egress.

# On a machine WITH mirror access — pull the installer + the .deb files you need
wget https://mirrors.siam.ai/get-cuda-mirror.sh
# (copy get-cuda-mirror.sh + the .deb set to the air-gapped node)

# On the air-gapped node — install from the pre-staged local repo
sudo ./get-cuda-mirror.sh --local-install=true --nvidia-driver=580 --cuda-toolkit=13-0

Verify after install

Run verify-install.sh with the same flags you installed with — it covers everything below plus the checks that are easy to miss by eye. See section 05. For a quick look by hand:

nvidia-smi -L | wc -l                       # GPU count
systemctl status nvidia-fabricmanager       # FM service
lsmod | grep nvidia_peermem                 # GPUDirect RDMA
dcgmi discovery -l                          # DCGM sees all GPUs
dpkg-query -W -f='${Package} ${Version}\n' nvidia-driver-580 nvidia-fabricmanager

Add libraries later (NCCL / cuDNN / TensorRT / DCGM)

Extra libraries install straight from the mirror with apt — no extra setup, no reaching out to NVIDIA.

# NCCL — multi-GPU / multi-node collective comms
sudo apt update
sudo apt install -y libnccl2 libnccl-dev

# cuDNN 9 (match your CUDA major — cuda-13 shown)
sudo apt install -y cudnn9-cuda-13 libcudnn9-dev-cuda-13

# TensorRT
sudo apt install -y tensorrt libnvinfer-dev python3-libnvinfer

# DCGM 4 — health / telemetry
sudo apt install -y datacenter-gpu-manager-4-cuda13

# NVIDIA Container Toolkit — GPUs in Docker
sudo apt install -y nvidia-container-toolkit

GPU stress test (gpu-burn) optional

Burn-in / thermal test — hammers every GPU with FP32 (or Tensor-core) matrix multiplies to surface unstable cards, throttling, or cooling problems. Needs the CUDA toolkit (nvcc) already installed.

# Build
sudo apt install -y git build-essential
git clone https://github.com/wilicc/gpu-burn.git
cd gpu-burn
make            # if nvcc not found: make CUDAPATH=/usr/local/cuda-13.0

# Run — 120 s across all GPUs
./gpu_burn 120

# Tensor-core (mixed precision) stress
./gpu_burn -tc 120

Node installed from the old mirror — apt update fails one-time fix

On a node set up before this mirror moved to curated metadata, apt update ends with:

E: The repository 'https://mirrors.siam.ai/compute/cuda/repos/ubuntu2404/x86_64  Release' is no longer signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.

Run this on that node. It repairs the apt entry and installs nothing:

wget https://mirrors.siam.ai/get-cuda-mirror.sh
chmod +x get-cuda-mirror.sh
sudo ./get-cuda-mirror.sh --fix-repo

Once per node, permanently. It is not needed again on a node that has had it, and a node installed from the current mirror never needs it at all.

Uninstall

# Interactive — lists only what's installed, then pick what to remove
sudo ./get-cuda-mirror.sh --uninstall

# Specific components only (no prompt)
sudo ./get-cuda-mirror.sh --uninstall=doca,driver

# Everything
sudo ./get-cuda-mirror.sh --uninstall=all

verify-install.sh checks a finished node against the flags it was installed with. Hand it the same flags as the install command and it answers "did this node get exactly what was asked for" — not just "is this node self-consistent". It is read-only: it installs nothing, changes nothing, starts no services. Exit code is 0 on a clean run, 1 if anything failed, so it drops straight into a provisioning pipeline.

wget https://mirrors.siam.ai/verify-install.sh
chmod +x verify-install.sh

# VM / no NVLink — same flags used to install
sudo ./verify-install.sh --nvidia-driver=580 --cuda-toolkit=13-0 --dcgm=true --nvitop=true --no-nv=true

# HGX with DOCA — same flags used to install
sudo ./verify-install.sh --doca=true --nvidia-driver=580 --cuda-toolkit=13-0 --dcgm=true --nvitop=true

# Also run dcgmi diag -r 2 (several minutes)
sudo ./verify-install.sh --nvidia-driver=580 --cuda-toolkit=13-0 --dcgm=true --deep

What it checks

The three that earn their keep

• One version across every driver package — a split stack is the exact failure this mirror exists to prevent, and nvidia-smi will not show it: it reports one number while a stray package sits on another build. The check compares all of them and prints the split by version if they disagree. Enablers that legitimately track their own release trains (dkms, DCGM, container toolkit, the EGL libraries) are identified by version shape and excluded.
• No Ubuntu-archive lineage — an archive build carries the distro triplet (-0ubuntu0.24.04.1) where NVIDIA's carries -1ubuntu1. Ubuntu publishes one build per branch and rewrites it in place, so anything from that lineage will silently land on a different build on the next machine. Any of it in the NVIDIA stack is a failure.
• Leftover apt pin — the installer writes a hard pin under /etc/apt/preferences.d/ while it works and removes it on exit. If a run is killed mid-way the pin survives, and from then on apt refuses packages for a reason nothing on screen explains. The check names the file and the rm that clears it.

Sample output

A VM node installed with --nvidia-driver=580 --cuda-toolkit=13-0 --dcgm=true --nvitop=true --no-nv=true, abridged:

==== 2. Driver version lock
[PASS] nvidia-smi reports driver 580.178.04  (max CUDA 13.0)
[PASS] all 17 driver packages at one version: 580.178.04-1ubuntu1
[PASS] running module matches the installed package (580.178.04)
[PASS] no Ubuntu-archive builds in the NVIDIA stack (NVIDIA lineage only)
[PASS] driver is on the requested branch 580 (build 580.178.04)
       kernel module flavor: open

==== 3. Secure Boot
[PASS] Secure Boot ON, nvidia module signed by: oa-test-mirros-0001 Secure Boot Module Signature key

==== 4. CUDA toolkit
[PASS] nvcc 13.0  (/usr/local/cuda/bin/nvcc)
[PASS] /usr/local/cuda -> /usr/local/cuda-13.0
[PASS] meta cuda-13-0 13.0.3-1 installed

==== 5. Fabric Manager
[PASS] no Fabric Manager, as expected with --no-nv=true

==== 6. NVLink
[PASS] NvLinkDisable: 1 — NVLink is off, as requested

==== 7. DCGM
[PASS] plugin datacenter-gpu-manager-4-cuda13 matches the driver's max CUDA (13.0)
[PASS] dcgmi discovery: 1 GPU(s), 0 NvSwitches found, 0 ConnectX found

==== 11. Hygiene
[PASS] no leftover apt driver pin
[PASS] apt dependency tree is clean
[WARN] only 4312 MB free where apt stages .debs — a CUDA toolkit install needs ~4500 MB
         sudo apt-get clean     (safe: pure cache)

========================================
RESULT: PASS  —  28 passed, 1 warning(s)
========================================

• FM version must match the driver's exact version — not just the branch; a mismatch causes nvidia-fabricmanager.service to fail at start. The mirror only syncs a driver build when a fabricmanager of the identical version exists, and the installer puts both in the same apt transaction, so this cannot drift apart on its own.
• Kernel module: open by default, forced open on Blackwell — B100/B200/B300/GB200/GB300 have no proprietary module, so the installer detects them and installs -open with no fallback. A node already running the proprietary module keeps it unless you pass --open=true. Under Secure Boot the module is DKMS-signed with the machine's MOK key, and the installer verifies it with modinfo -F signer nvidia after the build instead of leaving you to find out at reboot.
• Both distros carry the same CUDA set — CUDA 13.x is available for jammy (22.04) too, not just noble; running 13.x on jammy needs a driver ≥ 580 on that node.
• DOCA-OFED with driver, one reboot — run --doca=true together with the driver flags so nvidia_peermem links at DKMS build time; reboot once at the end.
• --no-nv=true needs reboot — adds kernel blacklist, takes effect after reboot.
• DCGM takes 10–20s after boot before reporting GPU state.
• DCGM plugin follows the driver, not the toolkit — dcgmi diag loads the plugin for the driver's max CUDA (driver 580/590/595 → cuda13), regardless of which CUDA toolkit you install. Ticking --dcgm in the builder above locks the CUDA menu to that same major so the toolkit and DCGM plugin stay aligned. Editing the command by hand can break this — if you keep a lower toolkit with DCGM, make sure the matching datacenter-gpu-manager-4-cuda<driver-major> plugin is installed or dcgmi diag will fail to load plugins.
• Both distros carry the same driver builds — noble and jammy are synced from the same NVIDIA CUDA index in the same run, so the version lists are identical; only an interrupted run can leave one briefly behind.