GPU & AI glossary

Plain-language definitions for the terms used throughout this site. Every term elsewhere on the site links back here — click any dotted-underlined label to jump to its definition.

Software & platforms

CUDA / CUDA Toolkit

NVIDIA's parallel computing platform and API for running general-purpose code on NVIDIA GPUs. The CUDA Toolkit — the compiler, runtime, and libraries developers install — must be paired with a compatible GPU driver, which is versioned separately.

ROCm

AMD's open-source software stack for running compute workloads on AMD GPUs — ROCm's equivalent of CUDA.

cuDNN

NVIDIA's library of optimized deep-learning primitives (convolutions, attention, and similar operations) built on top of CUDA. PyTorch and TensorFlow both depend on a specific cuDNN version.

GPU Driver

The low-level software the operating system uses to talk to the GPU. CUDA and ROCm each require a minimum driver version to run — the driver and the toolkit are versioned and installed separately, and mismatches are one of the most common sources of "it doesn't work" reports.

Compute Capability (sm_XX)

NVIDIA's version number for a GPU architecture's feature set and instruction set (e.g. sm_90 = Hopper). CUDA software support is generally granted per architecture generation, so this number is what actually determines what a given CUDA Toolkit release can run on.

GPU Target (gfx)

AMD's precise per-chip identifier (e.g. gfx942) — ROCm's equivalent of compute capability. Unlike NVIDIA, AMD grants ROCm support per exact chip rather than uniformly per architecture generation, so this identifier matters even more for ROCm than compute capability does for CUDA.

WSL2 & Docker (running CUDA without native Linux)

Two common ways to run CUDA workloads outside a native Linux install. WSL2 (Windows Subsystem for Linux) exposes the GPU into a Linux environment on Windows using only the Windows driver — installing a separate Linux driver inside WSL2 breaks it, and WSL2 GPU support itself needs a newer driver (R495 or later) than some older CUDA releases' own stated minimum. Docker containers work differently: only the host's GPU driver matters, since the CUDA Toolkit itself lives inside the container image, not on the host.

Compute & precision

TFLOPS / TOPS

Throughput: trillions of floating-point operations per second (TFLOPS) or trillions of integer operations per second (TOPS). Higher is faster, but figures are only comparable at the same numeric precision — a card's FP8 TFLOPS figure isn't comparable to another card's FP16 TFLOPS figure.

FP64 / FP32 / FP16 / BF16 / FP8 / FP6 / FP4

Floating-point number formats of decreasing precision (and size). Lower precision uses less VRAM and computes faster, at some cost to numerical accuracy — modern AI training leans on FP16/BF16, and inference increasingly runs at FP8 or lower. FP64/FP32 remain the norm for traditional scientific/HPC workloads.

See also: VRAM Calculator

INT8 / INT4 (quantization)

Integer number formats commonly used for quantized models. INT4 packs each parameter into half a byte — roughly a quarter of FP16's memory footprint — trading some accuracy for a much smaller model.

See also: VRAM Calculator

Quantization

Compressing a trained model's weights from a high-precision format (like FP16) down to a lower-precision one (like INT8 or INT4) to cut VRAM use and speed up inference, at some cost to output quality.

See also: VRAM Calculator

Tensor Cores / Matrix Cores

Dedicated GPU hardware for matrix-multiply-accumulate — the core operation neural networks are built from — distinct from a GPU's general-purpose shader/CUDA cores. NVIDIA calls them Tensor Cores, AMD calls them Matrix Cores; both do the same job.

Compute Units (CUs) / Streaming Multiprocessors (SMs)

A GPU's top-level processing blocks — the unit the hardware actually schedules work onto. Each one bundles shader/CUDA cores, its own register file, scheduler, and shared memory/cache, plus matrix cores on architectures that have them. AMD calls them Compute Units, NVIDIA calls them Streaming Multiprocessors, Intel calls them Xe-cores. Vendors bin the same die at different counts (fusing some off for lower SKUs), so within one architecture the CU/SM count scales almost linearly with throughput — but counts are not comparable across vendors or generations, since what fits inside one block differs.

Memory & hardware

VRAM (Video RAM)

The dedicated memory built onto a GPU, separate from a computer's regular system RAM. A model's weights and its working data (activations, KV cache) must fit in VRAM to run at all — if a model needs more VRAM than a GPU has, it simply won't load, regardless of how fast that GPU is.

See also: VRAM Calculator

Memory Bandwidth (GB/s)

How fast data moves between VRAM and the GPU's compute units. For LLM inference this is very often the real bottleneck — not raw TFLOPS — since generating each token means reading the entire model's weights from VRAM again.

TDP (Thermal Design Power)

The power, in watts, a GPU is designed to dissipate — a practical proxy for its power draw under sustained load, and what a power supply and cooling system need to be sized around.

ECC Memory

Memory that detects and corrects random bit errors. Standard on datacenter and workstation GPUs, often absent or optional on consumer cards — matters most for long-running training jobs where a silent bit-flip could corrupt results.

PCIe Generation

The physical slot and interface generation connecting a GPU to the rest of the system (PCIe 3.0 / 4.0 / 5.0...). Each generation roughly doubles per-lane bandwidth over the last one.

Interconnect (NVLink / Infinity Fabric)

High-bandwidth GPU-to-GPU links that bypass the PCIe bus entirely, used in multi-GPU systems to let GPUs share data far faster than PCIe allows — NVIDIA's NVLink and AMD's Infinity Fabric are the two implementations tracked on this site.

Multi-Instance GPU (MIG)

NVIDIA's feature for partitioning one physical GPU into several smaller, fully isolated GPU instances, each with its own dedicated slice of compute and memory — useful for running several smaller workloads on one large GPU.

Running LLMs

KV Cache

The memory an LLM accumulates while generating text: the attention "key" and "value" tensors for every token already processed, kept around so the model doesn't recompute them for each new token. It grows with context length and is one of the two big VRAM costs alongside the model's weights.

See also: VRAM Calculator

Context Length / Context Window

The maximum number of tokens — input plus output combined — a model can attend to at once. A longer context window needs proportionally more KV-cache VRAM to actually use.

See also: VRAM Calculator