The terms that keep showing up in the cards, and what changes once you know them.
25 terms
Nvidia's programming layer for running computation on its chips; virtually every deep learning library sits on top of it.
The industry cannot leave Nvidia because of this layer, not the silicon; a post bragging about a hand-written kernel is bragging about work done here.
The amount of memory on the accelerator itself; weights and activations must all fit inside it before a model will run.
A claim that a model runs on 24GB is a claim about this number, and doing the arithmetic first saves a lot of wasted downloads.
See also High Bandwidth Memory (HBM) · Quantization · GGUF · KV Cache
How much data per second moves between memory and the compute units; for token-by-token generation this, not math speed, is the ceiling.
Generation is usually starved for data movement rather than math, and that single fact is why quantization and batching work at all.
See also High Bandwidth Memory (HBM) · KV Cache · Continuous Batching · Quantization
A chip that runs one operation across thousands of cores at once; deep learning training and inference happen essentially nowhere else.
Most numbers on a model card translate into how many of these you must rent or own; parameter counts and quantization are that arithmetic.
Google's in-house accelerator for deep learning, rented through its cloud rather than bought, and the main alternative to Nvidia hardware.
The name that always shows up as the alternative in Nvidia-monopoly stories; the real price of using it is porting your stack off CUDA.
See also GPU · CUDA · GPU cluster
A raw count of floating-point multiply-adds, the shared unit for how much computation a training run or a single forward pass consumed.
A paper reporting 1e25 FLOPs is reporting a budget, and regulators draw their thresholds in the same unit, so the number turns up in law too.
See also GPU-hour · Scaling Law · GPU
One accelerator running for one hour, the standard unit for pricing a training or fine-tuning run on the rental market.
An announced training cost is almost always this unit times an hourly rate, so knowing the going rate lets you check the claim in seconds.
See also FLOPs · GPU cluster · Cost per token
Thousands to hundreds of thousands of accelerators wired by high-speed links so a single training job can span all of them.
Counting cards is the wrong metric: if the links between them are slow half the fleet idles, which is why vendors brag about interconnect speed.
See also GPU · GPU-hour · Datacenter power
A memory standard that stacks DRAM layers right beside the compute die to reach bandwidth a normal module cannot.
Chip shortage stories are usually memory shortage stories; the queue forms at this part and the packaging that attaches it, not at the logic fab.
See also VRAM · Memory bandwidth · GPU
An API mode that queues non-urgent requests and returns them within a day, typically at half the normal per-token price.
Evals, labelling, and bulk summarization do not need to be real time, so not using this mode means paying twice for the same work.
See also Cost per token · Prompt Caching · Throughput
The unit that prices model usage: input and output are billed by the piece, normally quoted per million, with output costing more.
Input and output are priced several times apart, so an app that stuffs documents in and one that writes long answers have different economics.
See also Inference · Batch API · Prompt Caching · Throughput
Building one enormous processor from an entire silicon wafer instead of dicing it into many separate chips.
The pitch is to delete the interconnect problem outright, so when this word shows up in an Nvidia-alternative story, that is the claim being made.
See also GPU cluster · GPU · TPU
The electricity a training or serving site needs, contracted in megawatts and increasingly the factor that decides where it is built.
The binding constraint has shifted from chips to electricity, which is why utilities and national governments now appear in AI infrastructure stories.
See also GPU cluster · Scaling Law · GPU-hour
The dedicated links wiring accelerators and servers together, carrying the results every card must exchange at each training step.
Nvidia sells this harder than the chips: when it is the bottleneck, adding cards stops shortening training, so read cluster claims for topology, not card count.
See also GPU cluster · Tensor parallelism · Rack-scale · High Bandwidth Memory (HBM)
Splitting each weight matrix across several accelerators so a single layer is computed by all of them at once.
It is the usual way to run a model too big for one card; the number sits in your vLLM launch flags, and a wrong value leaves cards idle or blows past memory.
See also Interconnect · VRAM · vLLM · GPU cluster
Packing dozens of accelerators into one rack wired as a single unit, so the rack rather than the card becomes what you buy and deploy.
Recent announcements count racks, not cards, and the power and cooling arrive as a rack-sized requirement most existing datacenters cannot simply absorb.
See also Interconnect · Liquid cooling · Datacenter power · GPU cluster
Pulling heat off the chips with liquid instead of air, which stops being optional once a single rack draws tens of kilowatts.
When a new chip generation is slow to land, the blocker is usually the building, not supply: retrofitting plumbing is a construction project, not an order.
See also Datacenter power · Rack-scale · GPU cluster
The delay when a request hits an idle server and the weights must first be loaded into accelerator memory before anything runs.
It is why serverless GPU is cheap and also why it feels slow: a 100GB model can take tens of seconds to answer, so you weigh that against paying to keep it warm.
See also Time to First Token · VRAM · Spot instance · Serving
Renting spare capacity at a steep discount on the condition that the provider can reclaim the machine at a few minutes' notice.
It cuts training bills by more than half, but only if the job survives being killed, which is where all that advice about frequent checkpointing comes from.
See also GPU-hour · Checkpoint · Reserved capacity · Neocloud
Committing to accelerator capacity for months or years up front, buying both a discount and a place in the queue, used or not.
Those multi-billion-dollar cloud deals in the headlines are commitments to spend, not revenue already earned, and what is scarce is the slot, not the price.
See also GPU-hour · Spot instance · Neocloud · Total cost of ownership
A newer breed of provider that rents GPUs and little else, usually cheaper and quicker to allocate than the hyperscalers.
When an unfamiliar cloud name shows up in a training announcement it is usually one of these; the trade is raw machines you must operate yourself.
See also GPU-hour · Reserved capacity · GPU cluster · Total cost of ownership
The full bill for running compute (hardware, power, cooling, staff, depreciation), and the number that decides whether to buy or rent.
Owning looks cheap only when power and headcount are left off the sheet; at low utilization the cards you bought are the most expensive option you had.
See also GPU-hour · GPU utilization · Neocloud · Datacenter power
The ceiling an API puts on how many requests and tokens one account may spend per minute; go over and calls come back as 429s.
A new model is often unusable at launch because of this number rather than its quality, so when an eval run dies mid-way, check here before blaming the model.
See also Batch API · Cost per token · Throughput · Serving
Which country's servers your requests and stored data physically sit in, fixed by contract and law rather than by preference.
Enterprise adoption usually stalls here rather than on quality: the same model is simply unavailable to a company if there is no region it is allowed to use.
See also Zero Data Retention · PII · EU AI Act · Serving
The share of rented accelerator time that actually did math; the rest is the fleet waiting on data, storage, or the network.
You pay by the hour but get work only in proportion to this; MFU is the honest version of the number, and 40% counts as bragging.
See also GPU-hour · FLOPs · Total cost of ownership · Memory bandwidth