The terms that keep showing up in the cards, and what changes once you know them.
24 terms
Stored attention keys and values for tokens already processed, so generating each new token does not recompute the whole prompt.
This is why long chats slow down and GPU memory runs out: the cache grows linearly with context length and with every concurrent user.
See also PagedAttention · Context window · VRAM · Prefill Phase
Converting model weights to fewer bits per value so the model needs less memory and bandwidth to run.
FP8, Q4 and GGUF tags on a model name all mean this; the model gets smaller and slightly worse, and how much worse is the whole argument.
See also FP8 · Q4 / 4-bit quantization · GGUF · Quantization-Aware Training
An 8-bit floating point format that halves memory versus 16-bit while keeping enough range for most inference workloads.
An FP8 tag means the same model runs in half the memory, but native FP8 math needs a Hopper, Ada or newer GPU, so an A100 gets less out of it.
See also Quantization · BF16 / FP16 · VRAM · Quantization-Aware TrainingShowed up in Models
A single-file model format used by llama.cpp, packing quantized weights and metadata so a model runs without a Python stack.
A GGUF tag on a model page means it was uploaded to run on your own laptop, not in a datacenter; Ollama and LM Studio both load this format.
See also llama.cpp · Quantization · Q4 / 4-bit quantization · OllamaShowed up in Models
An attention implementation that tiles the computation in fast on-chip GPU memory, cutting slow memory round trips.
It made long contexts practical by cutting memory traffic without changing the math, and is now on by default in most engines.
See also Memory bandwidth · Context window · KV Cache · CUDA
Apple's machine learning framework for Apple silicon, plus the converted weight format models are published in so they run on it.
An MLX tag means the upload targets Macs; unified memory means the VRAM ceiling works differently there than on a discrete GPU.
See also GGUF · VRAM · On-Device Inference · Quantization
Running a model behind an API or server so many concurrent requests are scheduled, handled, and billed.
Running a model once and serving thousands of users are different problems; titles with serving are about concurrency and unit cost, not model quality.
See also Continuous Batching · Throughput · vLLM · Token Pricing
A quantization level storing each weight in about four bits, roughly a quarter the size of the original 16-bit model.
Q4 is usually the line between a 27B model fitting a 24GB GPU or not; suffixes like Q4_0 and Q4_K_M are different recipes at the same bit width.
See also Quantization · GGUF · VRAM · Quantization-Aware Training
The 16-bit floating point formats most released weights ship in, and the baseline that quantized versions get compared against.
This is the yardstick for model size: a 27B model in BF16 is about 54GB, FP8 halves it and Q4 quarters it.
See also FP8 · Q4 / 4-bit quantization · VRAM · Quantization
Managing KV cache in small fixed blocks like OS paging, so GPU memory is not wasted on reserved-but-unused space per request.
This is what made vLLM famous; claims about serving several times more concurrent requests on the same GPU usually trace back to it.
See also KV Cache · vLLM · Continuous Batching · VRAM
Grouping requests into one GPU pass and immediately slotting new requests into finished slots instead of waiting for the whole batch.
A GPU pass costs about the same for one user or thirty, which is why serving cost per user falls as traffic rises.
See also Serving · Throughput · PagedAttention · vLLM
A small draft model guesses several next tokens and the big model verifies them in one pass, accepting the correct prefix.
A rare optimization that speeds up output without changing it, so it is the usual answer to why the same model suddenly got faster.
See also Throughput · Prefill Phase · Serving
The opening phase of a request, where the whole prompt is computed at once to fill the KV cache before any token appears.
It is compute-bound, so a longer prompt directly delays the first character, and it is exactly the stretch that prompt caching bills away.
See also Time to First Token · Decode Phase · KV Cache · Prompt Caching
The delay between sending a request and the first token appearing, separate from how fast the rest streams out.
Perceived slowness lives here: a great tokens-per-second number still feels slow if the first token takes two seconds.
See also Prefill Phase · Throughput · Serving
How many tokens per second a model produces, or how many requests per second a server sustains under load.
Per-user and whole-server numbers are both called throughput, so the first question about any benchmark is which one it measured.
See also Continuous Batching · Time to First Token · Memory bandwidth · Token Pricing
Keeping layers that do not fit in GPU memory on CPU RAM or disk and pulling them in when needed.
It makes an oversized model run at all, several times slower, which is why the same setup gets both it works and it is unusable reports.
See also VRAM · Memory bandwidth · llama.cpp · Quantization
An open-source serving engine built to pack many concurrent requests onto the same GPU efficiently.
It is the default in self-hosted serving, so papers and company posts usually use it as the throughput baseline they compare against.
See also Serving · PagedAttention · Continuous Batching · Throughput
A C++ inference engine that runs quantized models on ordinary laptops and desktops, on CPU or a single GPU.
Almost every run-it-yourself story sits on top of this: GGUF, Ollama and LM Studio are all layers above it.
See also GGUF · Ollama · Offloading · Q4 / 4-bit quantization
A local runner that downloads a model and exposes it behind a local API with a single command.
Most I ran it locally posts mean this tool; the real work happens in the llama.cpp-derived engine underneath, so performance questions belong there.
See also llama.cpp · GGUF · On-Device Inference · Q4 / 4-bit quantization
Reusing the computed state of a prompt prefix that repeats across requests, so repeated input is billed and processed cheaply.
For agents that resend the same system prompt and documents every turn, the cache hit rate is the bill: a miss can cost several times more.
See also KV Cache · Token Pricing · Context window
API billing measured per million input and output tokens, usually priced separately with output several times more expensive.
Output costs several times more than input, so models that think longer bill more for the same question, and benchmarks now quote dollars next to scores.
See also Prompt Caching · Context window · Serving · Throughput
Running a model on the user's own laptop or phone instead of sending the request to a server.
No per-token bill and no data leaving the device, in exchange for that device's memory and bandwidth, which is why small MoE and 4-bit models keep shipping.
See also MLX · Ollama · Q4 / 4-bit quantization · Quantization · VRAM
Knobs for how the next token is drawn: temperature flattens the probability distribution, top-p and top-k trim the candidate list.
Release notes that specify 'temperature 0.6, top_p 0.95' mean it: the same weights score differently when these drift, so published numbers assume them.
See also Autoregressive generation · Decode Phase · Stop sequence · Reproducibility
The one-token-at-a-time stretch after prefill, where every step re-reads the full weight set and the cache out of memory.
This stretch is bound by memory bandwidth, not compute, which is why raw FLOPS figures mislead and why HBM specs decide serving speed.
See also Prefill Phase · KV Cache · Memory bandwidth · Batch size