What is Tensor Cores?
Specialized hardware units on NVIDIA GPUs designed for matrix multiplication — the core math operation in neural networks. 5th-gen Tensor Cores (Blackwell) are significantly faster than 4th-gen (Ada Lovelace) for AI inference.
Full Explanation
Tensor Cores are dedicated matrix-multiplication accelerators built into NVIDIA GPU die since the Volta architecture (2017). Each generation doubles or triples throughput: 5th-generation Tensor Cores in Blackwell (RTX 5070) support FP4 and FP8 precision natively, which AI inference frameworks can exploit for 2–4× the throughput of FP16. For LLM inference specifically, the memory bandwidth ceiling usually limits real-world throughput before Tensor Core compute does — but for batch inference (processing many prompts simultaneously), Tensor Core speed becomes the primary constraint.
Why It Matters for Local AI
For single-user interactive chat, Tensor Core generation matters less than raw memory bandwidth. For deploying a shared local AI server serving multiple simultaneous users, 5th-gen Tensor Cores in Blackwell cards provide a meaningful throughput advantage.
Hardware Relevant to Tensor Cores
gpu · Check Price on Amazon · 12 GB VRAM · 672 GB/s
gpu · Check Price on Amazon · 12 GB VRAM · 672 GB/s
Related Terms
CUDA→
NVIDIA's proprietary parallel computing platform. Industry standard for AI/ML. Nearly every AI framework (PyTorch, Ollama, ComfyUI) supports CUDA natively and first.
VRAM→
Video RAM — dedicated memory on a GPU. Determines the maximum model size you can run with full GPU acceleration. Once a model exceeds VRAM, it spills to system RAM over the slow PCIe bus.
Memory Bandwidth→
How fast data moves between memory and the processor, measured in GB/s. Tokens per second scales nearly linearly with bandwidth — this is the single most important GPU spec for LLM speed.
Blackwell→
NVIDIA's 2024–2025 GPU architecture generation. Features 5th-generation Tensor Cores, GDDR7 memory, and significant AI inference performance improvements over Ada Lovelace (RTX 40 series).