An open API service indexing awesome lists of open source software.

cpu-performance-engineering

A reading path for CPU performance engineering, from one instruction to production inference. Primary sources only, with a runnable benchmark for every section.
https://github.com/usamahz/cpu-performance-engineering

Last synced: 29 days ago
JSON representation

10. NUMA and multi-socket

Migration, balancing and measured effects

NUMA and Linux memory placement

  • NUMA (Non-Uniform Memory Access): An Overview - The one account tying first touch, policy scope, zone reclaim and page movement together from the implementer's side.
  • What is NUMA? - Defines nodes, zonelists and the distance-ordered fallback that places an allocation once local memory runs out.
  • NUMA Memory Policy - The normative statement of policy scopes, every mode including weighted interleave, and the cpuset intersection rule.
  • Numa policy hit/miss statistics - Defines numa_hit, numa_miss and numa_foreign, the counters that show whether a policy put pages where it said.
  • numactl - Reference implementation of the policy API, prints the distance table and binds a binary that cannot be rebuilt.

Topology and interconnects

11. OS and I/O

Cache and bandwidth partitioning

Interrupts and kernel bypass

  • NAPI - Defines the polling, software coalescing, busy polling and IRQ suspension knobs that trade interrupts against latency.
  • DPDK Programmer's Guide - Defines the full bypass model, pinned poll-mode cores with no interrupts, that every kernel path is measured against.
  • The eXpress Data Path - Measures an in-kernel programmable path against DPDK and the stack per core, with the full configuration published.
  • Kernel vs. User-Level Networking: Don't Throw Out the Stack with the Interrupts - Separates direct and indirect NIC interrupt cost, measures the stack against bypass, and is where IRQ suspension began.
  • AF_XDP - Defines the socket and UMEM rings handing XDP frames to user space, and the zero-copy and need-wakeup modes.

Scheduling, affinity and isolation

  • EEVDF Scheduler - Defines lag and virtual deadline, which the default class schedules by, and the slice request in sched_setattr.
  • The Linux Scheduler: a Decade of Wasted Cores - Proves cores sit idle while runnable threads queue, and gives the invariant checker that found the load-balancer bugs.
  • Control Group v2 - Defines cpu.max throttling, cpu.weight and the cpusets that bound affinity, the controls behind every container limit.
  • CPU Performance Scaling - Defines the governors, driver and boost switch that set a core's frequency, the sysfs state a measurement records.
  • CPU Isolation - Ties isolcpus, nohz_full, IRQ affinity, RCU offload and cpusets into one recipe, and lists the jitter it leaves.

Syscalls and asynchronous I/O

12. Tail latency and production systems

Load generation and production workloads

Measuring the tail

  • The Tail at Scale - Shows why fan-out makes a rare slow server a common slow request, and names the techniques that tolerate variance.
  • Attack of the Killer Microseconds - Defines the stall band that out-of-order hardware cannot hide and a context switch cannot amortise.
  • How NOT to Measure Latency - Shows that a summary without a max discards the samples that define the tail, and closed-loop load never records them.
  • Coordinated Omission - The original definition of the recording error, with arithmetic for how far a reported percentile sits from the truth.
  • HdrHistogram - Keeps the whole distribution at fixed relative precision in constant time, so the far percentiles and max survive.

Mechanical sympathy

  • Inter Thread Latency - Measures with code the floor for handing a cache line between cores, which every queue and lock is built on.
  • Single Writer Principle - States the design rule that removes write contention outright, using a contended increment's cost as the argument.
  • Optimizing a Ring Buffer for Throughput - Adds cached indices to a single-producer single-consumer ring and shows with counters the coherence traffic removed.
  • LMAX Disruptor - Applies the single writer rule and cache-line padding to a ring buffer, with the queue comparison that motivated it.
  • Aeron - Carries the single writer and batching rules through a whole transport, the reference beyond one in-process queue.

Where jitter comes from

  • rt-tests - The reference wakeup-latency measurement for Linux, whose README states that an unloaded run proves nothing.
  • osnoise tracer - Counts the noise a spinning thread suffers and attributes each event to NMI, IRQ, softirq, thread or hardware.
  • Tales of the Tail - Derives the queueing-ideal tail and attributes the excess to scheduling, interrupt placement, power saving and NUMA.
  • Latency Implications of Virtual Memory - Measures with code the page-fault, TLB-shootdown and writeback stalls that memory mapping hides from the caller.
  • The KVM halt polling system - Defines the host-side polling after a vCPU halt that trades idle host CPU for guest wakeup time, unseen by the guest.

13. Inference on CPU

GEMM and BLAS

Matrix extensions

Quantization

Runtimes

  • oneDNN - Generates VNNI and AMX kernels at run time under PyTorch, TensorFlow and OpenVINO, and calls Compute Library on Arm.
  • ggml - Defines the quantized block formats and the per-ISA dot-product kernels over them that every llama.cpp type rests on.
  • llama.cpp - Where new quantization types, kernels and thread pools land first, each with the perplexity and speed table behind it.
  • ONNX Runtime MLAS - Holds the CPU provider's GEMM, int8 and int4 MatMul kernels, dispatched per ISA at run time from SSE to AMX and SME.
  • OpenVINO CPU Device - States precision defaults per ISA, the int8 path through oneDNN and the streams model that turns cores into throughput.

Threading for inference

  • Thread management - Sets the physical-core default, the affinity it implies and the spin-wait controls that trade idle CPU for latency.
  • Performance Hints and Thread Scheduling - States the vendor defaults, one thread per core, SMT siblings off, core type by precision and one socket for latency.
  • Threadpool: take 2 - Defines the explicit thread pool with CPU masks, strict placement, priority and polling that ggml runs without OpenMP.
  • llama-bench - Defines the prompt and generation tests, repetitions and mean with deviation behind any comparable llama.cpp number.
  • Dual Epyc Genoa/Turin token generation performance bottlenecks - Traces poor decode scaling across sockets to remote NUMA access from weight placement, with numatop counts as evidence.

When CPU beats GPU

14. Hardware generations

AMD EPYC

Arm Neoverse server parts

Independent measurement across vendors

Intel Xeon

15. Benchmarks

Methodology and what suites miss

Microbenchmark suites

Standard suites

16. Watchlist

ISA extensions without a shipped server part

Kernel paths and generated code

  • Extensible Scheduler Class - Lets a BPF program schedule at run time with safe fallback, once a run against the default scheduler states every field.
  • io_uring zero copy Rx - Lands payloads straight in user memory on header-splitting NICs, pending the implementer's epoll run naming every field.
  • T-MAC - Table-lookup kernels for low-bit weights, with a baseline stated but no frequency, compiler or flags, pending those.
  • Faster sorting algorithms discovered using deep reinforcement learning - Generated small sorts shipped in libc++, timed by CPU family with no model, compiler or flags stated, pending those.

Memory and interconnect

Parts without a public measurement

  • 6th Gen AMD EPYC Server CPUs - The Zen 6 server family, so far a press release with no shipped part, pending shipment and a public run against Zen 5.
  • Intel Xeon 6+ Processors - The E-core-only sockets after Sierra Forest, shipped with vendor multiples footnoted off the page, pending a public run.
  • NVIDIA Vera CPU - Custom Arm cores with statically partitioned SMT and no architecture document, pending a specification and a public run.
  • Arm Neoverse V3 Core Software Optimization Guide - Vendor timing tables for the core shipped in Graviton 5 and previewed in Cobalt 200, pending a public run on the core.

1. Start here

2. One instruction, end to end

Execute

Fetch and decode

Memory access and retire

Rename and issue

3. Microarchitecture

Branch prediction and speculation

Limits of ILP and SMT

Vendor estimates and measured tables

  • uops.info - Where each latency, throughput and port entry links to its microbenchmark, so any value can be re-run.
  • applecpu: Firestorm Overview - Measured per-instruction tables for an Apple AArch64 core, each entry linked to the counter experiment behind it.

What the manuals leave out

4. Memory hierarchy

Cache geometry, replacement and misses in flight

Store buffers, ordering and cache-line contention

Struct layout, software prefetch and page size

TLBs, page walks and prefetchers

5. Measurement

Counters, events and precise sampling

CPU profilers and flame graphs