Memory is Slow, Disk is Fast - Part 1 TL;DR Hardware improvements focus on width (more cores, bandwidth, vector units) rather than speed (clocks, IPC, latency). Traditional truths like "memory is faster than disk" are breaking. To achieve speed now, one has to adapt to the new performance landscape. --- CPUs Keep Getting Faster? Not Exactly Computers are more capable and some specs improve exponentially. However, the assumption "CPUs keep getting faster every generation" is misleading. Improvements target parallelism, bandwidth, and vectorization—not raw single-thread speed. --- What is Scaling? What Hardware Metrics Are Improving? Transistor Count: Continues to increase, packaging advances help. Core Counts: Easily scaled due to lithography progress. Vector Operations: Rapid innovation aiding graphics, physics, AI; compilers partly keep pace. Memory Bandwidth: Steady increases via cooperative standards essential to avoid stalls. Caches: L1/L2 caches grow slowly; L3+ caches steadily increase to feed processor demands. PCIe Bandwidth: Doubles per generation, enabling fast NVMe storage and networking. Storage and Network: NVMe SSDs and network interfaces have kept pace with PCIe improvements. --- What Is NOT Scaling? Clock Speed: Plateaued around 20 years ago; stopped rapid increases. Instructions Per Clock (IPC): Improved slightly but slow growth overall. Instructions Per Second (IPS): Flatlined for non-vector workloads, showing stagnation in raw core compute power. Memory Latency: DRAM latency unchanged for 30 years due to physical limits. Disk Latency: SSDs cut latency 100x from HDDs but then stagnated around the same level. Latency costs in terms of lost instructions have exploded: Cache misses now cost hundreds of instructions, up from dozens in the 90s. Disk I/O latency still high relative to CPU, though SSDs have improved bandwidth exponentially. --- The Future Outlook Good News Scaling in cores, bandwidth, and vector ops continues. AI workloads thrive on these improvements. No immediate technical roadblocks for continued growth in these areas. Bad News Clock speeds and latency improvements hit physical limits (quantum, material science). Stagnation for traditional single-threaded, non-vectorized software. --- The Problem with Traditional Software Most traditional applications (e.g., TypeScript apps, system software) are single-threaded and non-vectorized. These can't leverage modern hardware scaling effectively. Performance gains that used to be automatic now require new software paradigms. Old dogmas like "memory is always faster than disk" no longer hold universally true. --- What Does This Mean? Longstanding computer science dogma needs revisiting. Performance cost differences between operations (memory vs. disk, cache misses) have grown orders of magnitude. Rethinking software design is essential to harness modern hardware effectively. --- Coming Next: Part 2 Preview Exploring when disk access can be faster than memory. Questioning Big O notation relevance in the new hardware era. Will AI dominate everything? Implications are complex and profound. --- Data for charts was compiled with ChatGPT’s assistance and validated for accuracy. — Jared Hulbert