Amdahl’s Law
Amdahl’s Law (Gene Amdahl, 1967) describes the theoretical maximum speedup achievable by parallelising a computation:
Speedup = 1 / (S + (1 − S) / N)
Where S is the fraction of the task that is inherently sequential (cannot be parallelised) and N is the number of processors.
Key Implication
Even with infinite processors, speedup is capped at 1/S. If 10% of a task is sequential, maximum possible speedup is 10× — regardless of how many GPUs or chips are added. This is why rack-scale-computing and distributed AI infrastructure must minimise sequential bottlenecks (e.g., data loading, memory transfers) to realise the benefit of massive parallelism.
Relevance to AI
Jensen Huang (fridman-huang-2026-nvidia-ai-revolution) frames much of nvidia’s cuda stack as an engineering effort to push S toward zero — optimising memory bandwidth, interconnects (rack-scale-computing), and compiler pipelines so that GPU parallelism is not wasted on sequential overhead.