PyTorch

PyTorch is an open-source deep learning framework developed by Meta AI (originally Facebook AI Research). It is the dominant library for neural network research and increasingly for production deployment. PyTorch implements automatic-differentiation over tensors (multi-dimensional arrays), enabling efficient backpropagation across GPU hardware.

---

Key Features

Feature	Description
Dynamic computation graph	Graph is built at runtime (eager mode); flexible Python control flow
Tensor operations	N-dimensional array operations executed on GPU via CUDA
Autograd engine	torch.Tensor.backward() triggers backpropagation
nn.Module	Base class for all neural-network components (layers, models)
Optimisers	SGD, Adam, etc. implement gradient-descent variants

---

Relationship to Micrograd

micrograd is a pedagogical scalar version of what PyTorch does at tensor scale. The mathematics is identical:

• Value in micrograd → torch.Tensor in PyTorch
• _backward closure → PyTorch’s autograd function
• Manual .grad accumulation → tensor.grad attribute

Karpathy explicitly positions micrograd as “the engine at the heart of PyTorch or JAX” — but at scalar granularity for clarity.

---

Usage in Knowledge Base

PyTorch is the implementation framework for:

• raschka-2024-build-llm-from-scratch — GPT implementation from scratch
• The production equivalent of micrograd (per karpathy-2022-micrograd-backpropagation)

---

Sources

• karpathy-2022-micrograd-backpropagation — cited as the production counterpart of micrograd
• raschka-2024-build-llm-from-scratch — used throughout for GPT implementation