Cheat SheetsMath for AIMath Behind Neural Nets & LLMs

Math Behind Neural Nets & LLMs — Cheat Sheet

Math for AI · 2 topics. Download the PDF or the Instagram carousel and share it.

Cheat Sheet · AiCanCode.org
Math Behind Neural Nets & LLMs
Math for AI2 topicsQuick revision reference
1

The Mathematics Behind Neural Networks

A neural network is a stack of linear transformations and non-linear activations, trained by gradient descent using backpropagation — this concept assembles every earlier topic into one working picture.

  • ✓A network alternates linear transforms (XW+b) with non-linear activations
  • ✓Forward pass produces a prediction; a loss scores it (cross-entropy/MSE)
  • ✓Backpropagation = chain rule computing every weight's gradient in one backward sweep
  • ✓An optimizer (SGD/Adam) steps weights downhill; repeat over mini-batches to learn
Forward pass: relu(XW₁+b₁) → softmax(HW₂+b₂)
import numpy as np

def relu(x):    return np.maximum(0, x)
def softmax(x):
    e = np.exp(x - x.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)

X  = np.random.randn(4, 3)            # 4 examples, 3 features
W1, b1 = np.random.randn(3, 5), np.zeros(5)
W2, b2 = np.random.randn(5, 2), np.zeros(2)

H     = relu(X @ W1 + b1)             # hidden layer (linear + non-linear)
probs = softmax(H @ W2 + b2)          # output as a probability distribution
print(probs.shape, np.round(probs.sum(axis=1), 3))   # (4,2) rows sum to 1
2

The Mathematics Behind Transformers & LLMs

A transformer turns tokens into vectors and lets them exchange information through attention — a few matrix multiplies, a softmax, and a dot product — which is the mathematical core of every modern LLM.

  • ✓Tokens become embedding vectors; Q/K/V are three learned matrix projections
  • ✓Attention scores = scaled dot products (relevance), turned into weights by softmax
  • ✓Each token output is a weighted sum of value vectors — information mixing across tokens
  • ✓LLMs train by minimising cross-entropy on next-token prediction via gradient descent
Attention = softmax(QKᵀ/√d) · V
import numpy as np

def softmax(x):
    e = np.exp(x - x.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

seq, d = 3, 4                       # 3 tokens, dim 4
X = np.random.randn(seq, d)        # token embeddings
Wq, Wk, Wv = (np.random.randn(d, d) for _ in range(3))

Q, K, V = X @ Wq, X @ Wk, X @ Wv   # three projections
scores  = (Q @ K.T) / np.sqrt(d)   # dot-product relevance, scaled
weights = softmax(scores)          # rows sum to 1 -> a distribution
out     = weights @ V              # each token = weighted sum of values
print(weights.shape, out.shape)    # (3,3) attention map, (3,4) new vectors
Learn this free with Aria, your AI tutor → AiCanCode.org/learn/math-for-ai