Probability & Statistics — Cheat Sheet

Mean, variance, and standard deviation summarise a dataset's center and spread — the first thing you compute about any data and the basis of normalization that makes models train well.

• ✓Mean = center of mass; median = middle value (robust to outliers)
• ✓Variance = average squared distance from the mean; std = its square root (same units)
• ✓Standardization (z-score) rescales features to mean 0, std 1
• ✓Normalized features help gradient descent converge and prevent scale dominance

import numpy as np

data = np.array([10, 12, 11, 13, 12, 100])   # one outlier (100)

print("mean  ", np.mean(data))     # 26.33 -> pulled up by the outlier
print("median", np.median(data))   # 12.0  -> robust to it
print("var   ", round(np.var(data), 1))
print("std   ", round(np.std(data), 1))

Probability quantifies uncertainty from 0 to 1; conditional probability P(A|B) updates that belief once you know something — the mechanism behind every model that predicts under uncertainty.

• ✓A probability lies in [0,1]; exclusive outcomes sum to 1
• ✓Conditional P(A|B) = P(A and B) / P(B): belief updated by evidence
• ✓Prediction is conditional probability: P(label | features)
• ✓Independence lets joints factor into products; the chain rule factors them into conditionals (LLMs)

import numpy as np

# columns: [is_spam, has_word_free]
data = np.array([[1,1],[1,1],[1,0],[0,0],[0,1],[0,0],[0,0],[1,1]])
spam, free = data[:,0], data[:,1]

p_free = free.mean()
p_spam_and_free = ((spam==1) & (free==1)).mean()
p_spam_given_free = p_spam_and_free / p_free        # P(spam | free)
print(round(p_spam_given_free, 3))   # higher than base spam rate

Bayes’ theorem flips a conditional probability, letting you update a prior belief with new evidence to get a posterior — the formal engine of learning from data.

• ✓Bayes: posterior ∝ likelihood × prior — update belief with evidence
• ✓P(A|B) = P(B|A)·P(A) / P(B) flips the conditional you can measure into the one you want
• ✓Base rates (priors) can dominate — rare events make "accurate" tests misleading
• ✓Naive Bayes = Bayes + conditional independence; a strong, fast baseline

# Disease prevalence 1%; test 99% sensitive, 99% specific.
p_disease = 0.01
p_pos_given_disease = 0.99
p_pos_given_healthy = 0.01

p_pos = (p_pos_given_disease * p_disease +
         p_pos_given_healthy * (1 - p_disease))
p_disease_given_pos = p_pos_given_disease * p_disease / p_pos
print(round(p_disease_given_pos, 3))   # ~0.5 -> only 50% despite "99% accurate"

A distribution describes how likely each value is; the normal (bell curve) is the default model of natural variation and shows up in weight initialization, noise, and the Central Limit Theorem.

• ✓A distribution lists every possible value and its probability
• ✓Softmax outputs a categorical distribution over classes
• ✓The normal is set by mean & std; 68/95/99.7% within 1/2/3 std
• ✓Central Limit Theorem: sums/averages of many effects → normal; models noise & weight init

import numpy as np
rng = np.random.default_rng(0)

samples = rng.normal(loc=0.0, scale=1.0, size=10000)   # mean 0, std 1
print(round(samples.mean(), 3), round(samples.std(), 3))  # ~0 ~1

# ~68% of a normal lies within 1 std, ~95% within 2 std:
within1 = np.mean(np.abs(samples) < 1)
print(round(within1, 3))    # ~0.68

We almost never see all the data, so we estimate from samples — and confidence intervals tell us how much to trust those estimates, which is how you compare models honestly.

• ✓We estimate population quantities from finite samples — every estimate has uncertainty
• ✓Standard error of the mean = std/√n; uncertainty shrinks slowly with data
• ✓A confidence interval reports a range, not a single fragile number
• ✓Overlapping CIs mean a metric difference may be noise — essential for honest model comparison

import numpy as np
rng = np.random.default_rng(0)

pop_std = 1.0
for n in [25, 100, 400]:
    se = pop_std / np.sqrt(n)          # standard error of the mean
    print(f"n={n:>3}  standard error={se:.3f}")
# n= 25  0.200
# n=100  0.100   -> 4x data -> 2x tighter
# n=400  0.050