Mathematical Foundations for Machine Learning

Machine learning operates almost exclusively on multi-dimensional geometric spaces. Datasets are organized as matrices of feature vectors, model weights are represented as parameter tensors, and transformations correspond to rotations, reflections, and scalings in vector space.

1. Vectors & Vector Spaces

A vector Vector x in n-dimensional space (n real numbers) is an ordered sequence of n real numbers representing a single point or directional arrow in an n-dimensional Euclidean space:

2. Matrices & Matrix Operations

A matrix Matrix A with m rows and n columns is a rectangular 2D array consisting of m rows and n columns. Algebraically, an m × n matrix represents a linear transformation mapping vectors from n-dimensional real space into m-dimensional real space.

3. Matrix Multiplication Mechanics & Complexity

Matrix multiplication C = AB is defined if and only if the inner dimensions match: A in R^{m × k and B in R^{k × n yields product C in R^{m × n:

When a square matrix Square Matrix A of size n × n acts on a general vector x, the resulting vector Ax typically rotates and changes magnitude. However, there exist special non-zero vectors whose spatial direction is completely invariant under the transformation—they are merely scaled by a scalar factor.

Where:

• v in n-dimensional real space setminus {0 is the eigenvector.
• scalar eigenvalue λ (or C) is the corresponding eigenvalue.
• For non-trivial solutions (non-zero eigenvector v), the matrix (A - λ I) must be singular, establishing the characteristic equation:

If linear algebra provides the language to represent data and models, calculus provides the engine to train them. All supervised learning revolves around minimizing an objective loss function L(w) with respect to model parameters w.

1. Derivatives & Instantaneous Rate of Change

The single-variable derivative measures the sensitivity of function output f(x) with respect to infinitesimal shifts in input x:

2. Partial Derivatives & Multivariable Functions

In machine learning, loss functions depend on millions of parameter weights w = [w_1, w_2, dots, w_d]^T. The partial derivative with respect to w_i measures how the scalar loss changes as w_i varies while holding all other parameters strictly constant:

3. The Gradient Vector (∇ f)

The gradient of a scalar-valued multivariable function f: R^d to R is the column vector collecting all d first-order partial derivatives:

4. Gradient Descent Optimization Algorithm

Gradient Descent iteratively adjusts model parameters opposite to the gradient of the loss surface until reaching a minimum:

Machine learning algorithms operate under uncertainty caused by noisy measurements, unobserved variables, and stochastic environments. Probability theory provides the formal calculus to model and reason about uncertainty.

1. Probability Axioms & Conditional Probability

For a sample space Omega and event A subseteq Omega, Kolmogorov's axioms demand Prior Probability P(A) ≥ 0, P(Omega) = 1, and countable additivity for disjoint events. The conditional probability of event A given that event B has occurred (Evidence / Marginal Probability P(B) > 0) is:

2. Bayes’ Theorem

Bayes’ Theorem provides the mathematical bridge to invert conditional probabilities, converting prior beliefs into updated posterior probabilities upon observing empirical evidence:

3. Common Probability Distributions in ML

Statistical metrics allow data scientists to summarize distributions, identify anomalous outliers, and measure multi-variable relationships across features.

Covariance vs. Pearson Correlation Coefficient

When evaluating two features X and Y, we quantify their joint linear variability:

The following standalone script implements and validates the fundamental mathematical algorithms covered in this unit using NumPy and SciPy:

# =====================================================================
# DevAscent Academy: Mathematical Foundations Verification Engine
# Topics: Linear Algebra, Eigendecomposition, Gradient Descent,
#         Bayes' Theorem, Covariance & Pearson Correlation
# =====================================================================

import numpy as np

def run_linear_algebra_demo():
    print("=== 1. LINEAR ALGEBRA & EIGENVALUES ===")
    # Define a symmetric 2x2 matrix
    A = np.array([[4.0, 1.0],
                  [1.0, 3.0]])
    
    # Determinant and Inverse
    det_A = np.linalg.det(A)
    inv_A = np.linalg.inv(A)
    print(f"Matrix A:\n{A")
    print(f"Determinant: {det_A:.4f")
    print(f"Inverse A^(-1):\n{inv_A")
    
    # Eigendecomposition: A * v = lambda * v
    eigenvalues, eigenvectors = np.linalg.eig(A)
    print(f"\nEigenvalues (lambda): {eigenvalues")
    print(f"Eigenvectors (columns):\n{eigenvectors")
    
    # Verification: A @ v0 == lambda0 * v0
    v0 = eigenvectors[:, 0]
    lambda0 = eigenvalues[0]
    lhs = A @ v0
    rhs = lambda0 * v0
    print(f"Verification: A @ v0 matches lambda0 * v0: {np.allclose(lhs, rhs)\n")

def run_gradient_descent_simulation():
    print("=== 2. CALCULUS: GRADIENT DESCENT OPTIMIZATION ===")
    # Minimize quadratic loss function: L(w) = (w - 4)^2 + 6
    # Analytical gradient: dL/dw = 2 * (w - 4)
    # Global minimum is at w* = 4.0, L(w*) = 6.0
    
    w = 0.0          # Initial parameter guess
    alpha = 0.1      # Learning rate
    iterations = 25  # Number of update steps
    
    print(f"Starting parameter w_0 = {w:.2f")
    for step in range(1, iterations + 1):
        grad = 2.0 * (w - 4.0)
        w = w - alpha * grad
        loss = (w - 4.0)**2 + 6.0
        if step % 5 == 0 or step == 1:
            print(f"Iteration {step:02d | Gradient: {grad:8.4f | w: {w:6.4f | Loss: {loss:6.4f")
    print(f"Convergence reached: w = {w:.4f (Optimal is 4.0000)\n")

def run_bayes_theorem_calculation():
    print("=== 3. PROBABILITY: BAYES' THEOREM MEDICAL DIAGNOSIS ===")
    # Disease prevalence in population: P(D) = 0.01 (1%)
    # Test Sensitivity: P(Positive | D) = 0.95 (95%)
    # Test Specificity: P(Negative | ~D) = 0.90 -> False Positive Rate P(Positive | ~D) = 0.10
    
    P_D = 0.01
    P_not_D = 1.0 - P_D
    P_Pos_given_D = 0.95
    P_Pos_given_not_D = 0.10
    
    # Marginal Evidence P(Positive)
    P_Pos = (P_Pos_given_D * P_D) + (P_Pos_given_not_D * P_not_D)
    
    # Posterior Probability P(D | Positive)
    P_D_given_Pos = (P_Pos_given_D * P_D) / P_Pos
    
    print(f"Prior Probability P(D): {P_D:.2%")
    print(f"Test Sensitivity P(+|D): {P_Pos_given_D:.2%")
    print(f"False Positive Rate P(+|~D): {P_Pos_given_not_D:.2%")
    print(f"Total Evidence Probability P(+): {P_Pos:.4f")
    print(f"Posterior Probability P(D|+): {P_D_given_Pos:.2%")
    print("Insight: Even with 95% sensitivity, low base rate yields ~8.7% posterior!\n")

def run_statistics_demo():
    print("=== 4. STATISTICS: COVARIANCE & PEARSON CORRELATION ===")
    np.random.seed(42)
    
    # Generate two correlated features: X and Y = 2.5 * X + noise
    X = np.random.normal(loc=50.0, scale=10.0, size=100)
    Y = 2.5 * X + np.random.normal(loc=0.0, scale=5.0, size=100)
    
    mean_X, mean_Y = np.mean(X), np.mean(Y)
    std_X, std_Y = np.std(X, ddof=1), np.std(Y, ddof=1)
    
    # Compute Covariance and Pearson Correlation Matrix
    cov_matrix = np.cov(X, Y)
    corr_matrix = np.corrcoef(X, Y)
    
    print(f"Feature X -> Mean: {mean_X:.2f, Sample Std: {std_X:.2f")
    print(f"Feature Y -> Mean: {mean_Y:.2f, Sample Std: {std_Y:.2f")
    print(f"\nSample Covariance Matrix:\n{cov_matrix")
    print(f"Covariance(X, Y): {cov_matrix[0, 1]:.4f")
    print(f"\nPearson Correlation Matrix:\n{corr_matrix")
    print(f"Pearson Correlation r(X, Y): {corr_matrix[0, 1]:.4f (Strong positive correlation)")

if __name__ == "__main__":
    run_linear_algebra_demo()
    run_gradient_descent_simulation()
    run_bayes_theorem_calculation()
    run_statistics_demo()