A rigorous architectural and mathematical study of core machine learning algorithms: Ordinary Least Squares (OLS) regression assumptions and polynomial expansions, Logistic Regression log-odds geometry, non-parametric KNN, Naive Bayes conditional probability, Support Vector Machines (SVM) maximum-margin hyperplanes, CART Decision Trees and Random Forest ensembles, K-Means and Hierarchical Agglomerative clustering, Principal Component Analysis (PCA) eigendecomposition, and Market Basket Association Rule mining.
Regression models predict a real-valued continuous target variable y from a set of predictor features X. The foundational benchmark of continuous statistical learning is Ordinary Least Squares (OLS) Linear Regression.
Simple Linear Regression models the relationship between an independent variable X and a dependent variable y as a linear equation:
Predicted_y = Intercept + (Slope * X)
OLS solves for the optimal parameters by minimizing the Residual Sum of Squares (RSS): the sum of the squared vertical distances between the observed ground-truth data points and the fitted regression hyperplane:
RSS = Sum_over_all_samples((Actual_y - Predicted_y) ^ 2)
Target (y)
^ * (Actual Data Point: y_i)
| /|
| / | <--- Vertical Residual Error: e_i = (Actual_y - Predicted_y)
| / |
| / v
| /----+ (Fitted Prediction Point on Line: y_hat)
| /
| /
| / Fitted OLS Hyperplane: y = Beta_0 + Beta_1 * x
| /
| /
+----------+----------------------------------------------------> Feature (x)
Intercept (Beta_0)
According to the Gauss-Markov theorem, OLS estimators are the Best Linear Unbiased Estimators (BLUE) if and only if the following five structural conditions are satisfied:
Expected_Value(Error | X) = 0. No omitted variable bias.
When relationships exhibit curvature, Polynomial Regression models non-linear phenomena by generating higher-order power terms of existing features (e.g., X^2, X^3). While the feature curve is non-linear in geometric space, it remains linear in the parameter space, allowing standard OLS matrix solutions.
MAE = Sum(|Actual_y - Predicted_y|) / n
MSE = Sum((Actual_y - Predicted_y) ^ 2) / n
RMSE = Square_Root(MSE)
R_Squared = 1 - (Residual_Sum_of_Squares / Total_Sum_of_Squares)
Applying standard OLS regression to binary classification produces fundamental mathematical flaws: linear lines produce unbounded predictions extending to negative infinity and positive infinity, violating the Kolmogorov probability axioms (where probability must strictly be bounded within [0.0, 1.0]).
Logistic Regression resolves this limitation through a two-stage mathematical transformation:
Odds = Probability / (1 - Probability)
(0, 1) to the entire real number line (-inf, +inf):
Logit(p) = log(p / (1 - p)) = Beta_0 + (Beta_1 * X_1) + ... + (Beta_d * X_d)
z back into a bounded probability score between 0.0 and 1.0:
Probability = 1 / (1 + exp(-z)) where z = Beta * X
Probability P(y=1)
1.0 +-----------------------------------------------------********* (Asymptote at 1.0)
| *******
| ****
| ***
0.5 +------------------------------------*-------------------------- (Decision Threshold)
| ***
| ***
| ****
0.0 +*********------------------------------------------------------ (Asymptote at 0.0)
+--------------+--------------+--------------+--------------+---> Linear Score (z)
-6 -3 0 +3 +6
Unlike OLS, which can be solved analytically via the closed-form Normal Equation, Logistic Regression parameters are solved using Maximum Likelihood Estimation (MLE) via iterative numerical optimization algorithms (such as L-BFGS or Newton-Raphson).
KNN is a non-parametric, instance-based (lazy learning) algorithm. It performs zero statistical parameter optimization during training; it simply stores the training feature vectors in memory.
During inference on a novel observation, KNN calculates distances to all stored points, identifies the k closest neighbors, and assigns the class label via majority vote. Distance is typically computed using Euclidean distance:
Distance = Square_Root(Sum((x_sample - x_neighbor) ^ 2))
Because KNN relies entirely on geometric distance, as the number of features d grows, the volume of the feature space expands exponentially. In high dimensions, all points become nearly equidistant from one another, causing distance metrics to lose discriminative power.
Mandatory Requirement: Features must always be standardized (e.g., via StandardScaler) prior to running KNN.
Naive Bayes is a probabilistic classifier founded upon Bayes' Theorem of conditional probability:
Posterior_Probability P(Class | Features) = [ P(Features | Class) * Prior_Probability P(Class) ] / Marginal_Probability P(Features)
The algorithm is termed "Naive" because it makes the strong independence assumption that every feature is strictly conditionally independent of all other features given the class label. Despite this unrealistic assumption, Naive Bayes performs exceptionally well in high-dimensional text classification (e.g., spam filtering, sentiment analysis).
Support Vector Machines identify the unique separating hyperplane that achieves the Maximum Geometric Margin: the maximum perpendicular distance between the decision boundary and the closest training points of any class (known as the Support Vectors).
Feature 2
^
| Class A (Circles) Optimal Separating Hyperplane
| O O (w * x + b = 0)
| O O |
| (O) <=== Support Vector |
| - - - - - - - - - - - - - - - - - - - - - v - - - - - - - - - - - - Margin Boundary (+1)
| | /
| | MARGIN WIDTH (2/||w||)
| | /
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Margin Boundary (-1)
| [X] <=== Support Vector /
| X X /
| X X Class B (Crosses)
+-----------------------------------------------------------------> Feature 1
When data is non-linearly separable in its native feature space, the Kernel Trick implicitly projects observations into an infinite- or higher-dimensional Hilbert space where linear separation becomes possible, without ever explicitly computing the coordinates of points in that high-dimensional space:
Tree-based algorithms partition the feature space into a hierarchy of non-overlapping rectangular decision regions using greedy recursive binary splitting.
At each node, the Classification and Regression Tree (CART) algorithm evaluates all features and split thresholds, selecting the split that maximizes the reduction in node impurity:
Gini = 1 - Sum_over_all_classes((Probability_of_class_k) ^ 2)
Entropy = -Sum_over_all_classes(Probability_of_class_k * Log_base_2(Probability_of_class_k))
Information_Gain = Parent_Entropy - Weighted_Average_of_Children_Entropy
Individual Decision Trees suffer from extreme high variance (overfitting); small perturbations in the training data produce completely different tree architectures. Random Forests resolve this via Bagging (Bootstrap Aggregating) and Feature Subspace Sampling.
TOTAL TRAINING DATASET (n Samples, d Features)
|
+---> [ Bootstrap Sample 1 ] ===> [ Decision Tree 1 (Trained on sqrt(d) Features) ] ===> Prediction: Class A
|
+---> [ Bootstrap Sample 2 ] ===> [ Decision Tree 2 (Trained on sqrt(d) Features) ] ===> Prediction: Class A
|
+---> [ Bootstrap Sample B ] ===> [ Decision Tree B (Trained on sqrt(d) Features) ] ===> Prediction: Class B
|
v
[ ENSEMBLE MAJORITY VOTING / AVERAGE ]
|
v
FINAL PREDICTION: CLASS A
If one predictor is exceptionally dominant, standard bagging trees will all select that same predictor at the root node, resulting in highly correlated individual trees. Averaging correlated trees yields minimal variance reduction.
Random Forests force each split to choose from a random subset of features (typically Square_Root(Total_Features)). This intentional constraint de-correlates the individual trees; when hundreds of de-correlated trees vote together, the ensemble variance plummets without increasing bias.
Evaluating binary and multiclass classifiers requires deep structural performance diagnostics beyond simple accuracy percentages.
| Evaluation Metric | Mathematical Formulation (Natural English) | Operational Focus |
|---|---|---|
| Accuracy | (True_Positives + True_Negatives) / Total_Samples |
Overall correctness; completely deceptive on class-imbalanced datasets. |
| Precision | True_Positives / (True_Positives + False_Positives) |
Minimizing False Alarms (Type I error). Critical in spam filtering and marketing spend. |
| Recall (Sensitivity) | True_Positives / (True_Positives + False_Negatives) |
Minimizing Missed Detections (Type II error). Critical in medical diagnosis and fraud. |
| F1-Score | (2 * Precision * Recall) / (Precision + Recall) |
Harmonic mean; penalizes extreme imbalance between Precision and Recall. |
| ROC-AUC | Area under True Positive Rate vs. False Positive Rate curve | Measures class ranking separation quality across all possible decision thresholds. |
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, roc_auc_score, confusion_matrix
# 1. Instantiate and train Random Forest Classifier
rf_model = RandomForestClassifier(
n_estimators=100,
max_depth=8,
max_features="sqrt",
random_state=42
)
rf_model.fit(X_train, y_train)
# 2. Generate predictions and class probabilities
y_pred = rf_model.predict(X_test)
y_probs = rf_model.predict_proba(X_test)[:, 1]
# 3. Diagnostic reports
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
print("\nDiagnostic Metrics:\n", classification_report(y_test, y_pred))
print(f"Area Under ROC Curve (AUC): {roc_auc_score(y_test, y_probs):.4f}")Unsupervised Clustering partitions unlabeled observations into distinct, internally homogeneous groups without any ground-truth training guidance.
K-Means partitions n observations into k pre-specified clusters by minimizing the Within-Cluster Sum of Squares (Inertia / WCSS):
Inertia = Sum_over_clusters(Sum_over_points((Point - Centroid) ^ 2))
The standard algorithm executes via two alternating iterative steps:
INITIAL RANDOM CENTROIDS: ASSIGNMENT & UPDATE: FINAL CONVERGENCE:
+-------------------------------+ +-------------------------------+ +-------------------------------+
| * * x [Centroid 2]| | * * | o o o | | [ Cluster 1 ] | [ Cluster 2 ] |
| * * * | ==> | * * * | o o o o | ==> | * * * * | o o o o |
| | | [Boundary] | | | |
| x [Centroid 1] | |---------------+---------------| |---------------+---------------|
| o o o | | | + + + + | | | [ Cluster 3 ] |
| o o o o | | | + + + | | | + + + + |
+-------------------------------+ +-------------------------------+ +-------------------------------+
Hierarchical clustering constructs a nested tree of partitions without pre-specifying k. Agglomerative Clustering operates bottom-up: every point begins in its own isolated cluster, and the two closest clusters are iteratively merged according to a defined Linkage Criterion:
The complete clustering hierarchy is visualized via a Dendrogram; horizontal cuts across the dendrogram tree at varying vertical distance thresholds determine the final discrete cluster count.
High-dimensional datasets contain substantial feature redundancy, severe multicollinearity, and sparse geometry (the Curse of Dimensionality). Principal Component Analysis (PCA) is an unsupervised linear transformation that projects data from a high-dimensional space into an orthogonal lower-dimensional subspace while preserving maximal data variance.
X_centered = X - Mean(X).(d-by-d) Covariance Matrix quantifying all pairwise feature covariances.k eigenvectors to obtain the reduced (n-by-k) coordinates.from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
# 1. Standardize features (PCA requires zero mean and unit variance)
X_scaled = StandardScaler().fit_transform(X)
# 2. Fit PCA retaining 95% of total cumulative variance
pca = PCA(n_components=0.95)
X_projected = pca.fit_transform(X_scaled)
print(f"Original Dimension Count: {X.shape[1]}")
print(f"Reduced Dimension Count: {X_projected.shape[1]}")
print(f"Explained Variance Ratio per Component: {pca.explained_variance_ratio_.round(3)}")Association Rule Learning identifies non-obvious, high-probability co-occurrence patterns in transaction databases. In retail enterprise analytics, this is known as Market Basket Analysis.
An association rule takes the form {Antecedent A} => {Consequent B} (e.g., {Diapers} => {Beer}). The strength and validity of a rule are evaluated across three primary metrics:
Support(A => B) = Count(Transactions containing A and B) / Total_Transactions
Confidence(A => B) = Count(Transactions containing A and B) / Count(Transactions containing A)
Lift(A => B) = Confidence(A => B) / Support(B)
Lift = 1.0: Items A and B are completely independent; co-occurrence is pure random chance.Lift > 1.0: Items A and B are positively associated; buying A actively increases the likelihood of buying B.Lift < 1.0: Items A and B are substitute products; buying A reduces the likelihood of buying B.
Calculating all possible item combinations across a grocery inventory of 50,000 SKUs creates an intractable combinatorial explosion of 2^50000 subsets.
The Apriori Principle enforces an anti-monotonicity rule: All non-empty subsets of a frequent itemset must also be frequent. Conversely, if an itemset {Bread, Butter} is infrequent (below the minimum support threshold), any larger superset containing it (e.g., {Bread, Butter, Jam}) is guaranteed to be infrequent and is immediately pruned without calculation.