A first-principles comparative study of parametric and non-parametric discrete decision models: Logistic Regression, K-Nearest Neighbors, Naive Bayes, Decision Trees, Random Forest ensembles, Support Vector Machines (SVM), the Kernel Trick, and full-spectrum diagnostic evaluation.
Despite its historical name, Logistic Regression is a foundational linear classification algorithm. Instead of modeling continuous outcomes, it estimates the posterior probability P(Y = 1 | x) that an instance belongs to a positive binary class.
The linear combination w^T x + b is unbounded ((-infty, +infty)), making it impossible to interpret directly as a probability. We model the log-odds (logit):
Inverting this logit expression yields the Sigmoid (Logistic) Function, which maps any real logit z in R monotonically into the valid probability range (0, 1):
Because the sigmoid introduces non-linearity, Mean Squared Error yields a non-convex surface. We train parameters by maximizing Likelihood under the Bernoulli distribution, which is equivalent to minimizing Binary Cross-Entropy Loss:
The decision boundary corresponds to the hyperplane where the log-odds equal zero (z = 0), yielding equal class probability p_predicted = 0.5:
A non-parametric, lazy learning algorithm that stores all training instances with zero offline training cost (O(1)). At inference time, it computes distance between query point x_q and all stored points:
d(x_q, x_i) = √{Sum j=1^d (xqj - xij)^2} quad (L₂ Euclidean)k closest neighbors:
y_predicted = arg max_c Sum i in N_k(x_q) I(y_i = c)As dimensions d > 50, points become equidistant in high-dimensional space ((d max - d min / d min) → 0), rendering proximity meaningless.
Applies Bayes' Theorem by assuming conditional feature independence given class y:
P(y x) propto P(y) Product j=1^d P(x_j y)y_predicted = arg max_{c in {1, dots, K }} [ ln P(y = c) + Sum j=1^d ln P(x_j y = c) right]P(x_j|y) follows N(μjc, σjc^2)).α=1 to counts to prevent zero-probability collapse (P(x_j|y) = 0).Decision trees recursively partition the feature space into axis-aligned rectangular hyper-rectangles using greedy top-down splitting.
Measures impurity or disorder in a node containing K classes:
H(S) = -Sum k=1^K p_k log₂(p_k)A maximizing entropy reduction:
IG(S, A) = H(S) - Sum v in values(A) (S_v / S) H(S_v)Probability of incorrectly labeling a randomly chosen element if it were randomly labeled according to the class distribution:
G(S) = 1 - Sum k=1^K p_k^2log_2 evaluations. Maximum at 0.5 for binary balanced split; minimum at 0.0 for pure node.
Single decision trees have high variance and overfit rapidly. Random Forest (Leo Breiman, 2001) resolves this through two layers of stochastic randomization:
B trees trains on a distinct bootstrap sample D_b drawn with replacement (N instances). ~36.8% of records are unpicked (Out-Of-Bag, OOB).
m = √(d) features is considered. This decorrelates the trees, preventing a single dominant feature from dictating all trees.
y_predicted = mode {T_1(x), dots, T_B(x) }. Drastically reduces variance without increasing bias.
While logistic regression seeks any decision boundary separating the classes, Support Vector Machines identify the unique hyperplane that maximizes the geometric margin between the closest points of each class.
The training observations that lie precisely on the margin boundary hyperplanes (w^Tx_i + b = ± 1). If any non-support vector is removed from the dataset, the learned hyperplane remains 100% identical.
C)
Allows bounded margin violations via slack variables xi_i ≥ 0:
min (1 / 2)w^2 + C Sum i=1^N xi_iC ⇒ hard margin, narrow margin, low bias/high variance. Small C ⇒ soft margin, wider margin, higher tolerance for misclassifications.
When data is not linearly separable in the original space R^d, we map it into a higher-dimensional Hilbert space φ(x) in R^D (D gg d) where a linear hyperplane exists. The Kernel Trick evaluates inner products directly in the original space without ever computing the high-dimensional coordinates:
φ(x) = [x_1, x_2, x_1^2 + x_2^2] ⇒Evaluating classification performance requires balancing trade-offs between false alarms (Type I error) and missed detections (Type II error).
| Diagnostic Metric | Mathematical Equation | Operational Focus | Real-World Domain Priority |
|---|---|---|---|
| Accuracy | (TP + TN / TP + TN + FP + FN) |
Total correct ratio | Symmetric class distributions only. |
| Precision (PPV) | (TP / TP + FP) |
Minimizing False Positives (FP) |
Spam filtering, recommendation systems (avoid annoying user). |
| Recall (Sensitivity / TPR) | (TP / TP + FN) |
Minimizing False Negatives (FN) |
Oncology diagnosis, fraud, defect detection (missing positive is fatal). |
| F1-Score | 2 · (Precision · Recall / Precision + Recall) |
Harmonic mean balance | Single summary metric for imbalanced classes. |
| ROC-AUC | int_0^1 TPR(FPR-1(t)) dt |
Threshold-agnostic ranking power | Credit scoring risk models, clinical ranking. |
The following standalone script trains and benchmarks all 6 classification algorithms on a non-linear dataset, printing the Confusion Matrix, Classification Report, and ROC-AUC metrics:
# =====================================================================
# DevAscent Academy: Verified Classification Algorithms Benchmark
# Compares: Logistic Regression, KNN, Naive Bayes, Decision Tree,
# Random Forest, and Support Vector Machine (RBF Kernel)
# =====================================================================
import numpy as np
import pandas as pd
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix, classification_report, roc_auc_score, f1_score
def execute_classification_suite():
print("=== MULTI-MODEL CLASSIFICATION BENCHMARK ===")
np.random.seed(42)
# 1. Generate Non-Linear Two-Moons Classification Dataset
X, y = make_moons(n_samples=1000, noise=0.25, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
print(f"Dataset Size: {X.shape} | Training: {X_train.shape[0]} | Testing: {X_test.shape[0]}\n")
# 2. Instantiate Classifiers with Recommended Architectures
classifiers = {
"Logistic Regression": Pipeline([("scaler", StandardScaler()), ("clf", LogisticRegression())]),
"K-Nearest Neighbors (k=7)": Pipeline([("scaler", StandardScaler()), ("clf", KNeighborsClassifier(n_neighbors=7))]),
"Gaussian Naive Bayes": GaussianNB(),
"Decision Tree (Depth=4)": DecisionTreeClassifier(max_depth=4, random_state=42),
"Random Forest (100 Trees)": RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42),
"SVM (RBF Kernel, C=1.0)": Pipeline([("scaler", StandardScaler()), ("clf", SVC(probability=True, kernel='rbf', C=1.0, random_state=42))])
}
results = []
for name, model in classifiers.items():
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]
cm = confusion_matrix(y_test, y_pred)
auc = roc_auc_score(y_test, y_prob)
f1 = f1_score(y_test, y_pred)
acc = model.score(X_test, y_test)
results.append({
"Classifier": name,
"Accuracy": acc,
"F1-Score": f1,
"ROC-AUC": auc,
"TN": cm[0, 0],
"FP": cm[0, 1],
"FN": cm[1, 0],
"TP": cm[1, 1]
})
results_df = pd.DataFrame(results)
print(results_df[["Classifier", "Accuracy", "F1-Score", "ROC-AUC"]].to_string(index=False))
print("\n=== Confusion Matrix Breakdown ===")
print(results_df[["Classifier", "TP", "TN", "FP", "FN"]].to_string(index=False))
# Detailed Classification Report for Best Non-Linear Performer (SVM RBF)
print("\n=== Detailed Classification Report: Support Vector Machine (RBF) ===")
svm_best = classifiers["SVM (RBF Kernel, C=1.0)"]
y_pred_svm = svm_best.predict(X_test)
print(classification_report(y_test, y_pred_svm, target_names=["Moon 0", "Moon 1"], digits=4))
if __name__ == "__main__":
execute_classification_suite()p_predicted = σ(w^T x + b). A classification decision between classes 0 and 1 occurs at threshold p_predicted = 0.5, which happens precisely when σ(z) = 0.5 ⇔ z = 0. This implies w^T x + b = 0, which is the exact mathematical definition of an affine linear hyperplane. The sigmoid function merely compresses the linear output into probability values; it does not introduce non-linear curvature to the decision boundary.
C.
minw, b, xi (1 / 2)w₂² + C Sum i=1^N xi_i quad s.t. y_i(w^T x_i + b) ≥ 1 - xi_i, ; xi_i ≥ 0xi_i are slack variables measuring margin violations (xi_i = 0 for points outside margin, 0 < xi_i ≤ 1 for points within margin but correctly classified, and xi_i > 1 for misclassified points). The hyperparameter C > 0 controls the trade-off: a large C heavily penalizes slack violations, creating a narrow, rigid margin (low bias, high risk of overfitting); a small C tolerates more slack violations, producing a wider margin that generalizes better on noisy real-world data (low variance, higher bias).
K(x, z) can be expressed as an inner product langle φ(x), φ(z) rangle in some reproducing kernel Hilbert space (RKHS). The Kernel Trick evaluates this inner product directly in the low-dimensional input space: K(x, z) = exp(-γ x - z^2). The Taylor expansion of the RBF kernel corresponds to an infinite polynomial feature mapping φ(x) in R^infty. Explicitly mapping features into infinite dimensions is computationally impossible, but computing the scalar Gaussian kernel takes just O(d) time and O(1) extra memory.
B identically distributed, independent random variables each with variance σ^2, the variance of the mean is (σ^2 / B). By growing deep unpruned trees (maintaining low bias) and decorrelating them via bootstrap bagging and random feature subspaces (√(d)), Random Forest averages hundreds of low-bias trees, shrinking the ensemble variance down toward zero while keeping the overall bias exceptionally low.
(P + R / 2). If a pathological classifier predicts the positive class for every single instance, it achieves Recall = 1.0 and Precision = 0.01 under heavy class imbalance. The arithmetic mean would be (1.0 + 0.01 / 2) ≈ 0.505, giving a deceptively acceptable score for a useless model. The Harmonic Mean:
F₁ = (2 / (1 / P) + (1 / R)) = (2 · P · R / P + R)P or R approaches zero). For the pathological classifier, F_1 = (2 × 0.01 × 1.0 / 1.01) ≈ 0.0198, correctly penalizing extreme imbalance between precision and recall.
P(w_1, w_2|y) = P(w_1|y)P(w_2|y)) is statistically violated in natural language (words like "credit" and "card" strongly co-occur), classification accuracy depends on the sign of the posterior ratio (the ranking), NOT the exact numerical calibration of probability magnitudes. Even if dependent words cause the estimated probabilities to be unrealistically overconfident (shifting toward 0.00001 or 0.99999), as long as the correct class maintains the highest log-likelihood sum arg max_c P(y=c|x), the decision boundary remains completely correct.
AUC = P Big( p_predicted(x^+) > p_predicted(x^-) Big)0.5 represents random guessing (equal probability of ranking either higher); an AUC of 1.0 denotes perfect separation where every positive instance is ranked above every negative instance. This makes AUC completely independent of the selected classification probability threshold τ and invariant to monotonic positive probability transformations.
0.0 (pure node) and maximum at 0.5 (Gini) or 1.0 (Entropy) for binary splits. In 98% of machine learning applications, Gini and Entropy produce virtually identical tree structures because their underlying curves are functionally similar (G(p) ≈ (1 / 2) H(p) ln 2). However, Gini Impurity is computationally faster because calculating Sum p_k^2 involves basic floating-point multiplications, whereas Entropy requires costly transcendental logarithmic evaluations (log_2(p_k)). Entropy slightly penalizes mixed probabilities more aggressively, occasionally yielding slightly more balanced subtrees at the expense of slower training throughput.