Unit 5: Statistics and Probability

Statistical analysis begins by distinguishing between a Population (the entire collection of all possible subjects or events under study) and a Sample (a representative subset drawn from that population). Characteristics describing a population are called parameters (conventionally denoted by Greek letters), whereas values computed from sample data are called statistics (denoted by Latin letters).

Measures of Central Tendency

Measures of central tendency pinpoint the single representative value around which a numerical distribution clusters:

• Mean (Arithmetic Average): The sum of all observed values divided by the total count of observations. Highly sensitive to extreme outliers; a single massive value pulls the mean sharply toward the tail.
• Median (50th Percentile): The middle value when observations are sorted in ascending order. If the sample size is even, the median is the arithmetic mean of the two central values. The median is robust to outliers, making it the preferred metric for skewed distributions (such as personal income or real estate prices).
• Mode: The most frequently occurring value in the dataset. Useful for categorical variables (e.g., most common customer operating system) and identifying multimodal distributions where multiple peaks exist.

Measures of Dispersion (Spread)

Centrality alone is incomplete; two distributions can have identical means of 100 while possessing vastly different risks or variabilities. Measures of dispersion quantify the extent to which data points scatter around the center:

To understand the complete profile of numerical data, analysts construct Frequency Distributions by partitioning continuous data into non-overlapping intervals (bins) and plotting them as a Histogram. The shape of this histogram reveals two fundamental higher-order statistical moments: Skewness and Kurtosis.

   POSITIVE (RIGHT) SKEW                        SYMMETRIC (NORMAL)                    NEGATIVE (LEFT) SKEW
   (Long tail on the right)                      (Bell-shaped curve)                  (Long tail on the left)

         ^                                              ^                                      ^
        / \                                            / \                                    / \
       /   \                                          /   \                                  /   \
      /     \                                        /     \                                /     \
     /       \____                                  /       \                          ____/       \
----+-------------\--------->                   ---+---------\-------->             --/-------------\-------->
   Mode  Med  Mean                                    Mean=Med=Mode                 Mean  Med  Mode

Condition: Mean > Median > Mode                 Condition: Mean = Median = Mode     Condition: Mode > Median > Mean
Examples: Wealth, Startup Valuations            Examples: Human Height, Errors      Examples: Age at Death (Developed)
        

Skewness (The Third Moment: Asymmetry)

Skewness measures the degree of asymmetry of a probability distribution around its mean:

• Symmetric Distribution (Skewness ≈ 0): The left and right halves are mirror images. The mean, median, and mode coincide at the exact center.
• Positive / Right-Skewed (Skewness > 0): The right tail is significantly longer or fatter than the left. A small number of unusually large values pull the arithmetic mean upward: Mean > Median > Mode.
• Negative / Left-Skewed (Skewness < 0): The left tail is significantly elongated. Infrequent, extremely low values pull the arithmetic mean downward: Mode > Median > Mean.

Kurtosis (The Fourth Moment: Tail Weight & Outliers)

Kurtosis measures the "tailedness" of the probability distribution relative to a standard normal distribution. It quantifies how much of the distribution's variance is driven by rare, extreme outliers versus modest frequent deviations:

• Mesokurtic (Excess Kurtosis = 0): Follows standard normal tail thickness (e.g., classical Gaussian bell curve).
• Leptokurtic (Excess Kurtosis > 0): Sharp, slender central peak with heavy, fat tails. High propensity for extreme outlier events ("Black Swan" events in financial risk modeling).
• Platykurtic (Excess Kurtosis < 0): Broad, flat central peak with thin, short tails. Extreme outliers are exceptionally rare.

Probability provides the mathematical framework for quantifying uncertainty. Formally established by Andrey Kolmogorov, the probability of an event E satisfies three fundamental axioms:

1. The probability of any event is non-negative and bounded: 0 ≤ P(E) ≤ 1.
2. The probability of the entire sample space S equals certainty: P(S) = 1.
3. For any mutually exclusive events, the probability of their union is the sum of their individual probabilities.

1. The Binomial Distribution (Discrete)

Models the number of successful outcomes k across a fixed number n of independent Bernoulli trials, where each trial has only two mutually exclusive outcomes (Success or Failure) with a constant probability of success p.

• Expected Mean: n × p
• Variance: n × p × (1 − p)
• Analytical Use Cases: Modeling how many customers convert out of 100 email clicks, predicting how many manufactured chips fail quality inspection on a conveyor line.

2. The Normal (Gaussian) Distribution (Continuous)

The ubiquitous symmetric, bell-shaped distribution governed entirely by two parameters: the population mean and the population standard deviation.

                           Mean
                             |
                      +------+------+
                     /       |       \
                    /    68.27% of Data \   (Within +/- 1 Standard Deviation)
             +-----+---------+---------+-----+
            /      |         |         |      \
           /     95.45% of Total Observations   \  (Within +/- 2 Standard Deviations)
    +-----+--------+---------+---------+--------+-----+
   /      |        |         |         |        |      \
  /    99.73% of All Physical Observations Fall Here     \  (Within +/- 3 Standard Deviations)
-+--------+--------+---------+---------+--------+--------+--->
-3 SD   -2 SD    -1 SD     Mean      +1 SD    +2 SD    +3 SD
(Z=-3)  (Z=-2)   (Z=-1)    (Z=0)     (Z=+1)   (Z=+2)   (Z=+3)
        

Standard Normal Distribution & Z-Scores

Any normal variable can be transformed into the Standard Normal Distribution (having a mean of 0 and a standard deviation of 1) by calculating its Z-Score:

Z = (Observation - Population_Mean) / Standard_Deviation

A Z-score indicates precisely how many standard deviations an individual observation lies above or below the mean, allowing direct comparison of variables measured on completely different physical scales (e.g., comparing a student's SAT score against their high school GPA).

Gathering data on an entire population is frequently physically impossible or financially prohibitive. Consequently, data scientists extract representative samples using rigorous probability sampling techniques:

• Simple Random Sampling: Every individual in the population possesses an equal probability of selection.
• Stratified Sampling: The population is partitioned into homogeneous subgroups (strata) based on demographic attributes (e.g., age bracket, state), and random samples are drawn proportionally from each stratum to ensure representation of minority groups.
• Systematic Sampling: Selecting every k-th individual from an ordered list after a random starting point.
• Cluster Sampling: The population is divided into natural heterogeneous geographic or organizational clusters (e.g., schools, cities); entire clusters are randomly chosen for full sampling.

The Central Limit Theorem (CLT)

The Central Limit Theorem is the foundational pillar of inferential statistics. It states:

The spread of this sampling distribution is quantified by the Standard Error of the Mean (SE):

Standard_Error = Population_Standard_Deviation / Square_Root(Sample_Size)

As sample size increases, standard error shrinks toward zero, meaning larger samples yield tighter, more accurate approximations of the true population parameter.

Confidence Intervals (CI)

A point estimate (such as a single sample mean) is never perfectly accurate. A Confidence Interval constructs a plausible range of values likely to contain the true population parameter at a specified confidence level (commonly 95%):

Confidence_Interval = Sample_Mean +/- (Critical_Value * Standard_Error)
/* For a 95% Confidence Level under a normal curve, Critical_Value = 1.96 */
CI_95 = Sample_Mean +/- (1.96 * (Sample_SD / Square_Root(n)))

Correct Interpretation: If we were to draw 100 independent random samples and compute a 95% confidence interval for each, approximately 95 of those 100 calculated intervals would successfully bracket the true population mean.

Hypothesis testing is a formal mathematical procedure for determining whether an empirical observation in sample data reflects a genuine, statistically significant population effect or is merely a product of random sampling noise.

1. Formulating Null and Alternative Hypotheses

• Null Hypothesis (H0): The baseline assertion of "no effect", "no difference", or "status quo". Assumes any observed variation is purely due to random chance (e.g., "The new website button design has no effect on user conversion rates.").
• Alternative Hypothesis (H1 / Ha): The experimental claim that a real effect, difference, or relationship exists (e.g., "The new website button design increases user conversion rates.").

2. The Decision Matrix: Type I vs. Type II Errors

                                        TRUE STATE OF THE POPULATION
                                  H0 is ACTUALLY TRUE            H0 is ACTUALLY FALSE
                            +------------------------------+------------------------------+
  STATISTICAL  Fail to      |       CORRECT DECISION       |        TYPE II ERROR         |
   DECISION    Reject H0    |     (Confidence: 1 - alpha)  |        (Beta: False Negative)|
                            +------------------------------+------------------------------+
               Reject H0    |        TYPE I ERROR          |       CORRECT DECISION       |
                            |   (Alpha: False Positive)    |    (Power: 1 - Beta)         |
                            +------------------------------+------------------------------+

Analogy:
- Type I Error (Alpha): Convicting an innocent defendant in court (False Alarm).
- Type II Error (Beta): Acquitting a guilty criminal due to insufficient evidence (Missed Defect).
        

3. The p-Value & Significance Level (Alpha)

• Significance Level (Alpha, α): The pre-determined risk tolerance for committing a Type I error (conventionally set to 0.05 or 5%). It represents the threshold probability below which we consider an event too improbable under H0 to be explained by chance alone.
• p-Value: The exact calculated probability of obtaining sample results at least as extreme as the observed data, assuming that the null hypothesis is completely true.
• Decision Rule:
  • If p-value ≤ α: Reject H0 (The result is statistically significant).
  • If p-value > α: Fail to Reject H0 (Insufficient evidence to claim an effect).

Selecting the appropriate statistical hypothesis test depends strictly on the number of groups compared and the measurement scales of the independent and dependent variables:

                           WHAT TYPE OF DATA ARE YOU COMPARING?
                                             |
                   +-------------------------+-------------------------+
                   |                                                   |
                   v                                                   v
         CATEGORICAL VARIABLES                               CONTINUOUS NUMERICAL VARIABLE
                   |                                                   |
         [CHI-SQUARE TEST]                                 HOW MANY GROUPS TO COMPARE?
         - Goodness of Fit (1 Var)                                     |
         - Test of Independence (2 Vars)             +-----------------+-----------------+
                                                     |                                   |
                                                     v                                   v
                                                 TWO GROUPS                     THREE OR MORE GROUPS
                                                     |                                   |
                                            [STUDENT'S T-TEST]                     [ONE-WAY ANOVA]
                                            - Independent (2 diff groups)          Evaluates F-statistic:
                                            - Paired (Before/After same group)     Between vs Within Var
        

1. Student's T-Test

Evaluates whether the difference between two sample means is statistically significant:

• One-Sample T-Test: Compares a single sample mean against a known theoretical or historical benchmark value.
• Two-Sample Independent T-Test: Compares the means of two distinct, unrelated groups (e.g., comparing test scores between Group A taught via Video vs Group B taught via Text).
• Paired T-Test: Compares repeated measurements taken on the exact same subjects across two time periods (e.g., patient blood pressure measured before and after drug administration).

2. Chi-Square Test (Χ²)

Evaluates discrete counts and categorical frequencies by comparing observed counts (O) against expected counts (E) derived under the null hypothesis of independence:

• Chi-Square Test of Independence: Tests whether an association exists between two categorical variables arranged in a cross-tabulation contingency table (e.g., testing whether customer churn status is independent of user subscription tier).

3. One-Way ANOVA (Analysis of Variance)

When comparing means across three or more groups (e.g., testing average sales performance across North, South, East, and West regional branches), performing repeated pairwise t-tests is unacceptable because it compounds the probability of a false positive (family-wise Type I error inflation).

ANOVA evaluates all groups simultaneously by computing the F-Statistic:

F = (Variance Between Groups) / (Variance Within Groups)

If the variance between the group averages is significantly larger than the random noise within the groups, the F-statistic yields a small p-value, prompting rejection of the null hypothesis that all group means are identical.

In bivariate and multivariate analysis, data scientists examine how two continuous variables behave in tandem:

Covariance vs. Pearson Correlation Coefficient

• Covariance: Quantifies the directional relationship between two variables. If high values of X correlate with high values of Y, covariance is positive; if inverse, covariance is negative. However, covariance is unbounded and scale-dependent (multiplying X by 1,000 inflates covariance by 1,000 without altering the underlying relationship).
• Pearson Correlation Coefficient (r): A standardized measure of linear association obtained by dividing covariance by the product of both variables' standard deviations. Strictly bounded between -1.0 and +1.0:
  • r = +1.0: Perfect positive linear correlation.
  • r = 0.0: Zero linear correlation.
  • r = -1.0: Perfect negative linear correlation.

Simple Linear Regression Basics

While correlation merely evaluates the strength of an association, Linear Regression models the functional mathematical relationship to predict a dependent response variable y from an independent predictor variable x:

y = Intercept + (Slope * x) + Residual_Error

• Ordinary Least Squares (OLS): The mathematical criterion that finds the unique line that minimizes the sum of squared differences (residuals) between observed points and the line.
• Coefficient of Determination (R-Squared, R²): Measures goodness of fit. Bounded between 0 and 1 (or 0% to 100%), it represents the exact proportion of variance in the dependent variable that is mathematically explained by the independent predictor.

import numpy as np
from scipy import stats

# 1. Independent Two-Sample T-Test
group_control = [82, 85, 88, 79, 84, 86, 81]
group_treatment = [89, 92, 94, 88, 95, 91, 90]
t_stat, p_val_ttest = stats.ttest_ind(group_control, group_treatment)
print(f"T-Test: t-statistic = {t_stat:.3f}, p-value = {p_val_ttest:.5f}")

# 2. One-Way ANOVA across 3 Groups
g1 = [12, 14, 15, 13, 16]
g2 = [18, 20, 19, 21, 17]
g3 = [25, 24, 28, 26, 27]
f_stat, p_val_anova = stats.f_oneway(g1, g2, g3)
print(f"One-Way ANOVA: F = {f_stat:.3f}, p-value = {p_val_anova:.6f}")

# 3. Simple Linear Regression (OLS)
study_hours = [1, 2, 3, 4, 5, 6, 7, 8]
exam_scores = [52, 58, 65, 70, 78, 82, 88, 95]
reg = stats.linregress(study_hours, exam_scores)
print(f"Regression: y = {reg.intercept:.2f} + {reg.slope:.2f}*x | R-squared = {reg.rvalue**2:.3f}")