Unit 2: Data Fundamentals

In empirical research and data science, data represents a systematic collection of observations, measurements, or facts recorded about entities of interest. Before performing statistical computations, a data practitioner must understand how real-world phenomena are mapped into quantitative representations.

Variables, Attributes, and Observations

• Observation (Entity / Record / Sample): A single unit of analysis representing an individual entity (such as a customer, a transaction, a sensor reading, or a hospital patient). In tabular datasets, observations correspond to rows.
• Variable (Feature / Dimension / Column): A measurable characteristic, trait, or quantity that can assume different values across different observations. In tabular datasets, variables correspond to columns.
• Attribute (Value / State): The specific realization, measurement, or category recorded for a given variable on a specific observation (e.g., the attribute value "34.5" for the variable "Body Temperature").

Stevens' Four Levels of Measurement

In 1946, psychologist Stanley Smith Stevens formulated the foundational taxonomy of measurement scales. The level of measurement dictates which mathematical operations and statistical tests are mathematically legitimate on a given variable:

Beyond Stevens' measurement levels, data science models classify data into two overarching branches based on their mathematical nature: Qualitative (Categorical) Data and Quantitative (Numerical) Data.

                                  ALL DATA
                                     |
              +----------------------+----------------------+
              |                                             |
              v                                             v
     QUALITATIVE (Categorical)                    QUANTITATIVE (Numerical)
     Non-numeric characteristics                 Measurable numeric quantities
              |                                             |
       +------+------+                               +------+------+
       |             |                               |             |
       v             v                               v             v
    NOMINAL       ORDINAL                         DISCRETE     CONTINUOUS
   Labels, tags  Ranked order                 Countable whole  Measurable real
   No order      Unknown gaps                     numbers       numbers (infinite)
   (e.g., Color, (e.g., Low,                  (e.g., Logins,   (e.g., Latency,
    Gender)       Med, High)                   Seats, Items)    Voltage, Weight)
        

Qualitative (Categorical) Data

Qualitative data describes characteristics, qualities, or classifications that cannot be quantified naturally on a continuous numeric line.

• Nominal Data: Variables that act strictly as identifiers or qualitative grouping keys (e.g., payment method: Credit Card, UPI, Net Banking).
• Ordinal Data: Variables that convey relative placement along a gradient (e.g., user seniority tier: Junior, Mid, Senior, Principal).

Quantitative (Numerical) Data

Quantitative data represents measurable quantities where values have direct numeric meaning and can be subjected to arithmetic computation.

• Discrete Data: Numeric variables restricted to distinct, separate values (most commonly non-negative integers). Values are counted rather than measured. There are no intermediate fractional values between two consecutive integers.
  • Examples: Number of active student enrollments, number of failed login attempts, quantity of items in a shopping cart.
• Continuous Data: Numeric variables that can take any infinite real value within a given interval. Values are measured and are limited only by the precision of the measuring instrument.
  • Examples: API request duration (e.g., 23.418 milliseconds), server CPU temperature (e.g., 68.32 degrees Celsius), revenue in currency with fractional cents.

Before data can be analyzed, it must be captured. In data engineering and statistics, data is categorized by its origin relative to the investigating organization:

A dataset is a structured collection of related data records managed as an atomic unit. In enterprise systems, datasets are serialized, transmitted, and persisted across a variety of file formats, each engineered with specific trade-offs between human readability, storage compression, and query performance:

Common Data Serialization Formats

• CSV (Comma-Separated Values): Plain text format storing tabular records separated by delimiters (commas or tabs). Lightweight and universally supported, but lacks data type definitions (all values are strings during transit) and is susceptible to delimiter collision issues when text contains commas.
• Excel (XLSX / XLS): Proprietary binary and zipped XML spreadsheet format supporting multi-tab workbooks, embedded formulas, and cell formatting. Prone to unintended automated type conversions (e.g., auto-converting gene names or leading zeroes in postal codes into numbers).
• JSON (JavaScript Object Notation): Lightweight, human-readable hierarchical key-value format standard for REST APIs and web services. Supports nested arrays and polymorphic records, but incurs significant storage overhead due to repeating string keys across every record.
• XML (Extensible Markup Language): Verbose, tag-delimited hierarchical markup language. Enforces schema validation via XSD (XML Schema Definition), widely used in legacy enterprise finance and healthcare pipelines (HL7).
• Relational & Columnar Databases: Relational DBMS (PostgreSQL, MySQL) store data in normalized rows optimized for transactional ACID writes. In contrast, modern analytical warehouses (Snowflake, BigQuery, Parquet files) store data in column-oriented format, reading only the specific queried attributes across billions of rows without full-table scans.

import json
import pandas as pd
import io

# 1. Ingesting CSV with explicit dtype definitions
csv_raw = """userId,region,monthlySpend,churnRisk
u101,North,145.50,False
u102,South,290.00,True
u103,West,85.25,False"""

df_csv = pd.read_csv(
    io.StringIO(csv_raw),
    dtype={"userId": "string", "region": "category", "monthlySpend": "float64", "churnRisk": "boolean"}
)
print("CSV Schema & Types:\n", df_csv.dtypes)

# 2. Ingesting Hierarchical JSON Payload
json_payload = [
    {"id": 1, "user": {"name": "Aarav", "tier": "Gold"}, "logins": [12, 14, 19]},
    {"id": 2, "user": {"name": "Diya", "tier": "Platinum"}, "logins": [25, 30]}
]
df_json = pd.json_normalize(json_payload)
print("\nNormalized JSON Structure:\n", df_json)

In production systems, machine learning models and analytical dashboards are strictly bound by the Garbage In, Garbage Out (GIGO) principle. Feeding noisy, corrupted, or inconsistent data into advanced algorithms will inevitably produce erroneous predictions. Professional data practitioners evaluate data hygiene across six core dimensions:

+-----------------------------------------------------------------------------------------------+
|                               SIX DIMENSIONS OF DATA QUALITY                                  |
+-----------------------------------------------------------------------------------------------+
| 1. ACCURACY     | Values represent the real-world truth without distortion or sensor drift.   |
| 2. COMPLETENESS | Critical fields have no unexpected nulls, omissions, or truncated records.  |
| 3. CONSISTENCY  | Identical facts recorded across multiple systems agree without discrepancy. |
| 4. TIMELINESS   | Data is current, fresh, and available within the operational decision window|
| 5. VALIDITY     | Attribute values strictly conform to domain rules, schemas, and ranges.     |
| 6. UNIQUENESS   | Each real-world entity is recorded exactly once without duplicate rows.      |
+-----------------------------------------------------------------------------------------------+
        

Evaluation Breakdown

1. Accuracy: Does a recorded temperature of 38 degrees Celsius match the physical state of the patient, or was the thermometer uncalibrated?
2. Completeness: If 40% of user profile records have null values for the "Age" variable, can age be reliably utilized in predictive churn modeling?
3. Consistency: If the billing system lists customer status as "Active" but the authentication database lists the same customer as "Suspended", a consistency defect exists.
4. Timeliness: Financial transaction feeds delayed by 3 hours cannot support real-time credit card fraud prevention models that require 50-millisecond response latencies.
5. Validity: Does an email field contain an "@" symbol and top-level domain? Does an age field contain negative numbers like -5?
6. Uniqueness: Are multiple user rows created when a single customer registers twice with subtle name variations (e.g., "Shivan Kumar" vs "S. Kumar")?

When profiling real-world datasets, three ubiquitous data pathologies inevitably corrupt analysis: Missing Values, Duplicate Records, and Statistical Outliers.

1. Missing Values: Root Mechanisms & Handling

Data is rarely missing purely by accident. Donald Rubin's missing data framework establishes three distinct missingness mechanisms:

• Missing Completely at Random (MCAR): The probability of a value being missing is completely unrelated to any observed or unobserved variable (e.g., a lab technician accidentally drops a test tube). Dropping MCAR records causes minimal bias, though it reduces sample size.
• Missing at Random (MAR): The probability of missingness depends on other observed variables, but not on the missing value itself (e.g., younger users are statistically less likely to report their annual income than older users). Imputation using regression or KNN conditioned on the observed variables is valid.
• Missing Not at Random (MNAR): The missingness depends directly on the unobserved value itself (e.g., individuals with severe depression are less likely to complete a depression mental health assessment). Dropping or naively imputing MNAR data introduces severe statistical bias.

2. Duplicate Records: Deduplication Mechanics

Duplicates arise from retry storms in webhooks, distributed event streaming without idempotence keys, and multiple system migrations.

• Exact Duplicates: Rows where every single column value is identical across multiple entries. Remediated via primary key constraints or hash deduplication (e.g., df.drop_duplicates()).
• Fuzzy Duplicates: Records referring to the same real-world entity with minor typographic discrepancies (e.g., "Main St." vs "Main Street"). Remediated through probabilistic entity resolution algorithms and Levenshtein string distance metrics.

3. Outliers: Nature & Detection Methods

An outlier is an observation that deviates so significantly from other observations as to arouse suspicion that it was generated by a different mechanism. Outliers may represent sensor measurement noise, data entry errors, or genuine high-value anomalies (such as a massive fraudulent transaction).

   Lower Outliers                 Normal Distribution Span                   Upper Outliers
  (Data entry errors)               (Q1 to Q3 = Middle 50%)                  (Genuine anomalies)
         *  *         |-----------------[===========|===========]-----------------|       *   *
                      ^                 ^           ^           ^                 ^
                      |                 |           |           |                 |
                   Lower Fence          Q1        Median        Q3           Upper Fence
                (Q1 - 1.5 * IQR)      (25%)       (50%)       (75%)       (Q3 + 1.5 * IQR)
                      |                                                           |
                      +<------------------- Valid Data Span --------------------->+
        

import numpy as np
import pandas as pd

# Sample dataset with missing values, duplicate row, and extreme outliers
data = {
    "id": [1, 2, 3, 4, 5, 5, 6, 7],
    "salary": [45000, 52000, None, 48000, 51000, 51000, 490000, 47000]
}
df = pd.DataFrame(data)

# 1. Deduplication
df = df.drop_duplicates(subset=["id"])

# 2. Missing Value Imputation (Using Median to protect against skew)
salary_median = df["salary"].median()
df["salary"] = df["salary"].fillna(salary_median)

# 3. Outlier Detection using the Interquartile Range (IQR) method
q1 = df["salary"].quantile(0.25)
q3 = df["salary"].quantile(0.75)
iqr = q3 - q1

lower_fence = q1 - (1.5 * iqr)
upper_fence = q3 + (1.5 * iqr)

# Filter out extreme outliers
df_clean = df[(df["salary"] >= lower_fence) & (df["salary"] <= upper_fence)]
print("Cleaned Dataset without Outliers:\n", df_clean)

Data science does not operate in a moral or legal vacuum. Because modern datasets routinely ingest intimate human behavioral, physiological, and financial records, software engineers and data practitioners bear legal and ethical responsibilities to safeguard individual privacy.

Personally Identifiable Information (PII)

PII is defined as any data element that can be used directly or in combination with other indirect data points to uniquely distinguish, locate, or trace an individual's real-world identity.

• Direct Identifiers: Full legal name, social security / Aadhaar number, email address, passport number, telephone number, facial biometric vectors.
• Quasi-Identifiers (Indirect): Date of birth, postal code, gender, job title. (In a landmark MIT study, computer scientist Latanya Sweeney proved that 87% of the US population can be uniquely identified by the combination of just three quasi-identifiers: 5-digit ZIP code, gender, and date of birth).

Global Data Protection Regulatory Frameworks

Technical Anonymization Paradigms

1. Data Masking: Obfuscating sensitive characters while preserving structural format (e.g., masking credit cards as ****-****-****-4242).
2. Cryptographic Salted Hashing: Passing PII through a one-way mathematical function (such as SHA-256 with an unpredictable cryptographic salt), allowing linkage analysis without storing raw identifiers.
3. K-Anonymity: A dataset possesses k-anonymity if each record cannot be distinguished from at least k-1 other records whose quasi-identifiers are identical, accomplished by bucket generalizing ages (e.g., "20-30" instead of "24") and suppressing rare postal codes.
4. Differential Privacy: Mathematically injecting calibrated statistical noise into query responses such that the presence or absence of any single individual in the database cannot be determined by an adversary.