Text preprocessing pipelines, tokenization algorithms, statistical vector spaces (BoW & TF-IDF), distributed word embeddings, sentiment classification, and modern Large Language Model foundations.
Raw human natural language is inherently unstructured, noisy, and high-dimensional. Statistical and deep learning models require deterministic numerical representations. Text preprocessing transforms raw unicode strings into sanitized, standardized lexical units:
Converts all characters to lowercase to eliminate vocabulary duplication (e.g., mapping "Neural", "NEURAL", and "neural" to a single token ID). Exception: In Named Entity Recognition (NER), case casing preserves entity signals (e.g., "Apple" vs. "apple").
Strips raw HTML tags, URLs, emoji sequences, punctuation, escape codes, and non-ASCII glyphs using compiled regular expressions (e.g., re.sub(r'[^a-zA-Z\s]', '', text)).
Removes ubiquitous grammatical function words ("the", "is", "at", "which") based on Zipf's Law. Because these words appear across nearly all documents, they carry near-zero discriminative mutual information for topic classification.
Tokenization segments continuous text streams into discrete atomic units (tokens) mapped to unique integer vocabulary indices.
| Tokenization Level | Operational Mechanism | Vocabulary Size (|V|) |
Out-of-Vocabulary (OOV) & Sequence Trade-offs |
|---|---|---|---|
| Word-Level | Splits on whitespace and punctuation (e.g., "unbelievable" → ["unbelievable"]). |
Massive (10^5--10^6) |
Suffers from severe Out-Of-Vocabulary (OOV) bottlenecks; misspellings and morphological variations produce <UNK> tokens. |
| Character-Level | Splits into individual characters (e.g., ["u", "n", "b", "e", ...]). |
Tiny (10^2--10^3) |
Zero OOV tokens; however, sequence lengths explode dramatically, taxing recurrent/transformer context limits. |
| Subword (BPE / WordPiece) | Iteratively merges the most frequent pairs of bytes/characters into common subword prefixes and suffixes. | Optimized (3 × 10^4--10^5) |
Industry Standard: Rare words decompose into known morphemes (e.g., "un" + "believ" + "able"). Zero OOV; optimal sequence length. Powers GPT, BERT, and LLaMA. |
Words appear in diverse inflected variants (e.g., "organize", "organizes", "organizing", "organizational"). Linguistic normalization unifies morphological variations to a common canonical base:
A rule-based heuristic approach (e.g., Porter Stemmer, Snowball) that algorithmically slices common derivational affixes without lexical vocabulary awareness.
O(L) string parsing), low memory usage."university" → "univers"); suffers from over-stemming ("organization" → "organ") and under-stemming.Uses lexical databases (e.g., WordNet) and Part-of-Speech (POS) tags to resolve words to their true linguistic lemma (dictionary root form).
"better" → "good" with POS=Adjective; "ran" → "run").
Traditional machine learning algorithms cannot ingest variable-length strings directly. Vector space models map documents into fixed-length numeric vectors in R|V|.
The Bag-of-Words (BoW) model represents a document as a histogram of token frequencies, discarding grammatical syntax and word order. A corpus of N documents across vocabulary V yields a sparse matrix X in RN × |V|, where Xi,j counts occurrences of token j in document i.
|V| = 50,000, a typical 20-word sentence is 99.96 % zeros, wasting memory. (2) Order Obliviousness: Cannot differentiate between "not bad, quite good" and "not good, quite bad". (3) Semantic Blindness: Words with identical meanings (e.g., "automobile" and "car") are assigned orthogonal basis vectors, with zero mathematical cosine similarity.
TF-IDF reweights token counts by their corpus-wide frequency. Frequent words like "the" or "data" that appear in nearly all documents receive low weights, while specific terms (e.g., "backpropagation") receive high weights:
TF(t, d)
The relative frequency of term t in document d:
TF(t, d) = (ft, d / Sum t' in d ft', d)
IDF(t)Penalizes ubiquitous corpus-wide terms (Scikit-Learn smoothed version):
IDF(t) = ln( (1 + N / 1 + DF(t)) right) + 1
Formulation: TF-IDF(t, d, D) = TF(t, d) × IDF(t) quad followed by L₂ normalization: v_d = (v_d / v_d₂)
To capture latent semantics and combat sparsity, Word Embeddings map words into dense, continuous vector spaces (R^d where d in [100, 300]). Grounded in the Distributional Hypothesis (J.R. Firth, 1957: "You shall know a word by the company it keeps"), words appearing in similar surrounding contexts occupy neighboring positions in the vector space.
Pioneered by Mikolov et al. (2013) using shallow 2-layer neural networks:
w_t given context window [wt-2, wt-1, wt+1, wt+2]. Fast training, high accuracy on frequent words.w_t. Slower, but better captures rare words and nuanced semantics.Dense embeddings encode linear relational semantics, allowing vector arithmetic via cosine similarity:
vec{v}King - vec{v}Man + vec{v}Woman ≈ vec{v}Queen
GloVe (Pennington et al.): Computes global co-occurrence matrix factorization. FastText (Bojanowski et al.): Embeds subword n-grams, enabling representations for out-of-vocabulary words.
Text Classification maps variable-length texts to discrete categorical labels (e.g., spam detection, customer intent routing). Sentiment Analysis focuses on detecting polarity (positive, negative, neutral) or emotional states.
| Architecture Class | Feature Representation | Classifier Model | Production Characteristics |
|---|---|---|---|
| Classical Linear NLP | N-gram TF-IDF (ngram _range=(1, 2)) |
Multinomial Naive Bayes, Linear SVM, Logistic Regression | Extremely fast training and inference (sub-millisecond); handles high-cardinality vocabularies well; strong baseline for sentiment tasks. |
| Recurrent Deep Models | Word2Vec / GloVe pre-trained embeddings | Bi-directional LSTM / GRU with attention pooling | Captures sequential word ordering and negation phrases (e.g., "not great"); slower inference; superseded by transformers. |
| Transformer Fine-Tuning | Contextualized token embeddings | BERT / RoBERTa / DeBERTa with classification head | State-of-the-art accuracy across complex idioms and sarcasm; high GPU compute requirements for training and inference. |
Static word embeddings (Word2Vec) suffer from a major limitation: Polysemy. The word "bank" receives identical vector coordinates regardless of whether it refers to a financial institution or a riverbank. Large Language Models (LLMs) resolve this by computing dynamic, context-dependent representations using self-attention.
Trained on web-scale token corpora using the Causal Language Modeling objective: predicting the next token in a sequence given all preceding tokens:
LCLM = -Sum t=1^T log P(w_t mid w₁, dots, wt-1)
Fine-tunes the base model on curated prompt-response pairs, teaching it to follow user instructions, answer questions, and format structured outputs rather than just auto-completing raw text.
Uses Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO) to align outputs with human intent—maximizing helpfulness, accuracy, and safety.
The following self-contained script implements an end-to-end NLP pipeline: text preprocessing, n-gram TF-IDF vectorization, Logistic Regression sentiment classification, out-of-sample evaluation, and feature attribution analysis with pure white syntax contrast (`#ffffff !important`).
# =====================================================================
# DevAscent Academy: Production NLP & Sentiment Classification Pipeline
# Requires: numpy, pandas, scikit-learn
# =====================================================================
import re
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import classification_report, accuracy_score, confusion_matrix
def clean_text_corpus(raw_text):
"""Sanitize raw text strings via compiled regex rules."""
# Convert to lowercase
text = raw_text.lower()
# Strip HTML tags
text = re.sub(r'<[^>]+>', ' ', text)
# Strip URLs
text = re.sub(r'http\S+|www\S+', ' ', text)
# Retain only alphabetic characters and single whitespace
text = re.sub(r'[^a-z\s]', ' ', text)
# Collapse multiple whitespaces
text = re.sub(r'\s+', ' ', text).strip()
return text
def execute_nlp_sentiment_benchmark():
# -----------------------------------------------------------------
# 1. Synthesize Representative E-Commerce Review Corpus (N=600)
# -----------------------------------------------------------------
positive_reviews = [
"Incredible product! The build quality is top-notch and battery life exceeds expectations.",
"Delighted with this purchase. Outstanding customer support and lightning fast shipping.",
"Superb performance. It runs cool, whisper quiet, and handles heavy workloads effortlessly.",
"Hands down the best purchase I have made this year. High quality, intuitive, and reliable.",
"Exceeded all my expectations! Highly recommend to anyone seeking durable performance.",
"Flawless integration with my existing setup. Setup was a breeze and works like a charm.",
"Five stars! Beautiful aesthetics, premium tactile feel, and worth every single penny.",
"Remarkable engineering. Solid construction and responsive controls. Truly love it."
] * 40 # 320 Positive instances
negative_reviews = [
"Terrible experience. The item arrived broken and customer support refused to issue a refund.",
"Complete waste of money. Low quality plastics, sluggish performance, and constant crashes.",
"Fails to work as advertised. Battery drains within two hours and constantly overheats.",
"Horrible build quality. Buttons fell off within three days of regular usage. Avoid!",
"Extremely disappointed. Glitchy software, terrible documentation, and unresponsive support.",
"Dreadful product. Broke after one week of light use. Save your money and look elsewhere.",
"Poor design and cheap manufacturing. Will never purchase from this brand ever again.",
"Frustrating and unusable. Constant connection dropouts and agonizingly slow speeds."
] * 35 # 280 Negative instances
raw_corpus = positive_reviews + negative_reviews
raw_labels = [1] * len(positive_reviews) + [0] * len(negative_reviews)
# -----------------------------------------------------------------
# 2. Text Preprocessing Stream
# -----------------------------------------------------------------
cleaned_corpus = [clean_text_corpus(doc) for doc in raw_corpus]
df = pd.DataFrame({'review_text': cleaned_corpus, 'sentiment': raw_labels})
print(f"[Corpus] Total Reviews: {len(df)} | Balance: "
f"{np.bincount(raw_labels)[1]} Positive vs {np.bincount(raw_labels)[0]} Negative")
# -----------------------------------------------------------------
# 3. Train-Test Splitting (Stratified)
# -----------------------------------------------------------------
X_train_raw, X_test_raw, y_train, y_test = train_test_split(
df['review_text'], df['sentiment'], test_size=0.25, random_state=42, stratify=df['sentiment']
)
# -----------------------------------------------------------------
# 4. Statistical N-Gram TF-IDF Vectorization
# -----------------------------------------------------------------
print("\n--- [Phase 1] N-Gram TF-IDF Vectorization ---")
tfidf = TfidfVectorizer(
ngram_range=(1, 2), # Capture both unigrams and bigrams
max_features=2500, # Top vocabulary capacity
sublinear_tf=True, # Replace TF with 1 + log(TF) to dampen burstiness
stop_words='english' # Strip standard low-entropy stop-words
)
X_train_tfidf = tfidf.fit_transform(X_train_raw)
X_test_tfidf = tfidf.transform(X_test_raw)
print(f"Vocabulary Size: {len(tfidf.vocabulary_)} n-grams")
print(f"Feature Matrix Shape: {X_train_tfidf.shape} (Sparsity: "
f"{(1.0 - X_train_tfidf.nnz / (X_train_tfidf.shape[0] * X_train_tfidf.shape[1])) * 100:.2f}%)")
# -----------------------------------------------------------------
# 5. Model Benchmarking: Logistic Regression vs Multinomial Naive Bayes
# -----------------------------------------------------------------
print("\n--- [Phase 2] Classifier Model Benchmarking ---")
# Model 1: Logistic Regression (L2 Regularized)
clf_lr = LogisticRegression(C=1.0, max_iter=1000, random_state=42)
clf_lr.fit(X_train_tfidf, y_train)
y_pred_lr = clf_lr.predict(X_test_tfidf)
acc_lr = accuracy_score(y_test, y_pred_lr)
# Model 2: Multinomial Naive Bayes
clf_nb = MultinomialNB(alpha=0.5)
clf_nb.fit(X_train_tfidf, y_train)
y_pred_nb = clf_nb.predict(X_test_tfidf)
acc_nb = accuracy_score(y_test, y_pred_nb)
print(f"Logistic Regression Test Accuracy: {acc_lr * 100:.2f}%")
print(f"Multinomial Naive Bayes Test Accuracy: {acc_nb * 100:.2f}%")
print("\nLogistic Regression Classification Report:")
print(classification_report(y_test, y_pred_lr, target_names=['Negative', 'Positive']))
# -----------------------------------------------------------------
# 6. Feature Attribution (Top Positive & Negative Predictive Words)
# -----------------------------------------------------------------
print("--- [Phase 3] Top Predictive Lexical Signals (LR Coefficients) ---")
feature_names = np.array(tfidf.get_feature_names_out())
coefs = clf_lr.coef_[0]
# Top 5 Positive and Negative n-grams
top_pos_idx = np.argsort(coefs)[-5:][::-1]
top_neg_idx = np.argsort(coefs)[:5]
attribution_df = pd.DataFrame({
'Top_Positive_Signals': feature_names[top_pos_idx],
'Pos_Weight': np.round(coefs[top_pos_idx], 3),
'Top_Negative_Signals': feature_names[top_neg_idx],
'Neg_Weight': np.round(coefs[top_neg_idx], 3)
})
print(attribution_df.to_string(index=False))
# -----------------------------------------------------------------
# 7. Real-Time Inference on Unseen Test Prompts
# -----------------------------------------------------------------
print("\n--- [Phase 4] Real-Time Sample Inference ---")
sample_queries = [
"The battery life is exceptional and the display is gorgeous!",
"Customer service was rude, and the device overheated immediately."
]
cleaned_samples = [clean_text_corpus(s) for s in sample_queries]
sample_vectors = tfidf.transform(cleaned_samples)
predictions = clf_lr.predict(sample_vectors)
probabilities = clf_lr.predict_proba(sample_vectors)
for query, pred, prob in zip(sample_queries, predictions, probabilities):
label = "POSITIVE" if pred == 1 else "NEGATIVE"
conf = prob[pred] * 100
print(f"Review: \"{query}\"")
print(f"--> Classification: {label} ({conf:.1f}% confidence)\n")
if __name__ == '__main__':
execute_nlp_sentiment_benchmark()
>500,000 entries) and fail on rare or misspelled words, mapping them to uninformative <UNK> tokens. Character-level tokenizers avoid OOV words but produce sequence lengths 5--10× longer, hitting memory limits in quadratic self-attention (O(T^2)). Subword tokenization (BPE/WordPiece) finds an optimal balance: it represents frequent words as single tokens, while decomposing rare or novel words into meaningful subword morphemes (e.g., "un" + "expected" + "ly"). This eliminates OOV tokens while keeping vocabulary sizes compact (32,000--128,000).
"be", and identifies that "better" is an adjective and returns the base lemma "good", preserving accurate semantics.
IDF(t) = ln((1 + N / 1 + DF(t)) right) + 1, what is the mathematical purpose of adding 1 to the numerator, denominator, and the total log quotient?
1 to both the numerator and denominator ((1 + N / 1 + DF(t))) acts as Laplace smoothing: it prevents division by zero when a term does not appear in the training corpus (DF(t) = 0). Adding 1 to the final logarithm ensures that terms appearing in every single document (DF(t) = N, where ln(1) = 0) receive an IDF weight of 1.0 rather than 0.0, preventing them from being completely erased from the vector space before L_2 normalization.
P(w_O mid w_I) = ( exp(v'w_O^T vw_I) / Sum w=1V exp(v'_w^T vw_I))|V| = 500,000 words, evaluating the denominator sum requires computing 500,000 dot products for every single training word, which is computationally intractable (O(|V|)). Negative Sampling (SGNS) reformulates training as multiple binary logistic regressions: it maximizes the probability of the true target word while minimizing the probability of only k randomly sampled "negative" noise words (k in [5, 20]), reducing complexity from O(|V|) to O(k).
v in R^d to each word in the vocabulary. For polysemous words (e.g., "apple" the fruit vs. "Apple" the tech company; or river "bank" vs. financial "bank"), their multiple distinct meanings are forced into a single averaged vector. Transformers compute dynamic contextualized embeddings: each token's vector is updated through multi-head self-attention by taking weighted combinations of all other tokens in the sentence. The representation for "bank" dynamically shifts depending on surrounding context words (e.g., "river" vs. "interest rates").
w_t given only the preceding tokens: L = -Sum t=1^T log P(w_t mid w) . To train on all sequence tokens in parallel without letting the model "cheat" by looking at future tokens, decoder-only models apply a Lower-Triangular Causal Attention Mask. Positions where j > i are set to -infty prior to the Softmax step (softmax( (QK^T + M / √(d_k)) right)), setting attention weights to future tokens to zero.
y_w succ y_l), followed by complex reinforcement learning using Proximal Policy Optimization (PPO) to update the actor policy against the reward model with a KL divergence penalty to avoid policy collapse. Direct Preference Optimization (DPO) mathematically derives a closed-form substitution that expresses the reward function directly in terms of the optimal policy:
LDPO = -E [ log σ ( β log (π_θ(y_w mid x) / πref(y_w mid x)) - β log (π_θ(y_l mid x) / πref(y_l mid x)) right) right]