Linux Operating System & The Command Line

Introduction to Linux & Kernel Architecture

Virtually every production backend service, cloud virtual machine, container (Docker), and database cluster in modern enterprise computing executes on the Linux operating system. Developed by Linus Torvalds in 1991 following the POSIX and Unix design philosophies, Linux is a preemptive, multi-user, multitasking monolithic kernel.

User Space versus Kernel Space

Modern CPU architectures enforce hardware privilege levels (known as rings in x86-64). Linux exploits this boundary to protect physical hardware and memory integrity:

• Ring 0 (Kernel Space): Unrestricted execution privileges. The Linux kernel has direct access to CPU execution pipelines, physical RAM paging tables, hardware registers, device drivers, and network interface cards. Kernel space code executes with zero CPU privilege barriers.
• Ring 3 (User Space): Restricted execution privileges. All user applications—including your Node.js processes, Python workers, Nginx reverse proxies, and bash shells—execute in User Space. They are strictly barred from directly addressing physical memory or disk hardware.

The System Call (Syscall) Interface

When a backend application reads a file from disk (fs.readFile), binds an HTTP port (listen(8080)), or allocates heap memory, it cannot perform the action directly. Instead, it must trigger a System Call (e.g., read(), write(), epoll_wait(), mmap()). The CPU executes a context switch from User Space to Kernel Space, the kernel verifies the process permissions, executes the hardware I/O, and transitions back to User Space with the return value.

Linux Distributions Taxonomy

A Linux distribution combines the Linux kernel with GNU system utilities, package managers, and default system services:

• Debian / Ubuntu: Industry standard for cloud VPS (AWS EC2, GCP Compute Engine). Uses apt and dpkg package management. Exceptional stability and LTS (Long-Term Support) lifecycles.
• RHEL / Rocky / Fedora: Enterprise corporate standard. Uses dnf / rpm with strict SELinux security policies enforced by default.
• Alpine Linux: Ultra-lean distribution built around musl libc and BusyBox. Typically ~5MB base size, making it the primary base image choice for production Docker microservices.

Filesystem Hierarchy Standard (FHS) & Inode Mechanics

In Linux, there are no drive letters (like Windows C:\ or D:\). The entire storage landscape is unified into a single hierarchical tree beginning at the root directory: /.

The Filesystem Hierarchy Standard (FHS)

Inodes: How Linux Tracks Files

In Linux, the filename is merely a human-readable label pointing to an underlying data structure called an Inode (Index Node). The inode contains all metadata about the file:

• File type (regular file, directory, socket, named pipe).
• File size in bytes.
• Owner UID and Group GID.
• Permission bits (rwx).
• Timestamps: Access time (atime), Modification time (mtime), and Inode Change time (ctime).
• Direct and indirect pointers to the physical disk sectors holding the actual raw bytes.

Essential Shell Commands & Stream Redirection

The shell is a command language interpreter that executes commands read from standard input devices or from files. Mastery of command pipelines allows backend engineers to diagnose production servers in seconds without deploying secondary telemetry agents.

The Unix Stream Trinity: stdin, stdout, stderr

Every process spawned by the Linux kernel automatically inherits three open file descriptors:

1. Standard Input (stdin, Descriptor 0): The input stream (defaults to terminal keyboard input).
2. Standard Output (stdout, Descriptor 1): The default output stream for program results.
3. Standard Error (stderr, Descriptor 2): The unbuffered diagnostic and error stream, decoupled from standard output.

# Stream Redirection Syntax Reference
# 1. Overwrite stdout to file
node dist/index.js > app.log

# 2. Append stdout to file without destroying existing contents
node dist/index.js >> app.log

# 3. Redirect stderr only to error.log
node dist/index.js 2> error.log

# 4. Redirect both stdout and stderr to the same consolidated logfile
node dist/index.js > combined.log 2>&1
# Modern Bash shorthand syntax:
node dist/index.js &> combined.log

# 5. Blackhole redirection: Discard output completely
python3 worker.py > /dev/null 2>&1

# 6. Pipeline: Connect stdout of Command A to stdin of Command B
cat /var/log/nginx/access.log | grep "500" | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10

Production Text Processing Toolkit

• grep: Searches files for regular expressions. Use grep -rn "ERROR" /var/log/ to recursively search line numbers.
• awk: A full Turing-complete pattern-scanning language. Commonly used to parse columnar log files: awk '{print $1, $7}' access.log extracts client IP and URL.
• sed: Stream editor used to filter and transform text in pipelines (e.g., sed -i 's/PORT=3000/PORT=8080/g' .env).
• tail -f: Follows growing log files in real-time, streaming new lines as they are appended by active server processes.

POSIX File Permissions, Octal Notation & Umask

Linux implements Discretionary Access Control (DAC). Every file and directory on disk is tagged with an ownership pairing (User Owner, Group Owner) and 9 fundamental permission bits grouped into three triads.

The Permission Triad & Binary Octal Calculation

Each triad governs access for: User (u), Group (g), and Others (o). The individual bits map directly to power-of-two values:

• Read (r): Binary 100 = 4. (Allows viewing file contents; for directories, allows reading filenames inside).
• Write (w): Binary 010 = 2. (Allows editing file contents; for directories, allows creating and deleting files).
• Execute (x): Binary 001 = 1. (Allows running binaries/scripts; for directories, allows entering via cd).

Special Permission Bits: SUID, SGID & Sticky Bit

• SUID (Set Owner User ID, Octal 4000): When an executable with SUID is run, it executes with the privileges of the file's owner, not the invoking user (e.g., /usr/bin/passwd runs as root to update /etc/shadow).
• SGID (Set Group ID, Octal 2000): Files created inside an SGID directory automatically inherit the directory's group owner, rather than the primary group of the creating user. Crucial for team-shared project directories.
• Sticky Bit (Octal 1000, t): Applied to shared write directories (such as /tmp). Prevents users from deleting or renaming files owned by other users, even if the directory is world-writable (drwxrwxrwt).

Users, Groups & Privilege Escalation (sudo)

Linux enforces multi-tenant identity security through unique numeric identifiers: UID (User ID) and GID (Group ID).

System Identity Databases

• /etc/passwd: World-readable database storing account metadata formatted as colon-separated fields:
  username:x:UID:GID:Comment:HomeDirectory:LoginShell
  (Notice the 'x' in the password field: passwords were historically stored here, but were extracted to prevent hash cracking).
• /etc/shadow: Restricted database (readable only by root) containing cryptographic hashes of passwords generated using modern key derivation functions (SHA-512, yescrypt) alongside expiration parameters.
• /etc/group: Defines local groups and lists their secondary user memberships.

The Principle of Least Privilege & Sudo

In cloud infrastructure, logging in directly as the root superuser (UID 0) is strictly prohibited. An accidental typo like rm -rf / will permanently destroy the OS without confirmation.

Engineers create unprivileged system users (e.g., deployer, shivan) and grant granular administrative authority via sudo (Superuser Do). The configuration file /etc/sudoers defines who can run what commands. It must only ever be edited using the visudo command, which performs syntax validation before saving, preventing locks caused by syntax errors.

Processes, POSIX Signals & Systemd Service Architecture

A process is an active instance of a running program in memory. The Linux kernel assigns each process a unique positive integer known as the PID (Process ID).

Process States in the Linux Scheduler

At any moment, a process exists in one of five core operational states:

• Running or Runnable (R): Currently executing on a CPU core or waiting in the run queue.
• Interruptible Sleep (S): Blocked waiting for an event (such as an incoming network socket connection or timer). Wakes up upon receiving signals.
• Uninterruptible Sleep (D): Blocked waiting directly for hardware I/O (typically disk access). Cannot be terminated even by kill -9 until the hardware responds.
• Stopped (T): Suspended by a job control signal (e.g., Ctrl+Z or SIGSTOP).
• Zombie (Z): A process that has terminated its execution via exit(), but whose exit status has not yet been read by its parent process via wait(). It occupies no memory, but holds a slot in the kernel PID table.

POSIX Inter-Process Communication Signals

Systemd: The Enterprise Init Daemon (PID 1)

When Linux boots, the kernel mounts the root filesystem and executes the first user-space process: systemd (PID 1). Systemd initializes hardware, mounts storage, manages network interfaces, and monitors background services (daemons).

[Unit]
Description=Production Node.js Backend API Service
Documentation=https://api.devascentacademy.me/docs
After=network.target postgresql.service redis.service
Wants=postgresql.service redis.service

[Service]
Type=simple
User=nodeapp
Group=nodeapp
WorkingDirectory=/var/www/backend-api
ExecStart=/usr/bin/node dist/index.js
Restart=always
RestartSec=5s

# Environment Configuration
Environment=NODE_ENV=production
EnvironmentFile=/etc/backend-api/production.env

# Security Hardening Directives
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/backend-api

# Resource Limits (Cgroups)
LimitNOFILE=65535
MemoryMax=2G

[Install]
WantedBy=multi-user.target

# Reload systemd manager configuration after creating or modifying unit file
sudo systemctl daemon-reload

# Enable service to boot automatically on OS start
sudo systemctl enable backend-api.service

# Start the service immediately
sudo systemctl start backend-api.service

# Inspect live status, uptime, and recent logs
sudo systemctl status backend-api.service

# Gracefully restart the service
sudo systemctl restart backend-api.service

Package Managers & Automated Provisioning

Linux does not use haphazard .exe downloaders. Software is installed from cryptographically verified, centralized package repositories.

Debian/Ubuntu Advanced Package Tool (APT)

• apt update: Downloads the newest list of package versions and checksums from remote repositories defined in /etc/apt/sources.list. It does not upgrade software.
• apt upgrade -y: Installs newest versions of all currently installed packages.
• apt install -y <package>: Resolves dependencies, verifies GPG signatures, downloads Debian archives (.deb), and unpacks them into system paths.
• apt autoremove -y: Prunes orphan dependencies that are no longer required by any installed application.

Secure Shell (SSH) & Cryptographic Server Hardening

Production backend servers run headless in remote cloud data centers without physical monitors or keyboards. OpenSSH (Secure Shell) is the encrypted network protocol used to manage remote operating systems.

Public-Key Asymmetric Cryptography

Passwords are fundamentally vulnerable to brute-force dictionary attacks and credential stuffing. SSH authentication relies on asymmetric cryptographic key pairs:

1. Private Key (~/.ssh/id_ed25519): Kept securely on the client workstation. Never transmitted across the network. Guarded by permissions 600.
2. Public Key (~/.ssh/id_ed25519.pub): Placed on the remote server inside the user's ~/.ssh/authorized_keys file.

# Enterprise Production SSH Hardening Policy
Port 2222                          # Evade automated generic port 22 bot scanners
PermitRootLogin no                 # Strictly disable direct root logins
PasswordAuthentication no          # Enforce cryptographic key pairs only
PubkeyAuthentication yes
MaxAuthTries 3                     # Drop connection after 3 failed attempts
X11Forwarding no                   # Disable graphical GUI forwarding
AllowUsers deployer shivan         # Whitelist specific administrative accounts

# 1. Generate modern high-security Ed25519 key pair
ssh-keygen -t ed25519 -C "shivan@devascentacademy.me"

# 2. Copy public key to remote production server
ssh-copy-id -i ~/.ssh/id_ed25519.pub -p 2222 deployer@203.0.113.50

# 3. Synchronize build artifacts using rsync over encrypted SSH tunnel
rsync -avzP -e "ssh -p 2222" ./dist/ deployer@203.0.113.50:/var/www/backend-api/dist/

Bash Shell Scripting & Defensive Automation

Repeated administrative workflows must be codified into deterministic scripts. A production Bash script must be written defensively: if an intermediate step fails, execution must halt immediately rather than silently proceeding to corrupt data.

The Unofficial Bash Strict Mode

Every production bash script must declare: set -euo pipefail:

• set -e: Exit immediately if any command exits with a non-zero exit status.
• set -u: Treat unset or uninitialized variables as errors and exit immediately.
• set -o pipefail: Ensures the return value of a pipeline is the status of the last command to exit with a non-zero status (preventing masked failures inside pipelines).

#!/usr/bin/env bash

# Enforce Strict Defensive Execution
set -euo pipefail

# Script Environment Variables
readonly APP_DIR="/var/www/backend-api"
readonly BACKUP_DIR="/var/backups/api"
readonly TIMESTAMP=$(date +"%Y%m%d_%H%M%S")

# Log Helper Function
log() {
  echo "[$(date +'%Y-%m-%dT%H:%M:%S%z')] [DEPLOY] $1"
}

# Graceful Error Cleanup Trap
cleanup_on_error() {
  log "❌ DEPLOYMENT FAILED! Initiating rollback..."
  sudo systemctl restart backend-api.service || true
}
trap cleanup_on_error ERR

log "Step 1: Creating snapshot backup of current build..."
mkdir -p "${BACKUP_DIR}"
tar -czf "${BACKUP_DIR}/release_${TIMESTAMP}.tar.gz" -C "${APP_DIR}" dist/

log "Step 2: Pulling latest Git commits..."
cd "${APP_DIR}"
git fetch --all --prune
git reset --hard origin/main

log "Step 3: Deterministic dependency installation & compilation..."
npm ci --production=false
npm run build
npm prune --production

log "Step 4: Executing database migrations..."
npm run db:migrate:deploy

log "Step 5: Zero-downtime service reload..."
sudo systemctl reload-or-restart backend-api.service

log "✅ DEPLOYMENT SUCCESSFULLY COMPLETED."

Server Logs, Journald & Logrotate Architecture

Linux centralizes logging via two main mechanisms: legacy plain-text files located under /var/log/, and the binary structured journal managed by systemd-journald.

Systemd Journal Administration with Journalctl

The journalctl command queries the systemd journal database directly. Because the journal indexes metadata, logs can be queried with extreme speed:

• journalctl -u backend-api.service -f: Follow live output of the backend service in real time.
• journalctl -u backend-api.service --since "15 minutes ago": Filter logs by time window.
• journalctl -p err -b: Display all system errors that have occurred since the current boot.

Disk Exhaustion Prevention with Logrotate

A high-traffic service emitting 50MB of logs daily will saturate server disk capacity within weeks. The logrotate utility automates log compression, truncation, and historical deletion.

/var/log/backend-api/*.log {
    daily                      # Rotate logs every 24 hours
    missingok                  # Do not throw error if file is missing
    rotate 14                  # Keep 14 days of historical logs, delete older
    compress                   # Gzip compress rotated archives
    delaycompress              # Delay compression until second rotation cycle
    notifempty                 # Do not rotate if log file is 0 bytes
    create 0640 nodeapp nodeapp# Create new empty log with secure permissions
    sharedscripts
    postrotate
        systemctl kill -s HUP backend-api.service
    endscript
}

University & Industrial Examination Review