System Administration Handbook

Linux Command Reference

The Complete Ubuntu System Administration Cheat Sheet & Reference Guide

Primary Platform: Ubuntu 24.04 LTS
Cross-referenced with Debian, RHEL, Rocky Linux, AlmaLinux & Fedora
For System Administrators · DevOps Engineers · RHCSA / LFCS Candidates

Chapter 1Linux Filesystem Hierarchy

Every Linux distribution follows the Filesystem Hierarchy Standard (FHS), a specification that defines where different types of files belong. Understanding this layout is the foundation of system administration — it tells you where configuration lives, where binaries are installed, where logs are written, and where user data should go. Ubuntu, Debian, RHEL, and Fedora all follow the FHS closely, with only minor distribution-specific differences noted below.
/ ├── bin → sbin, usr/bin (symlink on modern systems) ├── sbin → usr/sbin (symlink on modern systems) ├── boot → kernel, initramfs, GRUB files ├── dev → device files ├── etc → system-wide configuration ├── home → user home directories ├── lib → usr/lib (symlink on modern systems) ├── lib64 → usr/lib64 (symlink on modern systems) ├── media → auto-mounted removable media ├── mnt → temporary manual mount point ├── opt → optional/third-party software ├── proc → virtual filesystem, kernel & process info ├── root → home directory of the root user ├── run → runtime data since last boot ├── srv → data for services (web, ftp) ├── sys → virtual filesystem, kernel/device info ├── tmp → temporary files (cleared on reboot) ├── usr → user-space programs & data │ ├── bin → user command binaries │ ├── sbin → system admin binaries │ ├── lib → libraries for usr/bin and usr/sbin │ ├── local → locally compiled/installed software │ └── share → architecture-independent data (docs, icons, man pages) └── var → variable data (logs, mail, spool, cache)

Top-Level Directory Reference

DirectoryPurposeTypical Contents / Examples
/binEssential user command binaries needed in single-user mode/bin/ls, /bin/cp, /bin/bash
/sbinEssential system binaries, mostly for root/sbin/fdisk, /sbin/reboot
/bootFiles required to boot the systemvmlinuz, initrd.img, grub/
/devDevice nodes representing hardware and virtual devices/dev/sda, /dev/null, /dev/tty1
/etcSystem-wide configuration files (host-specific, static)/etc/passwd, /etc/hosts, /etc/fstab, /etc/ssh/
/homePersonal directories for regular users/home/ahmad, /home/deploy
/lib, /lib64Shared libraries needed by binaries in /bin and /sbinlibc.so, kernel modules
/mediaMount points for removable media, auto-created by the desktop/udisks/media/ahmad/USB_DRIVE
/mntConventional location for temporarily mounting a filesystem by hand/mnt/backup, /mnt/data
/optAdd-on software packages not part of the default OS install/opt/google/chrome, third-party agents
/procVirtual filesystem exposing kernel and process information in real time/proc/cpuinfo, /proc/meminfo, /proc/<pid>/
/rootHome directory for the root superuser (not inside /home for recovery reasons)/root/.bashrc
/runRuntime data since the last boot (replaces old /var/run)PID files, sockets
/srvData served by this system for specific services/srv/www, /srv/ftp
/sysVirtual filesystem exposing kernel/device/driver information (sysfs)/sys/class/net
/tmpTemporary files created by applications; usually cleared on rebootlock files, temp downloads
/usrThe bulk of installed software and read-only user data ("Unix System Resources")/usr/bin, /usr/share
/varVariable data that changes frequently during operation/var/log, /var/spool, /var/cache

Closer Look: /etc

Purpose
Contains all system-wide configuration files. Nothing executable lives here by convention — only text-based config that applications and the system read at startup or runtime.

Key Examples

FileRole
/etc/passwdUser account database (username, UID, GID, home, shell)
/etc/shadowEncrypted password hashes and aging policy, readable only by root
/etc/hostsStatic hostname-to-IP mappings, checked before DNS
/etc/fstabDefines filesystems to mount automatically at boot
/etc/ssh/SSH daemon and client configuration (sshd_config, ssh_config)
On modern Ubuntu (and most current distros), /bin, /sbin, /lib, and /lib64 are symlinks into their /usr equivalents (the "usrmerge"). Running ls -l /bin will show it pointing to usr/bin. Both paths work identically.
RHEL, Rocky Linux, AlmaLinux, and Fedora follow the exact same FHS layout. The main practical difference administrators notice is package-manager-specific paths (e.g. RPM database in /var/lib/rpm vs. dpkg's in /var/lib/dpkg), covered in Chapter 6.
↑ Back to top

Chapter 2Navigation Commands

Before manipulating files, an administrator must be able to move confidently around the filesystem, locate files quickly, and understand exactly where a command or script lives. This chapter covers the core navigation toolkit used dozens of times per day.

pwd — Print Working Directory

Purpose Displays the absolute path of the current directory.

Syntax pwd [OPTION]

Explanation Useful for confirming your location inside deep directory trees, especially inside scripts where the working directory isn't obvious.

$ pwd
/home/ahmad/projects/expovision

Common Options

-LPrint the logical path, following symlinks as named (default)
-PPrint the physical path, resolving all symlinks

ls — List Directory Contents

Purpose Lists files and directories.

Syntax ls [OPTION]... [FILE]...

Explanation The single most-used command in Linux. Combine flags to control detail, ordering, and visibility of hidden files.

$ ls -lah /var/log
total 3.2M
drwxr-xr-x  12 root root 4.0K Jul  9 22:14 .
drwxr-xr-x  14 root root 4.0K Jun 20 09:03 ..
-rw-r-----   1 syslog adm  842K Jul 10 08:00 syslog
-lLong listing (permissions, owner, size, date)
-aShow hidden files (dotfiles)
-hHuman-readable sizes (K, M, G)
-RRecursive listing
-tSort by modification time, newest first
-SSort by file size, largest first
-dList directories themselves, not their contents

cd — Change Directory

Purpose Moves the shell's current working directory.

Syntax cd [DIRECTORY]

$ cd /etc/ssh
$ cd ..              # one level up
$ cd ~               # home directory
$ cd -                # previous directory
cd - is a fast way to toggle between two directories you're actively working in.

tree — Directory Tree Viewer

Purpose Displays a directory's contents as an indented tree.

Syntax tree [OPTION] [DIRECTORY]

Explanation Not installed by default on Ubuntu — install with sudo apt install tree. Excellent for documentation and quickly understanding a project layout.

$ tree -L 2 /etc/nginx
/etc/nginx
├── nginx.conf
├── sites-available
└── sites-enabled
-L nLimit recursion depth to n levels
-aInclude hidden files
-dList directories only

find — Search for Files

Purpose Recursively searches a directory tree by name, type, size, age, permissions, and more.

Syntax find [PATH] [EXPRESSION]

$ find /var/log -name "*.log" -mtime -1
$ find / -type f -size +100M 2>/dev/null
$ find /home/ahmad -perm 777 -type f
$ find /tmp -mtime +7 -exec rm {} \;
-nameMatch by filename pattern (case-sensitive)
-inameCase-insensitive name match
-type f/d/lFile / directory / symlink
-mtime -n / +nModified less than / more than n days ago
-size +n / -nLarger / smaller than n (use c, k, M, G suffix)
-exec CMD {} \;Run a command on each match
Redirecting stderr with 2>/dev/null is standard practice when running find /, since it will otherwise flood the terminal with "Permission denied" messages from directories you can't read.

locate — Fast Filename Search

Purpose Searches a pre-built database for filenames instantly, much faster than find.

Syntax locate [PATTERN]

Explanation Requires mlocate (sudo apt install mlocate) and its database is updated nightly via cron, or manually.

$ sudo updatedb
$ locate sshd_config
/etc/ssh/sshd_config
Because the database is only refreshed periodically, locate will not find files created moments ago until updatedb runs again.

which — Locate a Command's Executable

Purpose Shows the full path of the executable that would run for a given command name.

Syntax which [COMMAND]

$ which python3
/usr/bin/python3

whereis — Locate Binary, Source, and Man Pages

Purpose Similar to which, but also finds the man page and source, if present.

Syntax whereis [COMMAND]

$ whereis nginx
nginx: /usr/sbin/nginx /etc/nginx /usr/share/man/man8/nginx.8.gz

realpath — Resolve Absolute Path

Purpose Converts a relative path or symlink into its canonical absolute path.

Syntax realpath [FILE]

$ realpath ../scripts/backup.sh
/home/ahmad/scripts/backup.sh

basename / dirname — Split a Path

Purpose basename extracts the filename component; dirname extracts the directory component. Frequently used inside shell scripts.

Syntax basename PATH · dirname PATH

$ basename /etc/nginx/nginx.conf
nginx.conf
$ dirname /etc/nginx/nginx.conf
/etc/nginx
↑ Back to top

Chapter 3File Management

File management commands are used constantly for creating, moving, inspecting, and viewing files. This chapter groups them by task: creation, copying/moving, viewing, and text-stream utilities.

Creating and Removing Files & Directories

touch — Create Empty File / Update Timestamp

Purpose Creates an empty file if it doesn't exist, or updates its modification timestamp if it does.

Syntax touch [OPTION] FILE

$ touch notes.txt
$ touch -d "2026-01-01" archive.log

mkdir — Make Directory

Purpose Creates one or more directories.

Syntax mkdir [OPTION] DIRECTORY

$ mkdir project
$ mkdir -p project/src/{bin,lib,docs}   # nested creation
-pCreate parent directories as needed, no error if they exist
-m MODESet permissions at creation time

rmdir — Remove Empty Directory

Purpose Deletes a directory only if it is empty.

Syntax rmdir DIRECTORY

$ rmdir old_folder

rm — Remove Files/Directories

Purpose Deletes files or, with -r, whole directory trees.

Syntax rm [OPTION] FILE...

$ rm file.txt
$ rm -rf /tmp/build_cache/
-rRecursive (needed for directories)
-fForce, ignore nonexistent files, no prompt
-iPrompt before every removal
There is no Recycle Bin. rm -rf is permanent and irreversible. Always double-check the path — especially with wildcards — before pressing Enter, and never run it as root against a path built from an unchecked variable in a script.

Copying, Moving & Linking

cp — Copy Files/Directories

Purpose Copies files or directories.

Syntax cp [OPTION] SOURCE DEST

$ cp report.docx ~/backup/
$ cp -r site/ site_backup/
$ cp -p config.yml config.yml.bak   # preserve mode/owner/timestamp
-rRecursive, required for directories
-pPreserve permissions, ownership, timestamps
-vVerbose output
-uCopy only when source is newer

mv — Move / Rename

Purpose Moves files/directories, or renames them if source and destination are in the same directory.

Syntax mv [OPTION] SOURCE DEST

$ mv draft.txt final.txt
$ mv *.log /var/log/archive/

ln — Create Links

Purpose Creates hard or symbolic links between files.

Syntax ln [OPTION] TARGET LINK_NAME

$ ln -s /etc/nginx/sites-available/mysite /etc/nginx/sites-enabled/mysite
$ ln /data/original.txt /data/hardlink.txt
-sCreate a symbolic (soft) link instead of a hard link
A symlink is a pointer to a path and can cross filesystems; a hard link is a second name for the same inode and must stay on the same filesystem. Deleting the original breaks a symlink but not a hard link.

Inspecting Files

stat — Detailed File Status

Purpose Shows inode number, size, permissions, and precise timestamps.

Syntax stat FILE

$ stat /etc/passwd
  File: /etc/passwd
  Size: 2847      Blocks: 8   IO Block: 4096 regular file
Access: (0644/-rw-r--r--)  Uid: (0/root)  Gid: (0/root)

file — Determine File Type

Purpose Identifies a file's type by inspecting its content (magic bytes), not just its extension.

Syntax file FILE

$ file backup.tar.gz
backup.tar.gz: gzip compressed data

Viewing File Contents

cat — Concatenate and Print

Purpose Prints entire file contents to the terminal, or concatenates multiple files.

Syntax cat [OPTION] FILE...

$ cat /etc/hostname
$ cat file1.txt file2.txt > combined.txt
-nNumber all output lines
-AShow non-printing characters (tabs, line ends)

tac — Reverse cat

Purpose Prints a file with lines in reverse order (last line first). Useful for reading recent log entries first.

Syntax tac FILE

$ tac /var/log/syslog | head -20

less / more — Paginated Viewing

Purpose View large files one screen at a time without loading the whole file into the terminal buffer.

Syntax less FILE · more FILE

Explanation less is the modern standard: it supports backward scrolling, searching with /pattern, and doesn't need to read the whole file first. more is older and forward-only, but always available on minimal systems.

$ less /var/log/syslog
# inside less: / to search, n for next match, q to quit

head / tail — Show Start/End of a File

Purpose Print the first (head) or last (tail) lines of a file.

Syntax head [-n N] FILE · tail [-n N] FILE

$ head -n 20 access.log
$ tail -f /var/log/syslog       # follow in real time
-n NShow N lines instead of the default 10
-fFollow the file as it grows (tail only)

nl — Number Lines

Purpose Prints a file with line numbers prepended.

Syntax nl FILE

$ nl script.sh | head -5

Text-Stream Utilities

wc — Word, Line, Byte Count

Purpose Counts lines, words, and bytes/characters.

Syntax wc [OPTION] FILE

$ wc -l /etc/passwd
42 /etc/passwd
-lLines only
-wWords only
-cBytes only

sort — Sort Lines

Purpose Sorts lines of text alphabetically or numerically.

Syntax sort [OPTION] FILE

$ sort names.txt
$ sort -n -r sizes.txt   # numeric, descending
-nNumeric sort
-rReverse order
-uRemove duplicate lines while sorting
-k NSort by field/column N

uniq — Filter Duplicate Lines

Purpose Removes or counts adjacent duplicate lines — input should typically be sorted first.

Syntax uniq [OPTION] FILE

$ sort access.log | uniq -c | sort -nr | head
-cPrefix lines with occurrence count
-dOnly show duplicated lines

cut — Extract Columns

Purpose Extracts fields/columns from delimited text.

Syntax cut -d DELIM -f FIELDS FILE

$ cut -d: -f1 /etc/passwd    # list all usernames

paste — Merge Lines of Files

Purpose Joins corresponding lines from multiple files side by side.

Syntax paste FILE1 FILE2

$ paste names.txt scores.txt

split — Split a File into Pieces

Purpose Breaks a large file into smaller chunks, useful before transferring over size-limited channels.

Syntax split [OPTION] FILE PREFIX

$ split -b 100M bigfile.tar part_

tee — Split Output to File and Screen

Purpose Writes command output to a file while simultaneously printing it to the terminal.

Syntax COMMAND | tee FILE

$ sudo apt update | tee update.log
$ echo "127.0.0.1 test.local" | sudo tee -a /etc/hosts
-aAppend instead of overwrite

shred — Securely Delete a File

Purpose Overwrites a file's contents multiple times before deletion to make recovery difficult.

Syntax shred [OPTION] FILE

$ shred -u -n 3 secrets.txt
-uRemove (unlink) the file after shredding
-n NNumber of overwrite passes
shred is not fully reliable on SSDs or journaling/copy-on-write filesystems (like most modern setups) due to wear-leveling and block remapping. For SSDs, use full-disk encryption from the start, or the drive's built-in secure-erase feature.

truncate — Resize a File

Purpose Shrinks or extends a file to an exact size, commonly used to instantly empty a log file.

Syntax truncate -s SIZE FILE

$ truncate -s 0 /var/log/huge.log   # empty a log without deleting it

rename — Batch Rename Files

Purpose Renames multiple files using a Perl-style regular expression.

Syntax rename 's/PATTERN/REPLACEMENT/' FILE...

$ rename 's/\.jpeg$/\.jpg/' *.jpeg
Ubuntu's rename (from the rename package, sudo apt install rename) uses Perl regex syntax. Some distributions ship a different rename with simpler rename OLD NEW FILE syntax — check man rename on unfamiliar systems.
↑ Back to top

Chapter 4File Permissions

Linux permissions control who can read, write, or execute a file, evaluated against three categories of user: the owner, the group, and others (everyone else). Mastering permissions is essential for security and for diagnosing "Permission denied" errors, one of the most common troubleshooting scenarios in Chapter 22.

The Permission Model

CategoryMeaning
Owner (u)The user who owns the file, usually its creator
Group (g)Users belonging to the file's associated group
Others (o)Every other user on the system

Permission Types

SymbolMeaning on a FileMeaning on a Directory
rRead file contentsList directory contents
wModify file contentsCreate/delete/rename files inside
xExecute the file as a program/scriptEnter the directory (cd into it)

Numeric (Octal) Permissions

Each permission has a value: r=4, w=2, x=1. Add them per category to get a three-digit mode.

ModeMeaningTypical Use
777rwxrwxrwx — everyone can read, write, executeAlmost never appropriate; a major security risk
755rwxr-xr-x — owner full control, others read/executeExecutables, scripts, directories
644rw-r--r-- — owner read/write, others read-onlyStandard config and data files
600rw------- — owner only, no access for anyone elseSSH private keys, credential files
400r-------- — owner read-only, no writeSensitive files that shouldn't be modified

chmod — Change Permissions

Purpose Modifies read/write/execute permissions on files or directories.

Syntax chmod [OPTION] MODE FILE

$ chmod 600 ~/.ssh/id_rsa
$ chmod +x deploy.sh
$ chmod -R 755 /var/www/html
$ chmod g+w shared_folder/    # add write for group
-RRecursive, apply to all files/subdirectories
u/g/o/aTarget user, group, others, or all (symbolic mode)
+/-/=Add, remove, or set exactly (symbolic mode)

chown — Change Owner

Purpose Changes the user (and optionally group) that owns a file.

Syntax chown [OPTION] USER[:GROUP] FILE

$ sudo chown ahmad:ahmad report.pdf
$ sudo chown -R www-data:www-data /var/www/html

chgrp — Change Group

Purpose Changes only the group associated with a file.

Syntax chgrp [OPTION] GROUP FILE

$ sudo chgrp developers project/ -R

umask — Default Permission Mask

Purpose Sets the default permissions subtracted from new files and directories at creation time.

Syntax umask [MODE]

Explanation The default Ubuntu umask is 022, meaning new files get 644 and new directories get 755 by default (base 666/777 minus the mask).

$ umask
0022
$ umask 027   # stricter: group loses write, others get nothing

Access Control Lists (ACLs)

Standard Unix permissions only support one owner and one group. When you need finer-grained control — e.g. "user X gets read-write, but the rest of the group only reads" — use ACLs.

getfacl — View ACLs

Purpose Displays the access control list of a file or directory.

Syntax getfacl FILE

$ getfacl /srv/shared/report.docx
# file: report.docx
# owner: ahmad
# group: staff
user::rw-
user:hana:r--
group::r--
mask::rw-
other::---

setfacl — Set ACLs

Purpose Grants or revokes fine-grained permissions for specific users or groups beyond the standard owner/group/other model.

Syntax setfacl [OPTION] RULE FILE

$ setfacl -m u:hana:rw report.docx
$ setfacl -R -m g:developers:rwx /srv/project
$ setfacl -x u:hana report.docx    # remove the rule
-mModify (add/update) an ACL entry
-xRemove an ACL entry
-RApply recursively
-bRemove all ACL entries
The filesystem must be mounted with ACL support (default on ext4 and XFS in Ubuntu 24.04 and current RHEL-family distros). Verify with mount | grep acl if setfacl reports "Operation not supported."
↑ Back to top

Chapter 5Users & Groups

Every process on Linux runs as a user, and every user belongs to one or more groups. Managing accounts correctly — creation, password policy, group membership, and privilege escalation via sudo — is core system administration.

Core Concepts

ConceptDescription
UIDNumeric User ID. 0 = root. System accounts typically use 1–999 (or 1–99 on RHEL-family); regular users start at 1000 (Ubuntu) or 1000 (RHEL 8+).
GIDNumeric Group ID, identifying a user's primary group.
/etc/passwdWorld-readable account database: username, UID, GID, home directory, login shell.
/etc/shadowRoot-only file holding encrypted password hashes and password-aging rules.
/etc/groupDefines groups and their member lists.
$ cat /etc/passwd | grep ahmad
ahmad:x:1000:1000:Ahmad:/home/ahmad:/bin/bash
# username : x(password in shadow) : UID : GID : comment : home : shell

Identity Commands

whoami — Current Username

Purpose Prints the effective username of the current session.

$ whoami
ahmad

id — UID, GID, and Group Memberships

Purpose Displays a user's UID, primary GID, and all supplementary group memberships.

Syntax id [USERNAME]

$ id ahmad
uid=1000(ahmad) gid=1000(ahmad) groups=1000(ahmad),27(sudo),999(docker)

groups — List Group Memberships

Syntax groups [USERNAME]

$ groups ahmad
ahmad : ahmad sudo docker

Managing Users

useradd — Create a User (low-level)

Purpose Creates a new user account. On Ubuntu it is the low-level tool — it does not create a home directory or set a password by default unless flagged.

Syntax useradd [OPTION] USERNAME

$ sudo useradd -m -s /bin/bash -G sudo ahmad
$ sudo passwd ahmad
-mCreate the home directory
-s SHELLSet the login shell
-G GROUPSComma-separated supplementary groups
-u UIDSpecify a custom UID

adduser — Create a User (interactive, Debian/Ubuntu)

Purpose A friendlier, interactive Perl script wrapping useradd — prompts for password and details, creates the home directory automatically.

Syntax adduser USERNAME

$ sudo adduser hana
adduser is Debian/Ubuntu-specific and not present by default on RHEL, Rocky, AlmaLinux, or Fedora, where useradd is the standard tool.

usermod — Modify a User Account

Purpose Changes an existing account's properties: shell, groups, home directory, lock status.

Syntax usermod [OPTION] USERNAME

$ sudo usermod -aG docker ahmad
$ sudo usermod -s /bin/zsh ahmad
$ sudo usermod -L ahmad    # lock the account
-aG GROUPAppend to a supplementary group (always use -a with -G!)
-s SHELLChange login shell
-L / -ULock / unlock the account password
Using -G without -a replaces all supplementary group memberships instead of adding to them — a common mistake that silently removes a user from groups like sudo or docker.

passwd — Set/Change Password

Syntax passwd [USERNAME]

$ sudo passwd ahmad
$ passwd            # change your own password
$ sudo passwd -l ahmad   # lock
$ sudo passwd -e ahmad   # force change at next login

userdel — Delete a User

Syntax userdel [OPTION] USERNAME

$ sudo userdel -r ahmad   # -r also removes home dir and mail spool

Managing Groups

groupadd / groupdel / groupmod

$ sudo groupadd developers
$ sudo groupmod -n devs developers   # rename group
$ sudo groupdel devs

Privilege Escalation

su — Switch User

Purpose Starts a shell as another user, prompting for that user's password.

Syntax su [-] [USERNAME]

$ su - postgres   # the dash loads the target user's full environment

sudo — Execute as Another User (usually root)

Purpose Runs a single command with elevated privileges, authenticated with your own password, and logged for audit purposes.

Syntax sudo COMMAND

$ sudo systemctl restart nginx
$ sudo -u www-data php artisan migrate
$ sudo -i    # interactive root shell

visudo — Safely Edit sudoers

Purpose Opens /etc/sudoers in a syntax-checked editor session, preventing a malformed file from locking out all sudo access.

Syntax sudo visudo

ahmad ALL=(ALL:ALL) ALL
%developers ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart nginx
Never edit /etc/sudoers directly with a plain text editor. Always use visudo — it validates syntax before saving and prevents a broken file that could lock every admin out of sudo.

Session & Login Information

last — Login History

Purpose Shows a log of recent logins, reboots, and logouts from /var/log/wtmp.

$ last -n 10

who / w — Currently Logged-In Users

Purpose who lists logged-in users and their terminals; w additionally shows what each is currently running and system load.

$ who
$ w

users — Simple List of Logged-In Users

$ users
ahmad hana
↑ Back to top

Chapter 6Package Management

Ubuntu (and Debian) use the APT ecosystem built on top of dpkg. Ubuntu also supports Snap, a containerized universal package format maintained by Canonical. RHEL-family distributions instead use dnf/yum over rpm — the concepts map directly even though the commands differ, as shown in the comparison table at the end of this chapter.

apt — High-Level Package Manager

Purpose The modern, user-friendly front end for installing, updating, and removing software on Debian/Ubuntu.

Syntax apt COMMAND [PACKAGE]

$ sudo apt update              # refresh package index from repositories
$ sudo apt upgrade              # upgrade all installed packages
$ sudo apt install nginx        # install a package
$ sudo apt remove nginx         # remove package, keep config
$ sudo apt purge nginx          # remove package AND config files
$ sudo apt autoremove           # remove orphaned dependencies
$ sudo apt search docker        # search available packages
$ sudo apt show nginx           # package details
$ sudo apt list --installed     # everything currently installed
SubcommandWhat It Does
updateDownloads the latest package index metadata from configured repositories. Does not install anything — run this before upgrade or install to see the newest available versions.
upgradeUpgrades all installed packages to the latest available version without removing any existing packages.
dist-upgrade (or full-upgrade)Like upgrade, but will also add or remove packages to satisfy new dependencies — used for major point-release upgrades.
autoremoveRemoves packages that were installed automatically as dependencies but are no longer required by anything.
cleanDeletes downloaded .deb files from the local cache (/var/cache/apt/archives) to free disk space.
purgeRemoves a package's binaries and its configuration files, unlike plain remove which leaves config behind.

apt-get — Lower-Level Package Manager

Purpose The original, script-friendly APT front end. Functionally similar to apt but with more stable output formatting for automation, and fewer "nice" interactive features (like progress bars).

$ sudo apt-get update
$ sudo apt-get install -y nginx
$ sudo apt-get build-dep somepackage
Use apt interactively at the terminal; prefer apt-get/apt-cache inside scripts, since their output format is guaranteed stable across versions while apt's is explicitly not.

dpkg — Low-Level Debian Package Tool

Purpose Installs, queries, and manages individual .deb package files directly, without resolving dependencies from a repository.

Syntax dpkg [OPTION] PACKAGE.deb

$ sudo dpkg -i application.deb
$ dpkg -l | grep nginx
$ dpkg -L nginx          # list files installed by a package
$ sudo dpkg -r nginx      # remove
-iInstall a local .deb file
-lList installed packages
-LList files owned by a package
-rRemove a package
If dpkg -i fails with dependency errors, run sudo apt --fix-broken install immediately afterward to let APT pull in the missing dependencies.

snap — Universal Package Manager

Purpose Installs sandboxed, self-contained "snap" packages that bundle their own dependencies, with automatic background updates.

Syntax snap COMMAND PACKAGE

$ sudo snap install code --classic
$ snap list
$ sudo snap refresh
$ sudo snap remove code
Snap is an Ubuntu/Canonical-driven technology. It is not present by default on Debian, RHEL, Rocky, AlmaLinux, or Fedora, which favor Flatpak or native RPM packages instead.

Ubuntu (APT) vs RHEL-Family (DNF/YUM) Command Equivalents

TaskUbuntu / Debian (APT)RHEL / Rocky / Alma / Fedora (DNF)
Refresh package metadatasudo apt updatesudo dnf check-update
Upgrade all packagessudo apt upgradesudo dnf upgrade
Install a packagesudo apt install nginxsudo dnf install nginx
Remove a packagesudo apt remove nginxsudo dnf remove nginx
Search for a packageapt search nginxdnf search nginx
List installed packagesdpkg -lrpm -qa
Install a local package filedpkg -i file.debrpm -ivh file.rpm or dnf install ./file.rpm
Package format.deb.rpm
Older RHEL 7 and CentOS 7 systems use yum, the predecessor to dnf. The command syntax is nearly identical (yum install, yum update) since dnf was designed as a drop-in successor.
↑ Back to top

Chapter 7Processes

Every running program is a process with a Process ID (PID), a parent, a priority, and a state. This chapter covers viewing, prioritizing, backgrounding, and terminating processes — essential for diagnosing high CPU/RAM usage and unresponsive services.

ps — Snapshot of Running Processes

Purpose Lists currently running processes at the moment the command executes.

Syntax ps [OPTION]

$ ps aux | grep nginx
www-data  1234  0.0  0.3  55120  6200 ?  S  08:00  0:00 nginx: worker
$ ps -ef --forest    # tree view showing parent/child relationships
aShow processes for all users
uDisplay user-oriented, detailed format
xInclude processes without a controlling terminal
-eAll processes (equivalent to a)
-fFull format listing showing PPID

top — Real-Time Process Monitor

Purpose Interactive, continuously updating view of CPU and memory usage per process.

$ top
# inside top: 'k' to kill a PID, 'M' to sort by memory, 'P' by CPU, 'q' to quit

htop — Enhanced Interactive Process Viewer

Purpose A friendlier, color, mouse-capable alternative to top with per-core CPU bars and easy process tree navigation.

Explanation Not installed by default — install with sudo apt install htop.

$ htop

kill — Terminate a Process by PID

Purpose Sends a signal (default SIGTERM) to a process, typically to ask it to shut down gracefully.

Syntax kill [-SIGNAL] PID

$ kill 1234
$ kill -9 1234       # SIGKILL, forceful, cannot be ignored
$ kill -HUP 1234     # many daemons reload config on SIGHUP

killall — Kill by Process Name

Purpose Sends a signal to all processes matching a given name.

$ killall nginx

pkill — Kill by Pattern Match

Purpose Kills processes matching a name or other attribute pattern, more flexible than killall.

$ pkill -f "python3 worker.py"
$ pkill -u ahmad     # kill all processes owned by a user

nice / renice — Process Priority

Purpose nice starts a new process with an adjusted scheduling priority; renice changes the priority of an already-running process. Values range from -20 (highest priority) to 19 (lowest).

$ nice -n 10 ./backup_script.sh     # start at lower priority
$ sudo renice -5 -p 1234            # raise priority of a running PID

jobs / bg / fg — Shell Job Control

Purpose Manage processes started from the current shell session — pausing, resuming in the background, or bringing back to the foreground.

$ long_task.sh
^Z                    # Ctrl+Z suspends it
$ jobs
[1]+  Stopped   long_task.sh
$ bg %1               # resume in background
$ fg %1               # bring back to foreground

nohup — Run Immune to Hangups

Purpose Runs a command so that it keeps running after the terminal session closes (ignores SIGHUP), redirecting output to nohup.out by default.

Syntax nohup COMMAND &

$ nohup ./long_running_job.sh > job.log 2>&1 &
For anything more serious than a quick background task, prefer a proper systemd unit (Chapter 8) over nohup — it gives you restart policies, logging integration, and boot-time startup.

A Word on systemd

Modern Ubuntu (and RHEL 7+, Debian 8+, Fedora) all use systemd as PID 1 — the first process started by the kernel, and the parent of every other process on the system. systemd doesn't just manage services; it also handles device management (udev), logging (journald), mounting, and boot targets. The next chapter is dedicated entirely to systemctl, the primary tool for interacting with it.
↑ Back to top

Chapter 8Services (systemd)

systemctl is the primary command for controlling systemd units — services, sockets, timers, mounts, and targets. It is identical across Ubuntu, Debian, RHEL, Rocky, AlmaLinux, and Fedora, making it one of the most portable skills in this entire guide.

systemctl status — Check a Service's State

Purpose Shows whether a unit is active, its recent log lines, main PID, and memory usage.

Syntax systemctl status UNIT

$ systemctl status nginx
● nginx.service - A high performance web server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
     Active: active (running) since Fri 2026-07-10 08:00:12 UTC

systemctl start / stop / restart / reload

Purpose Controls the running state of a service immediately (does not affect boot-time behavior).

SubcommandEffect
startStarts the unit now if it is not running
stopStops the unit now
restartStops then starts the unit (brief downtime)
reloadAsks the service to reload its configuration without dropping connections (only if the unit supports it)
$ sudo systemctl restart nginx
$ sudo systemctl reload nginx     # zero-downtime config reload

systemctl enable / disable — Boot-Time Behavior

Purpose Configures whether a unit starts automatically at boot. Does not affect its current running state.

$ sudo systemctl enable nginx           # start on every boot
$ sudo systemctl enable --now nginx     # enable AND start immediately
$ sudo systemctl disable nginx
enable alone does not start the service now, and start alone does not survive a reboot. For a permanent, immediately active service, use enable --now.

systemctl mask / unmask

Purpose mask completely prevents a unit from being started, even manually or as a dependency — stronger than disable. Useful to stop a service someone keeps auto-restarting.

$ sudo systemctl mask apache2
$ sudo systemctl unmask apache2

systemctl daemon-reload

Purpose Reloads systemd's own configuration after you've edited or created a unit file. Required before your changes take effect.

$ sudo nano /etc/systemd/system/myapp.service
$ sudo systemctl daemon-reload
$ sudo systemctl restart myapp

Useful Status & Listing Commands

CommandPurpose
systemctl list-units --type=serviceList all loaded service units
systemctl list-unit-files --state=enabledList all units enabled at boot
systemctl is-active nginxPrint just "active" or "inactive"
systemctl is-enabled nginxPrint just "enabled" or "disabled"
systemctl --failedList units that failed to start

journalctl — systemd's Unified Log Viewer

Purpose Queries the systemd journal, which centralizes logs from services, the kernel, and boot messages.

Syntax journalctl [OPTION]

$ journalctl -u nginx              # logs for one unit
$ journalctl -u nginx -f           # follow live, like tail -f
$ journalctl -u nginx --since "1 hour ago"
$ journalctl -p err -b             # errors since last boot
$ journalctl --disk-usage          # how much space the journal uses
$ sudo journalctl --vacuum-time=7d # trim logs older than 7 days
-u UNITFilter to a specific systemd unit
-fFollow in real time
-bOnly show logs since the current boot
-p LEVELFilter by priority (emerg, alert, crit, err, warning, ...)
--since / --untilTime range filter
This entire chapter applies identically on Ubuntu, Debian, RHEL, Rocky Linux, AlmaLinux, and Fedora — systemd unified service management across virtually all mainstream distributions years ago.
↑ Back to top

Chapter 9Networking

Networking commands fall into a few categories: viewing and configuring interfaces, checking connectivity, resolving DNS, and transferring data securely. Ubuntu has moved from the legacy net-tools suite (ifconfig, netstat, arp) to the modern iproute2 suite (ip, ss) as the default toolset, though the older commands still work if installed.

ip — Modern Network Configuration Tool

Purpose Shows and configures interfaces, addresses, and routes. The modern replacement for ifconfig and route.

Syntax ip [OBJECT] [COMMAND]

$ ip addr show                       # list all interfaces & IPs
$ ip a                               # shorthand
$ sudo ip addr add 192.168.1.50/24 dev eth0
$ sudo ip link set eth0 up
$ ip route show                      # routing table

ifconfig — Legacy Interface Tool

Purpose The classic interface configuration tool, deprecated in favor of ip but still widely recognized.

$ ifconfig
$ sudo ifconfig eth0 up
Not installed by default on Ubuntu 24.04. Install via sudo apt install net-tools if needed for compatibility with older scripts.

ss — Socket Statistics

Purpose Shows open sockets, listening ports, and established connections. The modern replacement for netstat.

Syntax ss [OPTION]

$ ss -tulpn
tcp LISTEN 0 511 0.0.0.0:80 users:(("nginx",pid=1234,fd=6))
-tTCP sockets
-uUDP sockets
-lListening sockets only
-pShow owning process
-nNumeric (skip DNS resolution)

netstat — Legacy Network Statistics

Purpose Older equivalent of ss, still common in scripts and documentation.

$ netstat -tulpn
Part of the deprecated net-tools package; not installed by default on Ubuntu 24.04.

ping — Test Reachability

Purpose Sends ICMP echo requests to test whether a host is reachable and measure latency.

$ ping -c 4 8.8.8.8
-c NSend N packets then stop

traceroute / tracepath — Trace the Network Path

Purpose Shows every router hop between you and a destination, useful for locating where connectivity breaks.

$ traceroute google.com
$ tracepath google.com   # no root required, similar output
traceroute may need installing (sudo apt install traceroute); tracepath is usually present by default and doesn't require special privileges.

dig / nslookup / host — DNS Lookup Tools

Purpose Query DNS records for a domain.

$ dig ahmadtec.com
$ dig ahmadtec.com MX +short
$ nslookup ahmadtec.com
$ host ahmadtec.com
dig +short gives clean, script-friendly output — the go-to choice for DNS troubleshooting.

curl — Transfer Data / Test HTTP

Purpose Makes HTTP(S) and other protocol requests from the command line — the standard tool for API testing and download scripting.

$ curl -I https://ahmadtec.com          # headers only
$ curl -o file.zip https://example.com/file.zip
$ curl -X POST -d '{"key":"val"}' -H "Content-Type: application/json" https://api.example.com
-IFetch headers only
-o FILESave output to a file
-X METHODHTTP method (GET, POST, PUT...)
-LFollow redirects

wget — Non-Interactive Downloader

Purpose Downloads files, with built-in retry and recursive-download support.

$ wget https://example.com/installer.tar.gz
$ wget -c https://example.com/large.iso   # resume a partial download

scp — Secure Copy Over SSH

Purpose Copies files between hosts using the SSH protocol.

$ scp report.pdf ahmad@server:/home/ahmad/
$ scp -r ./site/ ahmad@server:/var/www/html/

rsync — Efficient File Synchronization

Purpose Synchronizes files/directories, transferring only the differences — much faster than scp for repeated syncs and large trees.

$ rsync -avz ./site/ ahmad@server:/var/www/html/
$ rsync -avz --delete ./local/ /mnt/backup/
-aArchive mode: preserves permissions, symlinks, timestamps
-vVerbose
-zCompress during transfer
--deleteRemove files at destination that no longer exist at source

ssh — Secure Shell

Purpose Opens an encrypted remote shell session, covered in depth in Chapter 17.

$ ssh ahmad@192.168.1.10
$ ssh -p 2222 ahmad@server.example.com

nc (netcat) — The Network Swiss Army Knife

Purpose Reads and writes raw data across network connections — used for port testing, simple file transfer, and quick listeners.

$ nc -zv 192.168.1.10 22    # test if port 22 is open
$ nc -l 4444                # listen on a port

tcpdump — Packet Capture

Purpose Captures and displays network packets for deep troubleshooting.

$ sudo tcpdump -i eth0 port 443
$ sudo tcpdump -i any -w capture.pcap
-i IFACEInterface to capture on
-w FILEWrite raw packets to a file (open later in Wireshark)

arp — Address Resolution Table

Purpose Shows the local cache mapping IP addresses to MAC addresses.

$ arp -a

ip route — Routing Table

$ ip route show
$ sudo ip route add 10.0.0.0/24 via 192.168.1.1

hostnamectl — Manage System Hostname

Purpose Views and sets the system hostname via systemd.

$ hostnamectl
$ sudo hostnamectl set-hostname webserver01

nmcli — NetworkManager CLI

Purpose Controls NetworkManager — connections, Wi-Fi, and interface state — entirely from the command line.

$ nmcli device status
$ nmcli connection show
$ sudo nmcli connection up "Wired connection 1"
$ nmcli device wifi connect "MyNetwork" password "secret"
↑ Back to top

Chapter 10Disk Management

Disk management covers identifying storage devices, partitioning, formatting, mounting, and monitoring free space. A single bad fstab entry can prevent a server from booting, so this chapter pairs every command with practical caution.

lsblk — List Block Devices

Purpose Shows all disks and partitions in a readable tree, with size and mount point.

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   100G  0 disk
├─sda1   8:1    0     1G  0 part /boot
└─sda2   8:2    0    99G  0 part /

blkid — Show Block Device Attributes

Purpose Displays UUID, filesystem type, and label for each block device — the UUID is what you should reference in /etc/fstab.

$ sudo blkid
/dev/sda2: UUID="a1b2c3d4-..." TYPE="ext4"

fdisk — Partition Table Editor (MBR/GPT)

Purpose Interactive tool for creating, deleting, and viewing disk partitions.

$ sudo fdisk -l                # list all partitions
$ sudo fdisk /dev/sdb          # interactive editor: n=new, d=delete, w=write, p=print
Changes made inside fdisk are only written to disk when you press w. Always double-check the target device (/dev/sdb, not /dev/sda!) before writing — partitioning the wrong disk destroys its data.

parted — Advanced Partitioning

Purpose Handles both MBR and GPT partition tables and supports disks larger than 2TB, which classic fdisk historically struggled with.

$ sudo parted /dev/sdb print
$ sudo parted /dev/sdb mklabel gpt
$ sudo parted /dev/sdb mkpart primary ext4 0% 100%

mount / umount — Attach and Detach Filesystems

Purpose mount attaches a filesystem to a directory (mount point); umount detaches it.

$ sudo mount /dev/sdb1 /mnt/data
$ sudo mount -t nfs 192.168.1.20:/export /mnt/nfs
$ sudo umount /mnt/data
If umount reports "target is busy," a process still has an open file handle inside that mount — find it with sudo lsof +D /mnt/data or sudo fuser -m /mnt/data before forcing anything.

findmnt — Query Mounted Filesystems

Purpose Displays currently mounted filesystems in a readable tree, and can verify entries against /etc/fstab.

$ findmnt /
$ findmnt --verify   # check fstab entries are valid

df — Disk Free Space (filesystem level)

Purpose Shows total, used, and available space per mounted filesystem.

$ df -h
Filesystem   Size  Used Avail Use% Mounted on
/dev/sda2     99G   42G   52G  45% /
-hHuman-readable sizes
-TShow filesystem type

du — Disk Usage (directory/file level)

Purpose Shows how much space specific files or directories consume — complements df when you need to find what's filling a disk.

$ du -sh /var/log/*          # summarized size per item
$ du -h --max-depth=1 /home  # one level deep
-sSummarize (total only, no subdirectory breakdown)
-hHuman-readable

fsck — Filesystem Check and Repair

Purpose Checks a filesystem for corruption and attempts repairs. Must be run on an unmounted (or read-only) filesystem.

$ sudo umount /dev/sdb1
$ sudo fsck /dev/sdb1
Never run fsck on a mounted, actively-written filesystem — it can corrupt data further. For the root filesystem, this typically requires booting into rescue/single-user mode.

mkfs — Create a Filesystem

Purpose Formats a partition with a chosen filesystem type, erasing any existing data.

$ sudo mkfs.ext4 /dev/sdb1
$ sudo mkfs.xfs /dev/sdb1
Formatting is irreversible. Confirm the exact device name with lsblk first — this is a leading cause of accidental data loss during disk expansion tasks.

tune2fs — Adjust ext2/3/4 Parameters

Purpose Views or changes tunable filesystem parameters, such as the volume label or maximum mount count before a forced check.

$ sudo tune2fs -l /dev/sda2          # view current settings
$ sudo tune2fs -L mydata /dev/sdb1   # set a label

swapon / swapoff — Manage Swap Space

Purpose Activates or deactivates a swap partition or swap file.

$ sudo fallocate -l 2G /swapfile
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ swapon --show
$ sudo swapoff /swapfile

free — Memory and Swap Summary

$ free -h
              total    used    free   shared  buff/cache  available
Mem:            15G    4.2G     2G     420M        9G          10G
Swap:          2.0G      0B     2G

Mounting Explained

Mounting attaches a block device's filesystem to a location in the existing directory tree, making its contents accessible at that path. Nothing is accessible on a disk until it's mounted somewhere — a freshly attached drive is invisible to normal file operations until you run mount.

/etc/fstab Explained

This file defines filesystems that should be mounted automatically at boot. Each line has six fields:

UUID=a1b2c3d4-...  /data  ext4  defaults  0  2
# device      mountpoint  fstype  options  dump  fsck_order
FieldMeaning
DeviceBest referenced by UUID (from blkid) rather than /dev/sdX, since device letters can shift between boots
Mount pointDirectory where the filesystem will appear
Typeext4, xfs, nfs, etc.
Optionsdefaults, ro, noatime, etc.
DumpLegacy backup flag, almost always 0
fsck order0=never check, 1=root filesystem, 2=checked after root
After editing /etc/fstab, always test with sudo mount -a before rebooting. A malformed fstab entry can drop the system into an emergency shell at next boot.

Practical Scenario: Adding a New Disk

Scenario: Attach and permanently mount a new 100GB data disk

$ lsblk                                  # confirm the new disk is /dev/sdb
$ sudo parted /dev/sdb mklabel gpt
$ sudo parted /dev/sdb mkpart primary ext4 0% 100%
$ sudo mkfs.ext4 /dev/sdb1
$ sudo mkdir /data
$ sudo blkid /dev/sdb1                   # copy the UUID
$ sudo nano /etc/fstab                   # add: UUID=xxxx /data ext4 defaults 0 2
$ sudo mount -a                          # apply and verify without rebooting
$ df -h /data
↑ Back to top

Chapter 11Logs

Logs are the first place to look when anything goes wrong. Modern Ubuntu uses a hybrid model: the systemd journal for structured, indexed logs, alongside traditional flat-text files in /var/log for compatibility with older tools and applications.

The /var/log Directory

File / PathContents
/var/log/syslogGeneral system activity log (Debian/Ubuntu convention)
/var/log/auth.logAuthentication events: logins, sudo usage, SSH attempts (Debian/Ubuntu)
/var/log/kern.logKernel-generated messages (drivers, hardware events)
/var/log/dmesgBoot-time and kernel ring buffer messages
/var/log/apt/Package installation/upgrade history
/var/log/nginx/, /var/log/apache2/Per-application web server logs
RHEL, Rocky, AlmaLinux, and Fedora use /var/log/messages instead of syslog, and /var/log/secure instead of auth.log. The journal-based tools in this chapter work identically across all of them.

journalctl — Structured System Log Viewer

Purpose Query the systemd journal, which aggregates logs from every unit, the kernel, and boot process into a single, filterable, indexed store. (Covered in Chapter 8 for service-specific use; summarized here for general log analysis.)

$ journalctl                        # entire journal, oldest first
$ journalctl -r                     # newest first
$ journalctl -k                     # kernel messages only
$ journalctl --since "2026-07-10 08:00" --until "2026-07-10 09:00"
$ journalctl -f                     # live tail across the whole system

/var/log/auth.log — Authentication Events

Purpose Records every login attempt, sudo invocation, and SSH session — the first file to check for suspected brute-force attempts.

$ sudo tail -f /var/log/auth.log
$ sudo grep "Failed password" /var/log/auth.log

/var/log/syslog — General System Log

Purpose Broad log of daemon and system-level messages not covered by a more specific log file.

$ sudo tail -n 100 /var/log/syslog

/var/log/kern.log — Kernel Messages

Purpose Hardware and driver-level events: disk errors, USB device detection, OOM killer activity.

$ sudo grep -i "error" /var/log/kern.log

dmesg — Kernel Ring Buffer

Purpose Displays messages from the kernel's ring buffer, including all boot-time hardware detection.

$ dmesg | tail -50
$ dmesg -T | grep -i usb    # -T shows human-readable timestamps

tail -f — Live Log Following

Purpose The standard technique for watching a log file update in real time while reproducing an issue.

$ sudo tail -f /var/log/nginx/error.log
$ sudo tail -f /var/log/syslog /var/log/auth.log   # follow multiple files at once
Combine tail -f with grep --line-buffered to filter a live stream in real time: tail -f /var/log/syslog | grep --line-buffered -i error.
↑ Back to top

Chapter 12Searching & Text Processing

Filtering, transforming, and comparing text is a daily task for parsing logs, configuration files, and command output. This chapter covers the classic Unix text-processing toolkit — commands designed to be chained together with pipes.

grep — Search Text by Pattern

Purpose Searches input for lines matching a pattern (plain text or regular expression).

Syntax grep [OPTION] PATTERN [FILE]

$ grep "error" /var/log/syslog
$ grep -ri "failed" /var/log/auth.log
$ grep -v "^#" /etc/ssh/sshd_config   # exclude comment lines
$ grep -c "GET" access.log            # count matches
$ grep -A2 -B2 "panic" kern.log       # show 2 lines of context
-iCase-insensitive
-rRecursive through directories
-vInvert match (show non-matching lines)
-cCount matches instead of printing lines
-nShow line numbers
-A/-B/-C NShow N lines of context after/before/around

egrep — Extended Grep

Purpose Equivalent to grep -E, supporting extended regex syntax (+, ?, |, ()) without escaping.

$ egrep "error|warning|critical" /var/log/syslog

sed — Stream Editor

Purpose Performs find-and-replace and other line-based transformations on text streams, without opening an editor.

Syntax sed 's/PATTERN/REPLACEMENT/' FILE

$ sed 's/localhost/127.0.0.1/' /etc/hosts
$ sed -i 's/Port 22/Port 2222/' /etc/ssh/sshd_config   # -i edits in place
$ sed -n '5,10p' file.txt                              # print only lines 5-10
-iEdit the file in place (add .bak suffix for a backup: -i.bak)
-nSuppress automatic printing (used with explicit p)
Always test a sed -i command without -i first to confirm the output looks correct, especially on critical config files — there is no undo.

awk — Pattern Scanning and Field Processing

Purpose A full text-processing language built around columns/fields, ideal for extracting or computing on structured text like logs and CSVs.

Syntax awk 'PATTERN {ACTION}' FILE

$ awk '{print $1}' access.log             # first column of every line
$ awk -F: '{print $1, $3}' /etc/passwd     # custom delimiter
$ awk '$3 > 1000 {print $1}' /etc/passwd   # conditional filtering
$ awk '{sum+=$5} END{print sum}' sizes.txt # running total

xargs — Build Commands from Input

Purpose Converts piped input into arguments for another command, essential since many commands can't read stdin directly.

$ find . -name "*.tmp" | xargs rm
$ cat urls.txt | xargs -n1 curl -O
$ find . -name "*.log" | xargs -I{} mv {} /var/log/archive/
-n NPass N arguments per invocation
-I{}Placeholder for substituting each input item

find & locate

Covered in depth in Chapter 2 (Navigation). find searches live, locate queries a prebuilt index.

sort & uniq

Covered in Chapter 3 (File Management) — frequently combined with grep and awk in text-processing pipelines, e.g. sort access.log | uniq -c | sort -nr.

diff — Compare Two Files

Purpose Shows the line-by-line differences between two text files.

$ diff config_old.yml config_new.yml
$ diff -u file1.txt file2.txt   # unified diff format, used for patches

cmp — Byte-Level Comparison

Purpose Reports the first byte or line at which two files differ — faster than diff for binary files.

$ cmp file1.bin file2.bin

comm — Compare Sorted Files

Purpose Compares two sorted files and outputs lines unique to each, plus lines common to both, in three columns.

$ comm <(sort list1.txt) <(sort list2.txt)

strings — Extract Printable Text

Purpose Pulls human-readable text out of binary files — useful for inspecting executables or unknown file types.

$ strings /usr/bin/nginx | grep -i version
↑ Back to top

Chapter 13Compression

Archiving bundles multiple files into one; compression shrinks their size. Linux keeps these as separate concerns — tar archives, while gzip/bzip2/xz compress, and they are almost always used together.

tar — Tape Archive

Purpose Bundles multiple files/directories into a single archive, optionally compressing them in the same step.

Syntax tar [OPTIONS] ARCHIVE FILES

$ tar -cvf archive.tar /home/ahmad/project      # create, plain tar
$ tar -czvf archive.tar.gz /home/ahmad/project  # create, gzip compressed
$ tar -cjvf archive.tar.bz2 /home/ahmad/project # create, bzip2 compressed
$ tar -xvf archive.tar                          # extract
$ tar -xzvf archive.tar.gz -C /restore/         # extract to specific dir
$ tar -tvf archive.tar                          # list contents without extracting
-cCreate a new archive
-xExtract an archive
-tList archive contents
-vVerbose (print filenames as processed)
-fSpecify the archive filename (almost always required)
-zFilter through gzip
-jFilter through bzip2
-JFilter through xz
-C DIRChange to DIR before extracting

gzip / gunzip — GNU Zip Compression

Purpose Compresses a single file (replacing it with a .gz version); gunzip reverses this.

$ gzip logfile.txt          # produces logfile.txt.gz, removes original
$ gunzip logfile.txt.gz     # restores logfile.txt
$ gzip -k logfile.txt       # keep the original file too

zip / unzip — Cross-Platform Archive Format

Purpose Creates/extracts .zip archives — the most portable format for sharing with Windows/macOS users.

$ zip -r project.zip project/
$ unzip project.zip
$ unzip -l project.zip    # list contents without extracting

xz — High-Ratio Compression

Purpose Offers a significantly better compression ratio than gzip, at the cost of more CPU time — common for distributing source tarballs and kernel images.

$ xz archive.tar             # produces archive.tar.xz
$ xz -d archive.tar.xz       # decompress
$ xz -9 -k bigfile.txt       # maximum compression, keep original

bzip2 — Block-Sorting Compression

Purpose Better compression than gzip but slower; still common for some legacy pipelines and specific tools.

$ bzip2 datafile.txt
$ bunzip2 datafile.txt.bz2

Format Comparison

FormatCompression RatioSpeedTypical Use
gzip (.gz)ModerateFastGeneral purpose, log rotation, default choice
bzip2 (.bz2)BetterSlowerLarger text files where ratio matters more than speed
xz (.xz)BestSlowest (but fast to decompress)Source distribution, kernel images, long-term archives
zip (.zip)ModerateFastCross-platform sharing with Windows/macOS
↑ Back to top

Chapter 14Scheduling Tasks

Linux offers two main scheduling mechanisms: cron for recurring jobs and at for one-time future jobs. Modern systemd-based distributions also support systemd timers as a more powerful, loggable alternative to cron.

crontab — Manage Scheduled Jobs

Purpose Edits the current user's crontab, the file defining their recurring scheduled jobs.

Syntax crontab [OPTION]

$ crontab -e          # edit your own crontab
$ crontab -l          # list current cron jobs
$ sudo crontab -u www-data -e   # edit another user's crontab

Cron Syntax

# minute hour day-of-month month day-of-week  command
  *      *    *           *     *
  30     2    *           *     *   /home/ahmad/scripts/backup.sh
  0      */6  *           *     *   /usr/bin/certbot renew
  0      9    *           *     1-5 /home/ahmad/scripts/weekday_report.sh
FieldRangeNotes
Minute0–59
Hour0–2324-hour format
Day of month1–31
Month1–12
Day of week0–70 and 7 both mean Sunday
Shortcuts like @reboot, @daily, @weekly, and @hourly can replace the five time fields for common schedules, e.g. @reboot /home/ahmad/scripts/startup.sh.

at — Run a Command Once, in the Future

Purpose Schedules a one-time command to run at a specific future time, unlike cron's recurring model.

Syntax at TIME

$ at 22:00
at> systemctl restart nginx
at> <Ctrl+D>
$ atq                  # list pending jobs
$ atrm 3               # cancel job number 3
The at package may need installing: sudo apt install at. It's available under the same name on RHEL-family distros via dnf install at.

systemd Timers — Modern Scheduling

Purpose A systemd-native alternative to cron, with built-in logging via journalctl, dependency management, and "catch-up" execution for jobs missed while the system was off.

Explanation A timer requires two files: a .timer unit defining the schedule, and a .service unit defining what to run.

/etc/systemd/system/backup.service:
[Unit]
Description=Run nightly backup

[Service]
Type=oneshot
ExecStart=/home/ahmad/scripts/backup.sh

/etc/systemd/system/backup.timer:
[Unit]
Description=Trigger backup nightly

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now backup.timer
$ systemctl list-timers          # see all active timers and next run time
$ journalctl -u backup.service   # view job's execution logs
Use systemd timers over cron for anything you'll need to debug — every run is logged in the journal with exit codes, unlike cron which silently emails output (if configured) or discards it.
↑ Back to top

Chapter 15Environment Variables

Environment variables carry configuration into every process a shell launches — from where executables are found (PATH) to where a user's files live (HOME). Understanding how and where they're set is essential for debugging "command not found" errors and inconsistent script behavior.

PATH — Executable Search Path

Purpose A colon-separated list of directories the shell searches, in order, when you type a command name.

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/home/ahmad/bin
$ export PATH="$PATH:/opt/myapp/bin"   # append a new directory
Never put a "." (current directory) at the start of PATH — it's a classic security risk, allowing a malicious file in the current folder to shadow a real system command.

HOME — Current User's Home Directory

$ echo $HOME
/home/ahmad

export — Set an Environment Variable

Purpose Marks a shell variable as part of the environment, so child processes (including scripts you run) inherit it.

$ export EDITOR=nano
$ export API_KEY="xyz123"
A variable set without export (e.g. just EDITOR=nano) only exists in the current shell and is not passed to programs it launches.

env — Show or Modify the Environment

Purpose Displays all current environment variables, or runs a command with a modified environment.

$ env
$ env DEBUG=1 ./myscript.sh   # run with one variable temporarily set

printenv — Print Environment Variables

Purpose Prints the value of one or all environment variables.

$ printenv PATH
$ printenv

alias / unalias — Command Shortcuts

Purpose Creates a shorthand name for a longer command.

$ alias ll='ls -lah'
$ alias update='sudo apt update && sudo apt upgrade -y'
$ unalias ll
Aliases defined at the shell prompt only last for the session. Add them to ~/.bashrc to make them permanent.

source — Re-Read a Script into the Current Shell

Purpose Executes a script's commands in the current shell instead of a subshell, so variable/function/alias definitions persist afterward.

Syntax source FILE or the shorthand . FILE

$ source ~/.bashrc
$ . ./venv/bin/activate

Shell Startup Files

FileWhen It Runs
~/.bashrcEvery new interactive, non-login shell (e.g. opening a new terminal tab). The right place for aliases and functions.
~/.profileLogin shells only (e.g. console login, some SSH sessions). The right place for PATH and environment variable exports meant to apply system-wide for that user.
/etc/environmentSystem-wide environment variables, applied to all users at login, parsed without shell syntax (no export, no quoting logic).
On Ubuntu, interactive login shells typically source ~/.profile, which in turn sources ~/.bashrc — but the exact chain can vary by terminal emulator and SSH client configuration. If a variable "isn't sticking," check which of these files actually ran for that session type.
↑ Back to top

Chapter 16Bash Basics

Bash scripting turns repetitive command sequences into reusable, reliable automation. This chapter covers the essential building blocks: variables, conditionals, loops, functions, arguments, and exit codes.

Variables

#!/bin/bash
name="Ahmad"
count=5
echo "Hello, $name. Count is $count"
echo "Hello, ${name}_suffix"     # braces avoid ambiguity
No spaces around = when assigning (name="Ahmad", not name = "Ahmad") — a space makes bash interpret it as a command instead of an assignment.

Conditions

if [ "$count" -gt 3 ]; then
    echo "count is greater than 3"
elif [ "$count" -eq 3 ]; then
    echo "count is exactly 3"
else
    echo "count is 3 or less"
fi
TestMeaning
-eq / -neNumeric equal / not equal
-gt / -ltGreater than / less than
-ge / -leGreater or equal / less or equal
= / !=String equal / not equal
-z / -nString is empty / string is non-empty
-f / -d / -ePath is a file / directory / exists at all

Loops

# for loop
for i in 1 2 3 4 5; do
    echo "Number: $i"
done

# for loop over files
for file in /var/log/*.log; do
    echo "Processing $file"
done

# while loop
count=0
while [ $count -lt 5 ]; do
    echo "count=$count"
    count=$((count+1))
done

Functions

backup_dir() {
    local src="$1"
    local dest="$2"
    tar -czf "$dest/backup_$(date +%F).tar.gz" "$src"
    echo "Backup complete: $dest"
}

backup_dir /home/ahmad/project /mnt/backups
Use local for variables inside a function to keep them scoped there, preventing accidental collisions with variables of the same name elsewhere in the script.

Positional Arguments

#!/bin/bash
echo "Script name: $0"
echo "First arg: $1"
echo "Second arg: $2"
echo "All args: $@"
echo "Arg count: $#"
$ ./myscript.sh alpha beta
Script name: ./myscript.sh
First arg: alpha
Second arg: beta
All args: alpha beta
Arg count: 2

Exit Codes

Explanation Every command returns an exit code: 0 means success, any non-zero value indicates a specific failure. Scripts should check and propagate these deliberately.

ping -c1 8.8.8.8 &> /dev/null
if [ $? -eq 0 ]; then
    echo "Network is up"
else
    echo "Network is down"
    exit 1
fi
Prefer checking the command directly in the if rather than $? when possible: if ping -c1 8.8.8.8 &> /dev/null; then ... fi is cleaner and avoids accidentally clobbering $? with an intermediate command.
↑ Back to top

Chapter 17SSH Administration

SSH (Secure Shell) is the backbone of remote Linux administration. This chapter covers key-based authentication — the recommended replacement for password logins — and the configuration files that control client and server behavior.

ssh-keygen — Generate a Key Pair

Purpose Creates a public/private key pair for passwordless, cryptographic authentication.

Syntax ssh-keygen [OPTION]

$ ssh-keygen -t ed25519 -C "ahmad@ahmadtec.com"
$ ssh-keygen -t rsa -b 4096 -C "backup-key"
-t TYPEKey algorithm: ed25519 (recommended, fast & secure) or rsa
-b BITSKey size in bits (RSA only; 4096 recommended)
-C COMMENTLabel embedded in the public key, usually an email or purpose
Prefer ed25519 for new keys — it's faster and just as secure as RSA-4096 with a much shorter key. Use RSA only for compatibility with very old systems.

ssh-copy-id — Install Your Public Key on a Remote Server

Purpose Appends your public key to the remote user's ~/.ssh/authorized_keys, enabling passwordless login.

$ ssh-copy-id ahmad@192.168.1.10
$ ssh-copy-id -i ~/.ssh/id_ed25519.pub -p 2222 ahmad@server.example.com

scp / sftp — File Transfer Over SSH

Purpose scp copies files directly; sftp opens an interactive file-transfer session, similar to FTP but encrypted.

$ scp report.pdf ahmad@server:/home/ahmad/
$ sftp ahmad@server
sftp> put localfile.txt
sftp> get remotefile.txt
sftp> ls
sftp> bye

~/.ssh/config — Client-Side Shortcuts

Purpose Defines per-host aliases and connection options so you don't need to remember IPs, ports, and usernames every time.

~/.ssh/config:
Host expovision
    HostName 88.99.xxx.xxx
    User ahmad
    Port 2222
    IdentityFile ~/.ssh/id_ed25519

Host homeserver
    HostName ahmadtec.com
    User ahmad
    Port 22
$ ssh expovision       # instead of ssh -p 2222 ahmad@88.99.xxx.xxx

~/.ssh/known_hosts — Trusted Host Fingerprints

Purpose Records the cryptographic fingerprint of every remote host you've connected to, protecting against man-in-the-middle attacks. If a server's key changes unexpectedly, SSH will warn loudly and refuse to connect until you confirm.

$ ssh-keygen -R 192.168.1.10   # remove a stale/changed host entry

~/.ssh/authorized_keys — Server-Side Trusted Keys

Purpose Lists public keys permitted to log in as that user, one per line. This is the file ssh-copy-id writes to.

$ cat ~/.ssh/authorized_keys
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAA... ahmad@ahmadtec.com
Permissions matter: ~/.ssh must be 700 and authorized_keys must be 600, or sshd will refuse to use them for authentication as a security precaution.

Basic Hardening Checklist

Setting in /etc/ssh/sshd_configRecommended Value
PasswordAuthenticationno (once key-based login is confirmed working)
PermitRootLoginno or prohibit-password
PortNon-default port reduces automated scan noise (defense in depth, not a substitute for the above)
$ sudo systemctl restart ssh   # apply changes after editing sshd_config
Always keep a second terminal session open (or console access) while changing SSH settings. Test the new configuration in a fresh connection before closing your existing session — a mistake in sshd_config can lock you out entirely.
↑ Back to top

Chapter 18Monitoring & Performance

Beyond top/htop (Chapter 7), a deeper performance toolkit helps diagnose CPU, memory, disk I/O, and hardware bottlenecks precisely.

uptime — System Uptime and Load Average

Purpose Shows how long the system has been running and the load average over 1, 5, and 15 minutes.

$ uptime
 14:32:01 up 21 days,  3:12,  2 users,  load average: 0.45, 0.38, 0.31
Load average roughly represents the number of processes wanting CPU time. A value near or above your core count sustained over the 15-minute figure suggests the system is CPU-constrained.

vmstat — Virtual Memory Statistics

Purpose Reports memory, swap, I/O, and CPU activity in a single repeating snapshot — good for spotting swapping under load.

$ vmstat 2 5     # every 2 seconds, 5 times

iostat — Disk I/O Statistics

Purpose Reports per-device read/write throughput and utilization, key for diagnosing disk-bound performance issues.

Explanation Part of the sysstat package: sudo apt install sysstat.

$ iostat -x 2

mpstat — Per-CPU Statistics

Purpose Breaks down CPU usage per core, useful for spotting a single-threaded bottleneck on a multi-core system.

$ mpstat -P ALL 2

sar — System Activity Reporter

Purpose Collects and reports historical system activity (CPU, memory, disk, network) over time, based on data logged by sysstat.

$ sar -u 1 3      # CPU usage, 3 samples 1 second apart
$ sar -r          # memory usage report for today

iotop — Per-Process Disk I/O

Purpose Like top, but ranks processes by disk read/write throughput instead of CPU.

$ sudo iotop

free — Memory Summary

Covered in Chapter 10. Quick reminder:

$ free -h

lscpu — CPU Architecture Details

Purpose Displays CPU model, core/thread count, cache sizes, and virtualization support flags.

$ lscpu

lsmem — Memory Block Layout

Purpose Shows how much physical memory is installed and online, broken into memory blocks.

$ lsmem

hostnamectl & uname — System Identity

$ hostnamectl
$ uname -a
Linux server01 6.8.0-45-generic #45-Ubuntu SMP x86_64 GNU/Linux
-rKernel release only
-mMachine hardware (architecture)
-aAll information
↑ Back to top

Chapter 19Security

This chapter surveys the core security tooling and practices for a hardened Linux server: firewalls, intrusion prevention, SSH hardening, sudo hygiene, password policy, and the two major mandatory access control frameworks.

ufw — Uncomplicated Firewall

Purpose Ubuntu's user-friendly front end for iptables/nftables, designed for simple rule management without raw netfilter syntax.

$ sudo ufw enable
$ sudo ufw allow 22/tcp
$ sudo ufw allow from 192.168.1.0/24 to any port 3306
$ sudo ufw deny 8080
$ sudo ufw status verbose
$ sudo ufw delete allow 8080
Always allow SSH (or your admin port) before running ufw enable on a remote server — enabling the firewall with no allow rule for your current connection will lock you out immediately.

iptables — Low-Level Firewall (Overview)

Purpose The traditional Linux packet-filtering framework that ufw and firewalld both configure under the hood. Direct iptables use is now mostly reserved for advanced or legacy setups, since most distributions (including current Ubuntu) use nftables as the actual backend.

$ sudo iptables -L -n -v                      # list current rules
$ sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT
RHEL, Rocky, AlmaLinux, and Fedora typically use firewalld (firewall-cmd) as their high-level firewall manager, playing the same role ufw plays on Ubuntu.

fail2ban — Automated Intrusion Prevention

Purpose Watches log files (like auth.log) for repeated failed login attempts and temporarily bans the offending IP via firewall rules.

$ sudo apt install fail2ban
$ sudo systemctl enable --now fail2ban
$ sudo fail2ban-client status sshd
$ sudo fail2ban-client set sshd unbanip 203.0.113.5

SSH Hardening

Covered in depth in Chapter 17. Key practices: disable password authentication once keys work, disable root login, and keep sshd updated.

Sudo Best Practices

  • Grant privileges through group membership (e.g. the sudo group) rather than editing /etc/sudoers per user.
  • Use visudo exclusively for editing sudoers (Chapter 5) to avoid syntax errors locking out access.
  • Scope NOPASSWD exceptions to specific commands, never to ALL, to limit blast radius if an account is compromised.
  • Review /var/log/auth.log periodically for unexpected sudo usage.

Password Policies

Relevant Files /etc/login.defs (defaults for new accounts), /etc/security/pwquality.conf (complexity rules via PAM).

$ sudo chage -l ahmad          # view password aging info
$ sudo chage -M 90 ahmad       # force password change every 90 days

File Permissions

Covered in full in Chapter 4. From a security lens: audit for world-writable files and unnecessary SUID/SGID binaries.

$ find / -xdev -perm -0002 -type f 2>/dev/null     # world-writable files
$ find / -xdev -perm -4000 -type f 2>/dev/null      # SUID binaries

SELinux vs AppArmor (Overview)

Both are Mandatory Access Control (MAC) frameworks that confine what a process can do, beyond standard Unix permissions — even a compromised or misbehaving process is restricted to a defined policy.

SELinuxAppArmor
Used byRHEL, Rocky Linux, AlmaLinux, FedoraUbuntu, Debian, SUSE
ModelLabel-based: every file/process gets a security context; policy defines allowed context interactionsPath-based: profiles restrict what a specific executable can access by file path
Learning curveSteeper, more granularSimpler to read and author profiles
Key commandsgetenforce, setenforce, sestatus, semanageaa-status, aa-enforce, aa-complain
$ getenforce                    # SELinux: check current mode
$ sudo aa-status                # AppArmor: list loaded profiles
Ubuntu ships with AppArmor enabled by default; SELinux is not typically used on Ubuntu. Never disable AppArmor or SELinux to "fix" a permission problem in production — instead check the relevant deny log (journalctl for AppArmor denials, audit.log for SELinux) and adjust the profile/policy.
↑ Back to top

Chapter 20Boot Process

Understanding the boot sequence is essential for recovering a system that won't start. This chapter traces the journey from power-on to a usable login prompt.

The Boot Sequence, Step by Step

StageWhat Happens
1. BIOSLegacy firmware performs a power-on self-test (POST) and hands control to the boot device's Master Boot Record (MBR).
2. UEFIModern replacement for BIOS; reads boot entries from the EFI System Partition (ESP) and can boot GPT disks larger than 2TB, unlike legacy BIOS/MBR.
3. GRUBThe bootloader (GRand Unified Bootloader) presents a menu (or boots directly), then loads the selected Linux kernel and initramfs into memory.
4. KernelThe Linux kernel initializes hardware, mounts the initramfs as a temporary root filesystem, and starts the first userspace process.
5. initramfsA minimal temporary root filesystem containing just enough drivers and tools to find and mount the real root filesystem (e.g. LVM, RAID, or encrypted volume drivers).
6. systemd (PID 1)Once the real root filesystem is mounted, control passes to systemd, which brings up all configured services according to the selected target.

GRUB Basics

Config location /etc/default/grub (settings) and /boot/grub/grub.cfg (generated menu, do not edit directly).

$ sudo update-grub                  # regenerate grub.cfg after changing settings (Ubuntu/Debian)
$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg  # RHEL-family equivalent

Kernel & initramfs Files

$ ls /boot
vmlinuz-6.8.0-45-generic   initrd.img-6.8.0-45-generic   grub/

systemd Targets (Boot Levels)

Targets are systemd's replacement for the old SysV runlevels — a named set of units that should be active for a given system state.

TargetOld Runlevel EquivalentPurpose
poweroff.target0Shut down the system
rescue.target1Single-user mode with basic filesystems mounted
multi-user.target3Full multi-user mode, no graphical interface
graphical.target5Multi-user mode plus a display manager/desktop
reboot.target6Reboot the system
$ systemctl get-default
$ sudo systemctl set-default multi-user.target
$ sudo systemctl isolate rescue.target   # switch target immediately

Rescue Mode vs Emergency Mode

Rescue ModeEmergency Mode
FilesystemsLocal filesystems mountedOnly root filesystem mounted, often read-only
ServicesMinimal set of base services startedAlmost nothing started — bare minimum shell
When to useGeneral recovery, password resets, fixing fstabSevere boot failures, corrupted root filesystem needing fsck
Access via GRUBAdd systemd.unit=rescue.target to the kernel lineAdd systemd.unit=emergency.target to the kernel line
To edit a kernel boot line on the fly: at the GRUB menu, highlight an entry and press e, add the parameter to the line starting with linux, then press Ctrl+X or F10 to boot with it — this change is temporary and not saved to disk.
↑ Back to top

Chapter 21Essential Configuration Files

This chapter consolidates the configuration files every administrator should recognize on sight — several were introduced earlier in context; here they're gathered as a single reference.
FilePurpose
/etc/fstabDefines filesystems mounted automatically at boot: device (by UUID), mount point, type, options, dump flag, fsck order. See Chapter 10.
/etc/hostsStatic hostname-to-IP mappings, checked before DNS resolution. Format: IP  hostname  [alias...].
/etc/hostnameContains the system's static hostname as a single line; read at boot. Change with hostnamectl set-hostname rather than editing directly.
/etc/resolv.confSpecifies DNS nameservers the resolver should query. On modern Ubuntu this file is typically auto-generated by systemd-resolved or NetworkManager — manual edits are often overwritten.
/etc/passwdUser account database: username, UID, GID, comment, home directory, shell. World-readable. See Chapter 5.
/etc/shadowEncrypted password hashes and password-aging fields, root-readable only. See Chapter 5.
/etc/groupGroup definitions and member lists. See Chapter 5.
/etc/sudoersDefines who can run commands as root/other users via sudo, and under what conditions. Edit only with visudo. See Chapter 5.
/etc/ssh/sshd_configSSH server (daemon) configuration: port, authentication methods, root login policy. See Chapter 17.
/etc/crontabSystem-wide cron table with an extra "user" field (unlike per-user crontabs edited via crontab -e). See Chapter 14.
/etc/environmentSystem-wide environment variables applied at login for all users, parsed without shell syntax. See Chapter 15.
When troubleshooting, this table doubles as a checklist: most "why isn't X working" questions about networking, access, or scheduled jobs trace back to one of these eleven files.
↑ Back to top

Chapter 22Most Common Troubleshooting Scenarios

Each scenario below follows the same structure: how to investigate, the commands to use, and typical fixes. These are the situations that make up the bulk of day-to-day system administration work.

1. Disk Full

Investigate

$ df -h                              # which filesystem is full
$ du -sh /var/log/* | sort -rh | head -10   # biggest consumers
$ sudo journalctl --disk-usage

Typical Fixes

  • Truncate or rotate oversized logs: truncate -s 0 /var/log/huge.log
  • Vacuum the systemd journal: sudo journalctl --vacuum-size=200M
  • Clean package cache: sudo apt clean, sudo apt autoremove
  • Check for deleted-but-open files still holding space: sudo lsof +L1

2. Service Won't Start

Investigate

$ systemctl status myservice
$ journalctl -u myservice -n 50 --no-pager
$ sudo systemd-analyze verify myservice.service   # check unit file for errors

Typical Fixes

  • Fix syntax errors in the unit file, then sudo systemctl daemon-reload
  • Check the binary path in ExecStart actually exists and is executable
  • Verify the port it needs isn't already in use (see scenario 7)
  • Check file/directory permissions the service account needs to access

3. SSH Not Working

Investigate

$ systemctl status ssh
$ sudo ss -tulpn | grep :22
$ sudo journalctl -u ssh -n 50
$ sudo sshd -t                       # test config syntax

Typical Fixes

  • Ensure the service is running: sudo systemctl restart ssh
  • Check ufw status / firewall isn't blocking port 22
  • Verify authorized_keys permissions are 600 and .ssh is 700
  • Confirm you're connecting to the right port if Port was changed in sshd_config

4. DNS Issues

Investigate

$ dig example.com
$ cat /etc/resolv.conf
$ resolvectl status                  # systemd-resolved status, Ubuntu
$ ping -c2 8.8.8.8                   # is it DNS or full connectivity?

Typical Fixes

  • If ping 8.8.8.8 works but domain names fail, it's a DNS-specific problem, not general connectivity
  • Restart the resolver: sudo systemctl restart systemd-resolved
  • Check /etc/hosts for a stale, conflicting manual entry
  • Try an alternate nameserver temporarily to isolate a resolver outage: dig @8.8.8.8 example.com

5. High CPU Usage

Investigate

$ top                                # sort by CPU, default view
$ mpstat -P ALL 2                    # per-core breakdown
$ ps -eo pid,ppid,cmd,%cpu --sort=-%cpu | head

Typical Fixes

  • Identify and restart/kill a runaway process
  • renice a legitimate but low-priority background job
  • Check for an infinite loop or retry storm in application logs
  • Scale resources (add vCPUs) if load is genuinely legitimate and sustained

6. High RAM Usage

Investigate

$ free -h
$ ps -eo pid,cmd,%mem --sort=-%mem | head
$ sudo journalctl -k | grep -i "out of memory"   # check for OOM killer events

Typical Fixes

  • Restart a process with a suspected memory leak
  • Add or enlarge swap space (Chapter 10) as a stopgap, not a permanent fix for a leak
  • Tune application-level memory limits (e.g. PHP-FPM, JVM heap size)

7. Port Already in Use

Investigate

$ sudo ss -tulpn | grep :8080
$ sudo lsof -i :8080

Typical Fixes

  • Stop the conflicting process: sudo kill <PID>, or stop its owning service properly via systemctl
  • Reconfigure one of the two services to use a different port

8. Permission Denied

Investigate

$ ls -la /path/to/file
$ id                                 # confirm your own UID/groups
$ getfacl /path/to/file              # check for ACL overrides
$ sudo aa-status                     # AppArmor may be blocking even with correct Unix perms

Typical Fixes

  • Correct ownership/mode: chown, chmod (Chapter 4)
  • Add the user to the required group rather than loosening permissions to 777
  • Check SELinux/AppArmor denials if standard permissions look correct but access still fails

9. Package Broken / Dependency Errors

Investigate

$ sudo apt update
$ sudo dpkg --configure -a
$ apt-cache policy PACKAGE_NAME

Typical Fixes

  • Fix broken dependencies: sudo apt --fix-broken install
  • Reconfigure a partially-installed package: sudo dpkg --configure -a
  • Purge and reinstall as a last resort: sudo apt purge PACKAGE && sudo apt install PACKAGE

10. Network Unreachable

Investigate

$ ip a                               # does the interface have an IP?
$ ip route show                      # is there a default route?
$ ping -c2 <gateway_ip>
$ nmcli device status

Typical Fixes

  • Bring the interface up: sudo ip link set eth0 up
  • Restart networking: sudo systemctl restart NetworkManager or networking
  • Re-request a DHCP lease: sudo dhclient -r && sudo dhclient
  • Check for a missing or incorrect default route and add manually if needed
↑ Back to top

Chapter 23RHCSA-Style Practical Tasks

This chapter presents 50 hands-on administration tasks in the style of RHCSA/LFCS practical exams, each with a complete, runnable solution. They are grouped by domain: users & permissions, storage, services, networking, packages, logs & scheduling, and security & troubleshooting.

Users, Groups & Permissions

Task 1 — Create a user with a specific UID and home directory

$ sudo useradd -u 1500 -m -d /home/hana -s /bin/bash hana
$ sudo passwd hana

Task 2 — Create a group and add multiple existing users to it

$ sudo groupadd developers
$ sudo usermod -aG developers ahmad
$ sudo usermod -aG developers hana
$ getent group developers

Task 3 — Set a password to expire in 60 days

$ sudo chage -M 60 ahmad
$ sudo chage -l ahmad

Task 4 — Lock and later unlock a user account

$ sudo usermod -L hana
$ sudo passwd -S hana        # confirm status shows L (locked)
$ sudo usermod -U hana

Task 5 — Grant passwordless sudo for one specific command

$ sudo visudo -f /etc/sudoers.d/deploy
\# add this line:
deploy ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart myapp

Task 6 — Set the setgid bit so new files in a shared folder inherit the group

$ sudo mkdir /srv/shared
$ sudo chgrp developers /srv/shared
$ sudo chmod 2775 /srv/shared      # 2 = setgid

Task 7 — Grant a specific user read-write access via ACL without changing group ownership

$ setfacl -m u:hana:rw /srv/shared/report.docx
$ getfacl /srv/shared/report.docx

Task 8 — Set a restrictive default umask for a specific user session

$ echo "umask 027" >> ~/.bashrc
$ source ~/.bashrc
$ umask

Task 9 — Find all files owned by a deleted user (orphaned UID)

$ sudo find / -xdev -nouser 2>/dev/null

Task 10 — Reassign ownership of a whole directory tree after renaming a user

$ sudo find /home/olduser -xdev -exec chown newuser:newuser {} \;

Storage & Filesystems

Task 11 — Create a new partition, format it, and mount it permanently

$ sudo parted /dev/sdb mklabel gpt
$ sudo parted /dev/sdb mkpart primary ext4 0% 100%
$ sudo mkfs.ext4 /dev/sdb1
$ sudo mkdir /data
$ sudo blkid /dev/sdb1
$ echo "UUID=<uuid> /data ext4 defaults 0 2" | sudo tee -a /etc/fstab
$ sudo mount -a

Task 12 — Extend an existing LVM logical volume

$ sudo lvextend -L +5G /dev/vgdata/lvdata
$ sudo resize2fs /dev/vgdata/lvdata      # ext4; use xfs_growfs for XFS

Task 13 — Create a swap file and enable it permanently

$ sudo fallocate -l 2G /swapfile
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ echo "/swapfile none swap sw 0 0" | sudo tee -a /etc/fstab

Task 14 — Find and remove files older than 30 days in a temp directory

$ find /tmp/uploads -type f -mtime +30 -delete

Task 15 — Identify what's consuming the most space under /var

$ sudo du -h --max-depth=2 /var | sort -rh | head -15

Task 16 — Set up a bind mount to expose a directory at two paths

$ sudo mkdir -p /srv/shared/docs
$ sudo mount --bind /home/ahmad/docs /srv/shared/docs
# persistent entry: /home/ahmad/docs /srv/shared/docs none bind 0 0

Task 17 — Check and repair a non-root filesystem safely

$ sudo umount /dev/sdb1
$ sudo fsck -y /dev/sdb1
$ sudo mount /dev/sdb1 /data

Task 18 — Mount an ISO image without burning it to physical media

$ sudo mkdir /mnt/iso
$ sudo mount -o loop ubuntu-24.04.iso /mnt/iso

Task 19 — Set a disk quota for a user (ext4)

$ sudo apt install quota
$ sudo mount -o remount,usrquota /data
$ sudo quotacheck -cum /data
$ sudo setquota -u hana 5G 6G 0 0 /data

Task 20 — Rename a volume label for easier identification

$ sudo e2label /dev/sdb1 BACKUPDISK
$ lsblk -f

Services & Processes

Task 21 — Create and enable a custom systemd service

/etc/systemd/system/myapp.service:
[Unit]
Description=My Application
After=network.target

[Service]
ExecStart=/usr/bin/python3 /opt/myapp/app.py
Restart=on-failure
User=myappuser

[Install]
WantedBy=multi-user.target
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now myapp

Task 22 — Diagnose and fix a failed service

$ systemctl --failed
$ journalctl -u myapp -n 50 --no-pager
$ sudo systemctl restart myapp

Task 23 — Prevent a problematic service from ever starting

$ sudo systemctl mask apache2

Task 24 — Find and kill a process holding a port open

$ sudo lsof -i :8080
$ sudo kill -9 <PID>

Task 25 — Run a long job that survives terminal disconnection

$ nohup ./long_job.sh > job.log 2>&1 &
$ disown

Networking

Task 26 — Assign a static IP address using Netplan (Ubuntu)

/etc/netplan/00-installer-config.yaml:
network:
  version: 2
  ethernets:
    eth0:
      addresses: [192.168.1.50/24]
      routes:
        - to: default
          via: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8, 1.1.1.1]
$ sudo netplan apply

Task 27 — Add a temporary IP address to an interface without persisting it

$ sudo ip addr add 192.168.1.60/24 dev eth0
$ sudo ip link set eth0 up

Task 28 — Verify which process is listening on a given port

$ sudo ss -tulpn | grep :443

Task 29 — Test whether a remote port is reachable through the firewall

$ nc -zv 192.168.1.10 443

Task 30 — Configure a persistent hostname and matching /etc/hosts entry

$ sudo hostnamectl set-hostname webserver01
$ echo "127.0.1.1 webserver01" | sudo tee -a /etc/hosts

Task 31 — Set up SSH key-based login and disable password auth

$ ssh-keygen -t ed25519 -C "ahmad@ahmadtec.com"
$ ssh-copy-id ahmad@192.168.1.10
$ sudo sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
$ sudo systemctl restart ssh

Task 32 — Route traffic for a specific subnet through a different gateway

$ sudo ip route add 10.10.0.0/24 via 192.168.1.254 dev eth0

Task 33 — Capture and inspect traffic on a specific port for debugging

$ sudo tcpdump -i eth0 port 443 -w capture.pcap -c 100

Package Management

Task 34 — Install a package and hold it at its current version

$ sudo apt install nginx
$ sudo apt-mark hold nginx
$ apt-mark showhold

Task 35 — Find which package owns a specific file

$ dpkg -S /usr/sbin/nginx

Task 36 — Downgrade a package to a previous version from the apt cache

$ apt list -a nginx                      # list available versions
$ sudo apt install nginx=1.24.0-1ubuntu1

Task 37 — Fix a system with broken dependencies after a failed install

$ sudo dpkg --configure -a
$ sudo apt --fix-broken install

Task 38 — Add a third-party APT repository and install from it

$ curl -fsSL https://example.com/gpgkey.asc | sudo gpg --dearmor -o /usr/share/keyrings/example.gpg
$ echo "deb [signed-by=/usr/share/keyrings/example.gpg] https://repo.example.com stable main" | sudo tee /etc/apt/sources.list.d/example.list
$ sudo apt update
$ sudo apt install examplepackage

Logs, Cron & Scheduling

Task 39 — Schedule a nightly backup job with cron

$ crontab -e
\# add:
30 2 * * * /home/ahmad/scripts/backup.sh >> /home/ahmad/backup.log 2>&1

Task 40 — Create a systemd timer as a more robust cron alternative

See full backup.service / backup.timer example in Chapter 14.
$ sudo systemctl enable --now backup.timer
$ systemctl list-timers

Task 41 — Search authentication logs for failed SSH login attempts

$ sudo grep "Failed password" /var/log/auth.log | tail -20

Task 42 — Rotate a large application log manually

$ sudo mv /var/log/myapp.log /var/log/myapp.log.1
$ sudo touch /var/log/myapp.log
$ sudo systemctl reload myapp    # or send SIGHUP if the app supports it

Task 43 — Configure logrotate for a custom application log

/etc/logrotate.d/myapp:
/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
}

Task 44 — View all errors logged since the last reboot

$ journalctl -p err -b

Security & Troubleshooting

Task 45 — Configure ufw to allow only SSH, HTTP, and HTTPS

$ sudo ufw default deny incoming
$ sudo ufw allow 22/tcp
$ sudo ufw allow 80/tcp
$ sudo ufw allow 443/tcp
$ sudo ufw enable

Task 46 — Install and configure fail2ban for SSH protection

$ sudo apt install fail2ban
$ sudo systemctl enable --now fail2ban
$ sudo fail2ban-client status sshd

Task 47 — Find and remove world-writable files as a security audit step

$ sudo find / -xdev -perm -0002 -type f 2>/dev/null
$ sudo chmod o-w /path/to/offending/file

Task 48 — Diagnose why a web service returns "connection refused"

$ systemctl status nginx
$ sudo ss -tulpn | grep :80
$ sudo ufw status
$ curl -v http://localhost

Task 49 — Recover root access using rescue/single-user mode

At GRUB menu: press 'e' on the default entry, append to the linux line:
rd.break                      # or: init=/bin/bash on some setups
Boot with Ctrl+X, then remount root read-write and reset the password:
mount -o remount,rw /sysroot
chroot /sysroot
passwd root
touch /.autorelabel   # SELinux systems only
exit
reboot

Task 50 — Verify overall system health after applying a batch of changes

$ systemctl --failed
$ journalctl -p err -b --no-pager | tail -30
$ df -h
$ free -h
$ uptime
↑ Back to top

Final AppendixQuick-Reference Cheat Sheets

A consolidated set of at-a-glance references for the material covered throughout this handbook — ideal for printing separately or keeping open in a second window during hands-on work.

Top Commands Every Administrator Should Know

Organized by category rather than a flat top-200 list, so it stays usable as a lookup tool rather than a wall of text.
CategoryCommands
Navigationpwd, ls, cd, tree, find, locate, which, whereis, realpath, basename, dirname
File Opstouch, cp, mv, rm, mkdir, rmdir, ln, stat, file, cat, tac, less, more, head, tail, nl, wc, sort, uniq, cut, paste, split, tee, shred, truncate, rename
Permissionschmod, chown, chgrp, umask, getfacl, setfacl
Users & Groupswhoami, id, groups, useradd, adduser, usermod, passwd, userdel, groupadd, groupdel, groupmod, su, sudo, visudo, last, who, w, users, chage
Packagesapt, apt-get, dpkg, snap, apt-cache, apt-mark
Processesps, top, htop, kill, killall, pkill, nice, renice, jobs, bg, fg, nohup, disown
Servicessystemctl, journalctl, systemd-analyze
Networkingip, ifconfig, ss, netstat, ping, traceroute, tracepath, dig, nslookup, host, curl, wget, scp, rsync, ssh, nc, tcpdump, arp, hostnamectl, nmcli
Diskslsblk, blkid, fdisk, parted, mount, umount, findmnt, df, du, fsck, mkfs, tune2fs, swapon, swapoff, free, lvextend, resize2fs
Logsjournalctl, tail, dmesg, grep
Text Processinggrep, egrep, sed, awk, xargs, diff, cmp, comm, strings
Compressiontar, gzip, gunzip, zip, unzip, xz, bzip2
Schedulingcron, crontab, at, atq, atrm, systemctl (timers)
Environmentexport, env, printenv, alias, unalias, source
SSHssh, ssh-keygen, ssh-copy-id, sftp, scp
Monitoringuptime, vmstat, iostat, mpstat, sar, iotop, free, lscpu, lsmem, uname
Securityufw, iptables, fail2ban-client, getenforce, setenforce, aa-status
Bootupdate-grub, grub2-mkconfig, systemctl get-default/set-default/isolate

Terminal Keyboard Shortcuts

ShortcutAction
Ctrl + CKill the current foreground process (SIGINT)
Ctrl + ZSuspend the current foreground process (SIGTSTP)
Ctrl + DSend EOF / close the current shell
Ctrl + LClear the screen (like clear)
Ctrl + AMove cursor to start of line
Ctrl + EMove cursor to end of line
Ctrl + UDelete from cursor to start of line
Ctrl + KDelete from cursor to end of line
Ctrl + WDelete the word before the cursor
Ctrl + RReverse-search command history
TabAuto-complete command/filename
!!Repeat the last command
!$Last argument of the previous command

Vim Basics

Mode / KeyAction
iEnter insert mode before cursor
EscReturn to normal mode
:wSave
:qQuit
:wq or ZZSave and quit
:q!Quit without saving
ddDelete (cut) current line
yyYank (copy) current line
pPaste after cursor
/patternSearch forward
n / NNext / previous search match
:%s/old/new/gReplace all occurrences in the file
uUndo
Ctrl + rRedo
gg / GGo to first / last line

Nano Basics

ShortcutAction
Ctrl + OSave (Write Out)
Ctrl + XExit
Ctrl + KCut current line
Ctrl + UPaste
Ctrl + WSearch
Ctrl + \Search and replace
Ctrl + GHelp menu
Ctrl + CShow cursor position

Regular Expressions Cheat Sheet

PatternMatches
.Any single character
*Zero or more of the preceding character
+One or more (extended regex / egrep)
?Zero or one (extended regex)
^Start of line
$End of line
[abc]Any one of a, b, or c
[^abc]Any character except a, b, or c
[0-9]Any digit
[a-zA-Z]Any letter
\dDigit (PCRE / grep -P)
\wWord character (PCRE)
\sWhitespace (PCRE)
(abc\|xyz)abc or xyz (basic regex, escaped pipe)
a{2,4}Between 2 and 4 repetitions of "a"
$ grep -E "^[0-9]{3}-[0-9]{4}$" phones.txt     # extended regex example
$ grep -P "\d+\.\d+\.\d+\.\d+" access.log       # Perl regex, matches an IPv4

Exit Status Codes

CodeMeaning
0Success
1General error
2Misuse of shell built-in / incorrect command usage
126Command found but not executable (permission issue)
127Command not found
128+NProcess terminated by signal N (e.g. 137 = 128+9 = killed by SIGKILL)
130Terminated by Ctrl+C (SIGINT, 128+2)
$ echo $?    # check the exit code of the last command

Common Signals

SignalNumberMeaning
SIGHUP1Hangup; many daemons treat this as "reload configuration"
SIGINT2Interrupt from keyboard (Ctrl+C)
SIGKILL9Immediate, unblockable termination — process cannot clean up
SIGTERM15Polite request to terminate; the default signal sent by kill
SIGSTOP19Pause the process, cannot be caught or ignored
SIGTSTP20Terminal stop (Ctrl+Z); can be caught, unlike SIGSTOP
SIGCONT18Resume a stopped process
$ kill -15 1234     # SIGTERM, graceful (default)
$ kill -9 1234      # SIGKILL, forceful last resort
Always try SIGTERM first and give the process a moment to shut down cleanly (closing files, finishing transactions). Reach for SIGKILL only if it doesn't respond.

File Permission Cheat Sheet

OctalSymbolicCommon Use
777rwxrwxrwxAvoid — full access for everyone
755rwxr-xr-xExecutables, scripts, directories
750rwxr-x---Directory shared within a group only
644rw-r--r--Standard config/data files
640rw-r-----Config files with sensitive data, group-readable
600rw-------Private keys, credentials
400r--------Read-only sensitive files

Systemctl Cheat Sheet

CommandPurpose
systemctl status UNITShow current state and recent logs
systemctl start/stop/restart UNITControl running state now
systemctl reload UNITReload config without downtime (if supported)
systemctl enable/disable UNITControl boot-time behavior
systemctl enable --now UNITEnable and start in one step
systemctl mask/unmask UNITFully block/unblock a unit from starting
systemctl daemon-reloadReload unit file changes
systemctl list-units --type=serviceList loaded services
systemctl --failedList failed units
systemctl get-default / set-defaultView/set default boot target

Journalctl Cheat Sheet

CommandPurpose
journalctl -u UNITLogs for one unit
journalctl -fFollow live, system-wide
journalctl -u UNIT -fFollow live, one unit
journalctl -bLogs since current boot
journalctl -kKernel messages only
journalctl -p errFilter by priority
journalctl --since "1 hour ago"Time-based filter
journalctl --disk-usageJournal size on disk
journalctl --vacuum-time=7dTrim logs older than 7 days

Networking Cheat Sheet

CommandPurpose
ip aShow interfaces and IP addresses
ip route showShow routing table
ss -tulpnShow listening ports and owning processes
ping -c4 HOSTTest reachability
dig DOMAIN +shortQuick DNS lookup
curl -I URLFetch HTTP headers only
scp FILE user@host:/pathSecure copy a file
rsync -avz SRC DESTEfficient sync
nc -zv HOST PORTTest if a port is open
tcpdump -i IFACE port NCapture packets on a port

Disk Management Cheat Sheet

CommandPurpose
lsblkList block devices and mount points
blkidShow filesystem UUIDs and types
df -hDisk space per filesystem
du -sh DIRTotal size of a directory
mount / umountAttach / detach a filesystem
mkfs.ext4 DEVICEFormat a partition
fsck DEVICECheck/repair a filesystem (unmounted only)
swapon / swapoffEnable / disable swap
free -hMemory and swap summary
End of Reference — Linux Ubuntu System Administration Handbook
↑ Back to top