Linux — Cheat Sheet

Three permission sets — owner, group, and other — each with read, write and execute bits. For a file, read means view contents, write means modify, execute means run. For a directory the meanings differ and this is where confusion lives. Read lets you list the names in it. Write lets you create and delete entries — which means you can delete a file you cannot write, because deletion modifies the directory rather than the file. Execute lets you traverse into it and access files by name. So a directory with execute but not read allows accessing a known path while preventing listing, which is a legitimate pattern. The numeric form is octal: read is 4, write 2, execute 1, so 755 is full for the owner and read-execute for everyone else, and 644 is the usual for a regular file. The special bits: setuid runs a binary as its owner, setgid on a directory makes new files inherit the group, and the sticky bit on a shared directory such as /tmp means only the owner can delete their own files. umask determines the default permissions for new files.

An inode holds a file's metadata and the pointers to its data blocks: type, permissions, owner, size, timestamps, link count, and block addresses. What it does not hold is the filename. Names live in directory entries, which map a name to an inode number — and that separation explains several behaviours. Hard links are multiple directory entries pointing at the same inode, so they share content and permissions and deleting one does not remove the file. A file can be unlinked while still open: the directory entry goes, but the inode survives until both the link count and the open-file count reach zero. That is why deleting a large log file does not free disk space if a process still holds it open — df shows the disk full and du cannot account for it. lsof +L1 finds those files, and truncating through /proc/PID/fd frees the space without a restart. Inodes are a finite resource allocated at filesystem creation, so a partition can report "no space left on device" with free bytes but no free inodes — usually caused by millions of tiny files. df -i shows inode usage.

A hard link is an additional directory entry pointing at the same inode. A symbolic link is a small file whose contents are a path. Hard links are indistinguishable from the original — there is no original, just several names for one inode. Removing one leaves the data until the link count reaches zero. They cannot cross filesystems, because inode numbers are only meaningful within one, and conventionally cannot point at directories since that would allow cycles. Symlinks store a path, so they can cross filesystems and point at directories, but they break if the target moves or is deleted, leaving a dangling link. Resolving one costs an extra lookup. The practical distinction is what happens on deletion, and it is why deployment schemes use a symlink to a versioned directory — switching releases is repointing the link, which is atomic. The operational gotchas: many tools need -L or -h to follow or not follow symlinks, and getting that wrong in a backup or a recursive delete has consequences. And a relative symlink resolves relative to its own location, not to your working directory.

df -h shows usage per filesystem, which tells you which one is full — and checking that first matters, because the full filesystem is often not the one you assumed. Then du to find the large directories: du -h --max-depth=1 on the mount point, then descend into whichever is largest. ncdu is much nicer if available, giving an interactive browser. The cases where du and df disagree are the interesting ones. Deleted-but-open files: space is consumed but no directory entry exists, so du cannot see it. lsof +L1 lists them, and restarting the holding process or truncating through /proc frees it. This is the classic "I deleted the logs and nothing happened". A filesystem mounted over a non-empty directory hides the underlying files, which still consume space. Inode exhaustion, where df -i shows full but df -h does not. And reserved blocks — ext4 reserves 5% for root by default, so a filesystem can be unusable for a normal user while df shows space available.

/proc is a virtual filesystem exposing kernel and process information as files. Nothing is on disk — reading a file there executes kernel code. /proc/PID gives everything about a process: its command line, environment, open file descriptors, memory maps, status and limits. /proc/meminfo, /proc/cpuinfo and /proc/loadavg expose system state. /sys exposes the kernel's device model — hardware, drivers and their tunable parameters — in a more structured hierarchy. cgroup limits live under /sys/fs/cgroup, which is where you read what a container is actually constrained to. /dev holds device files: block devices for disks, character devices for terminals and things like /dev/null and /dev/urandom. The practical value for a backend engineer is that /proc is the source of truth behind almost every diagnostic tool. When you are in a minimal container without ps, top or lsof, you can read /proc directly — /proc/PID/status for memory, /proc/PID/fd for open files, /proc/PID/limits for ulimits. That is frequently the difference between diagnosing an incident and waiting for a package install.

Operations that need a new descriptor fail with EMFILE — "too many open files" — so the process cannot open files, accept connections, or make outbound calls. For a server the symptom is that it stops accepting connections while continuing to serve existing ones, which looks like a hang rather than a crash. Every socket, file, pipe and epoll instance is a descriptor, so a service handling many connections needs a high limit. The limits are per process, with a soft limit that is enforced and a hard limit that caps how high the soft limit may be raised. ulimit -n shows the soft limit; /proc/PID/limits shows what a running process actually has. The confusion in production is that raising the limit in a shell does not affect an already-running service, and systemd units ignore /etc/security/limits.conf entirely — LimitNOFILE in the unit file is what applies. That mismatch is why the setting appears to have no effect. The usual root cause is a leak: connections or files not closed on an error path. lsof -p or counting /proc/PID/fd over time reveals it, and the count growing steadily is the signature.

find for locating files by attributes: name, type, size, modification time, permissions, owner. It walks the tree, so it is accurate and can be slow on a large filesystem. The combinations worth knowing: find with -mtime for age, -size for large files, and -exec or -delete to act on results. Using -print0 with xargs -0 handles filenames with spaces, which the naive form breaks on. locate is much faster because it queries a prebuilt index, at the cost of being stale between updates. For content, grep -r is the baseline, but ripgrep is dramatically faster, respects gitignore, and skips binary files by default — it is worth installing everywhere. The flags that matter: -i for case-insensitive, -n for line numbers, -A and -B for context, -l for filenames only, and -v to invert. The practical combination for an incident is finding recently-modified files — find with -mmin — which answers "what changed just before this broke" faster than reading logs. And for a container, remember the tools may not exist; grep is usually present, ripgrep usually is not.

Mounting attaches a filesystem to a directory in the tree, so paths below that point resolve into it. The consequence people forget is that mounting over a non-empty directory hides its contents. The files still exist and consume space, but are inaccessible until unmounted — which is a real source of "the disk is full but I cannot find anything". A bind mount makes an existing directory appear at a second location. It is not a copy: both paths are the same inodes, so a change through one is visible through the other. Bind mounts are how Docker volume mounts work, which is why a bind-mounted config file edited on the host changes inside the container immediately. The options worth knowing: ro for read-only, noexec to prevent running binaries from a data mount, and nosuid. /etc/fstab defines mounts at boot; a malformed entry can prevent boot entirely, which is why testing with mount -a before rebooting matters. And unmounting fails with "target is busy" if any process has a file open or a working directory inside — lsof or fuser identifies the culprit.

logrotate, which rotates, compresses and removes old logs on a schedule defined per service in /etc/logrotate.d. The important configuration decision is copytruncate versus create. With create, logrotate renames the file and signals the process to reopen — which is correct but requires the process to handle the signal. With copytruncate, it copies the content and truncates the original in place, which works for processes that do not reopen but has a small window where writes are lost. A process that keeps writing to the renamed file is the classic failure: logs appear to stop, disk keeps filling, and du shows nothing because the file was deleted while held open. For systemd services, journald handles it with its own size limits configured in journald.conf — and an unbounded journal filling /var/log is a common oversight. The better approach in a container is not to write log files at all. Log to stdout and let the platform collect them, which removes rotation from your concern entirely — and writing files inside a container is an anti-pattern for exactly this reason.

The soft limit is what is currently enforced. The hard limit is the ceiling the soft limit may be raised to. An unprivileged process can raise its soft limit up to the hard limit, and can lower either, but only a privileged process can raise the hard limit. That lets an administrator set a maximum while letting applications opt into more within it. ulimit -n shows the soft limit and ulimit -Hn the hard one. For a running process, /proc/PID/limits shows both, which is the authoritative answer when you are wondering what actually applies. The limits that matter for services: open files, which bounds connections; max user processes, which bounds threads; and core file size, which determines whether you get a core dump on a crash. The production confusion is where limits are configured. A shell session uses /etc/security/limits.conf via PAM. A systemd service ignores that entirely and uses LimitNOFILE in the unit. A Docker container uses the daemon default or --ulimit. Setting it in the wrong place is why the change appears to do nothing, and it is one of the most common wasted debugging sessions.

The kernel keeps file contents in memory after reading or writing them, so repeated access is served without touching the disk. It uses all otherwise-idle memory for this. That is why free shows little free memory on a healthy system, and it is not a problem — the cache is reclaimable. When an application needs memory, the kernel drops clean cache pages instantly. So the metric to read is "available", not "free". Available accounts for reclaimable cache and is what tells you whether the system is actually under pressure. Alerting on free memory produces constant false positives. The distinction that matters is clean versus dirty pages. A clean page matches the disk and can be dropped for nothing. A dirty page has unwritten changes and must be flushed first, which is why a large dirty backlog causes latency spikes when writeback triggers. The practical consequences: benchmarks are meaningless unless you account for the cache; your application-level cache may be duplicating it; and databases often use O_DIRECT to bypass it because they manage their own buffer pool better than the kernel can guess.

rsync for anything non-trivial. It transfers only differences, resumes interrupted transfers, preserves permissions and timestamps, and can delete files removed from the source. The flags that matter: -a for archive mode which preserves everything, -v for verbose, -z for compression, -P for progress and partial resume, and --dry-run before anything destructive. The trailing slash on the source is the classic trap: with a slash it copies the directory's contents, without it copies the directory itself. Getting that wrong nests a directory inside itself. --delete makes the destination mirror the source, which is powerful and dangerous — always dry-run it, because a wrong path deletes the destination. scp is simpler for a single file and is fine for that. It has no resume and no delta transfer. For large or repeated transfers over an unreliable link, rsync over ssh with -P is the right tool. And for anything scripted, prefer explicit absolute paths and a dry-run flag controlled by a variable, since the difference between a backup and a deletion is one argument.

A block device is raw storage — a disk, a partition, an LVM volume — presenting a linear array of fixed-size blocks. It has no concept of files. A filesystem is the structure written onto a block device that organises those blocks into files and directories, with metadata, allocation and a namespace. So you create a filesystem on a block device with mkfs, then mount it to make it accessible as a path. The practical relevance: lsblk shows block devices and their mount points, which is how you see what storage exists and whether it is in use. df shows mounted filesystems and their usage. Adding a disk means partitioning it, creating a filesystem, and mounting it — three distinct steps that people conflate. Some things use block devices directly without a filesystem: swap, and some database configurations that manage their own layout. LVM sits between them, aggregating physical devices into a pool from which logical volumes are carved — which is what allows resizing without repartitioning, and is why production servers usually use it. And a loop device presents a file as a block device, which is how disk images are mounted.

fsync forces buffered writes for a file out of the page cache to durable storage, and does not return until the device confirms. It matters because an ordinary write only updates the page cache. The call returns immediately and the data may sit in memory for seconds before writeback. A power loss in that window loses acknowledged writes. So any database claiming durability must fsync its write-ahead log before acknowledging a commit. That is precisely why commits are expensive — you are waiting on physical storage rather than on memory — and why commit latency is bounded by disk fsync latency. The complications are notorious. Some drives lie, acknowledging a flush while data sits in a volatile cache. fdatasync is a cheaper variant skipping metadata when only content changed. And on some filesystems, fsync on a newly created file does not make the directory entry durable, so you must fsync the parent directory too. There was also the case where a failed fsync on Linux could clear the error state so a retry appeared to succeed while data was lost, which changed how PostgreSQL handles fsync failures. The practical implication: disk fsync latency is a first-order database performance metric.

SIGTERM is a polite request to terminate. It can be caught, so a process can flush buffers, close connections, deregister from a load balancer and exit cleanly. It is what kill sends by default and what orchestrators send first. SIGKILL cannot be caught, blocked or ignored. The kernel destroys the process immediately with no chance to clean up. Open files are closed by the kernel, but application state is lost — in-flight requests dropped, buffers unflushed. SIGSTOP suspends without terminating and also cannot be caught; SIGCONT resumes. Ctrl+Z sends SIGTSTP, which unlike SIGSTOP can be caught. The operational sequence matters. Kubernetes sends SIGTERM, waits for the termination grace period, then sends SIGKILL. If your application ignores SIGTERM or takes too long, it is killed mid-request — which appears to users as errors during every deployment. So handling SIGTERM properly is what makes a rolling deployment invisible: stop accepting new work, finish in-flight requests, close resources, exit. And a process stuck in uninterruptible sleep on I/O cannot be killed even by SIGKILL until the I/O completes.

The states shown by ps: R running or runnable, S interruptible sleep, D uninterruptible sleep, T stopped, Z zombie. S is the normal waiting state — blocked on something but able to be woken by a signal. Most idle processes are here. D is uninterruptible sleep, usually waiting on disk I/O. The process cannot be killed, not even with SIGKILL, until the operation completes. A process stuck in D is the signature of a storage problem — a failing disk, an unresponsive NFS mount, or an overloaded device. That matters because Linux includes D-state processes in the load average. So a machine with a failing disk can show a load average of 50 with near-zero CPU utilisation, which is confusing unless you know the definition. Z is a zombie: the process has exited but its parent has not collected the exit status. It consumes no resources except a process table entry, and accumulating them exhausts the PID table. The practical reading: many D-state processes means look at storage; many zombies means a parent is not reaping, which in containers usually means PID 1 is your application rather than an init.

A zombie has finished executing but its parent has not called wait to collect the exit status, so the kernel keeps a minimal entry holding that status. It consumes no memory or CPU — only a process table slot. That sounds harmless until a buggy parent leaks thousands and exhausts the PID table, at which point nothing on the machine can fork. You cannot kill a zombie, because it is already dead. The fix is to make the parent reap it: send the parent SIGCHLD, or fix the parent to call wait. If the parent will not cooperate, killing the parent works — the zombies are then re-parented to init, which reaps them properly. The container-specific version matters more for backend engineers. A container's PID 1 is usually your application, not a real init, and most applications do not reap adopted children. So any process that spawns subprocesses accumulates zombies inside the container. That is exactly why docker run --init exists, and why tini is bundled — it provides a minimal init as PID 1 that reaps correctly. An orphan, by contrast, is not a leak: it is re-parented to init and reaped normally.

ss -tlnp shows listening TCP sockets with the owning process. The flags are -t for TCP, -l for listening, -n to skip DNS resolution which is slow and noisy, and -p for the process, which requires privileges to see other users' processes. netstat -tlnp is the older equivalent and is often absent on modern systems, since ss replaced it and is considerably faster because it reads netlink rather than parsing /proc. lsof -i :8080 answers the same question and also shows established connections to that port. fuser -n tcp 8080 is the terser version. The practical case is "address already in use" on startup, where you need to know what is holding the port — usually a previous instance that did not exit, or a second copy started by mistake. The related diagnostic is counting connection states: ss -s gives a summary, and a large number in TIME_WAIT or CLOSE_WAIT points at specific problems — CLOSE_WAIT in particular means the application is not closing sockets, which is a leak. In a container, the network namespace is separate, so you must run these inside it.

The number of processes that are runnable or in uninterruptible sleep, averaged over one, five and fifteen minutes. The inclusion of uninterruptible sleep is Linux-specific and is what makes load average different from other Unixes — it means load reflects I/O wait as well as CPU demand. So it must be read relative to core count. A load of 8 on an 8-core machine means fully utilised without queueing; the same on a 2-core machine means significant queueing. And it must be read alongside CPU utilisation. High load with high CPU means genuinely CPU-bound. High load with low CPU means processes are blocked — usually on disk, sometimes on a lock. A machine with a failing disk can show a load of 50 at near-zero CPU. The three windows show trend: a one-minute figure far above the fifteen-minute one means a spike is developing; the reverse means it is subsiding. In a container, the load average is the host's, not the container's, which makes it misleading in Kubernetes — cgroup CPU statistics are the correct source there.

top or htop sorted by CPU, then narrow to threads with top -H -p PID, since a multi-threaded process shows aggregate usage and the interesting question is which thread. For a JVM, convert the thread ID to hexadecimal and match the nid in a jstack dump to get the Java stack. For Python, py-spy dump attaches and shows the stack without modifying the process. For native code, perf top gives a live profile by symbol, and perf record with a flame graph gives the full picture. pidstat -u gives non-interactive per-process figures suitable for scripting. The distinction to establish early is whether the process is doing useful work or spinning. A retry loop, a busy-wait, or a regular expression with catastrophic backtracking all look identical to legitimate computation from the outside — only the stack tells you. The other split worth reading is user versus system time. High system time means excessive syscalls or context switching, which is a different problem from high user time in application code, and points at I/O patterns rather than algorithms.

strace traces the system calls a process makes, showing each call with its arguments, return value and optionally timing. It answers questions about the boundary between an application and the kernel: which file did it fail to open, what address is it connecting to, why is startup slow, is it making thousands of tiny writes instead of buffering. The most useful invocation is often strace -c, which summarises counts and time per syscall and immediately shows whether one call dominates. -f follows children, -T shows time per call, and -e trace= filters to specific calls. When to avoid it: production, on a busy process. strace uses ptrace and stops the process on every syscall, which can slow it by an order of magnitude and can change timing-dependent behaviour. Attaching to a busy service has caused outages. The modern alternatives are perf trace and eBPF tools such as bpftrace, which give similar visibility at a fraction of the cost and are safe on live systems. And strace shows syscalls, not application logic — a process spinning in userspace shows nothing at all, which confuses people expecting it to reveal everything.

The kernel schedules tasks, and both processes and threads are tasks. The difference is what they share. A process has its own address space, file descriptor table and signal handlers. Threads within a process share the address space, file descriptors and most other state, but each has its own stack, registers and program counter. Both are created by clone; the flags determine what is shared. fork is clone with nothing shared; thread creation is clone with almost everything shared. That unification is why ps -T shows threads and why each thread has its own entry in /proc/PID/task. The practical consequences. Context switching between threads is cheaper because the address space does not change, so the TLB and much of the cache stay warm. Threads communicate through shared memory, which is fast but means every shared mutable variable is a potential race. Processes are isolated, so a crash contains, but they must communicate through pipes, sockets or shared memory segments. And resource limits such as max processes count threads too, so a thread-heavy application can hit a limit that sounds like it should not apply.

nohup prevents SIGHUP from reaching it when the terminal closes, and redirects output to a file. Combined with & it backgrounds the process. A better approach for interactive work is tmux or screen, which keep a session alive that you can detach from and reattach to later — so you can check on the process, see its output, and interact with it. That is much more useful than nohup for anything long-running you want to supervise. setsid detaches the process from the terminal entirely by putting it in a new session. But for anything that should run permanently, none of these are right. A systemd unit is the correct answer: it gives automatic restart on failure, start on boot, log integration through journald, resource limits, and dependency ordering. Using nohup for a production service is a common shortcut that fails the first time the machine reboots or the process crashes, with nobody noticing. And in a containerised environment, the container itself provides that supervision, so the process runs in the foreground and the orchestrator handles restart — which is why running a daemon inside a container is an anti-pattern.

When the kernel cannot satisfy an allocation and cannot reclaim enough memory, the out-of-memory killer terminates a process rather than letting the system fail. It scores processes primarily by memory consumption relative to the total, adjusted by oom_score_adj which ranges from -1000 to +1000, and kills the highest. That heuristic usually kills the biggest consumer, which is often your main application rather than whatever actually caused the pressure — a frequent source of confusion when a JVM is killed by a memory spike elsewhere. Finding out it happened: dmesg or the kernel log shows an "Out of memory: Killed process" line with the PID, name and memory figures. journalctl -k finds it on systemd systems. The application itself logs nothing, because SIGKILL cannot be handled — so a service that "just disappeared" with no error in its logs is the classic signature. In containers, the cgroup memory limit triggers a cgroup-scoped kill, which is what exit code 137 means in Kubernetes. The evidence is in the kernel log on the node, not in the pod logs, which is why it is often misdiagnosed.

The nice value ranges from -20 to 19, with lower meaning higher priority. Unprivileged users can only increase it, making a process less important; lowering it requires privileges. Under the Completely Fair Scheduler, nice acts as a weight on how quickly a task accumulates virtual runtime — a high nice value accrues virtual time faster and is therefore chosen less often. It is not a strict priority; a niced process still runs, just less. renice adjusts a running process. The practical uses are narrow: running a batch job or a backup at a high nice value so it uses only idle capacity. The more useful sibling is ionice, which sets I/O scheduling priority. A backup or a large copy can saturate the disk and cause latency for everything else even at a high nice value, because CPU priority does not affect I/O queuing. ionice -c3 puts it in the idle class. For real isolation, cgroups are the correct mechanism — CPU shares and I/O throttling per group — which is what containers use and what actually enforces limits rather than merely biasing the scheduler.

/proc is the starting point and needs no tools: /proc/PID/status for memory and thread counts, /proc/PID/limits for effective ulimits, /proc/PID/environ for the environment it was started with, /proc/PID/cwd and /proc/PID/exe as symlinks to its directory and binary, and /proc/PID/fd for open descriptors. That last one is particularly useful — counting entries reveals a descriptor leak, and reading the symlinks shows what is open, including deleted-but-held files. lsof -p gives the same in a friendlier form. For stacks: py-spy dump for Python and jstack for a JVM, both of which attach without stopping the process meaningfully. gdb can attach for native code but suspends it. For syscalls, prefer perf trace or bpftrace over strace on anything busy. For I/O, pidstat -d shows per-process read and write rates, which iostat cannot since it is per-device. The general principle is that Linux exposes almost everything through /proc, so when the convenient tool is missing — which is the normal case in a minimal container — you can still get the answer by reading files.

fork creates a near-identical copy of the calling process, returning twice — zero in the child and the child's PID in the parent, which is how each knows which it is. exec replaces the current process image with a new program, keeping the same PID. It does not return on success, because the code that called it no longer exists. The separation looks odd but is useful: between the fork and the exec, the child is still running your code and can adjust the environment the new program will inherit — redirect stdin and stdout, close descriptors, change directory, drop privileges. That is exactly how a shell implements pipes and redirection. A combined spawn call would have to expose every one of those adjustments as a parameter. The cost concern is answered by copy-on-write: fork does not physically copy memory, it marks pages shared and read-only and copies only on write. Since exec discards the address space immediately, almost nothing is ever copied. The practical relevance is understanding process trees, why a shell script spawns so many processes, and why fork in a large process can still spike memory when the child writes.

The kernel gives PID 1 two special responsibilities that application processes generally do not implement. First, reaping orphans. When a process dies and its parent is gone, it is re-parented to PID 1, which must call wait to collect the exit status. An application that does not accumulates zombie processes until the PID table is exhausted. Second, signal handling differs. The kernel does not apply default signal actions to PID 1, so a process that has not installed a SIGTERM handler simply ignores it. The container then never shuts down gracefully and is SIGKILLed after the grace period — which shows up as slow, unclean deployments and dropped requests. A third practical problem is shell form in Dockerfiles: CMD in shell form runs your process under /bin/sh, which becomes PID 1 and does not forward signals to its child. The fixes: use exec form so your process is PID 1 directly, install a real SIGTERM handler, and use --init or tini when the process spawns children. This is one of the most common containerisation mistakes and it is invisible until deployments start dropping requests.

Two separate output streams: file descriptor 1 for normal output and 2 for errors and diagnostics. The separation exists so you can pipe results while still seeing errors, or capture them independently. The redirections: > sends stdout to a file, 2> sends stderr, and &> or > file 2>&1 sends both. Appending uses >>. The ordering trap is the classic: command > file 2>&1 works, but command 2>&1 > file does not do what people expect. Redirections are applied left to right, so in the second form stderr is pointed at the terminal — where stdout currently goes — and only then is stdout moved to the file. Errors still appear on screen. 2>/dev/null discards errors, which is sometimes right and often hides the thing you needed to see. The practical guidance for scripts: write diagnostics to stderr so they do not contaminate piped output, and log errors somewhere rather than discarding them. And remember a pipe only carries stdout, so a command's errors bypass the pipeline entirely unless redirected.

A pipe connects one process's stdout to another's stdin. The shell creates the pipe, forks both processes, and they run concurrently — not sequentially — with the kernel handling the buffering. That concurrency matters: a pipeline processing a large file starts producing output before the first command finishes. The exit status of a pipeline is that of the last command by default. So a pipeline where an early command fails but the last succeeds reports success, which silently hides failures — a very common scripting bug. set -o pipefail changes it to return the rightmost non-zero status, which is almost always what you want in a script. PIPESTATUS is a bash array holding every command's status if you need them individually. The other detail is the buffer: a pipe holds around 64KB, after which the writer blocks until the reader consumes. A reader that never drains deadlocks the writer, which is a frequent bug when spawning subprocesses from code. And each stage of a pipeline runs in a subshell in bash, so variables set inside a while-read loop fed by a pipe do not survive — which surprises people constantly.

set -euo pipefail, and a shebang. -e exits on any command failure, so the script does not continue after something went wrong. -u errors on an undefined variable, catching typos rather than substituting an empty string — which is how rm -rf "$PREFIX/" becomes rm -rf / when the variable is misspelled. -o pipefail makes a pipeline fail if any stage fails. Together they turn silent partial failures into loud stops, which is the single highest-value habit in shell scripting. The caveats worth knowing: -e has surprising exceptions — it does not trigger inside a condition, or for a command followed by || — so it is not a substitute for checking. And it can make a script exit where you intended to handle a failure, which is what || true is for. Beyond that: quote every variable expansion, because unquoted variables word-split and glob on spaces. Use "${var}" consistently. Prefer $() over backticks. Use mktemp for temporary files and trap to clean up on exit. And run shellcheck, which catches most of these automatically and is worth putting in CI.

An unquoted expansion undergoes word splitting and glob expansion. So a variable containing a space becomes two arguments, and one containing an asterisk expands against the filesystem. The practical consequences are severe. A filename with a space breaks a loop. A variable that is empty disappears entirely rather than becoming an empty argument, so a command receives fewer arguments than expected and does something different. And a variable containing a glob character can match unintended files. The rm example is the memorable one: rm -rf $DIR/* with an empty or misspelled DIR becomes rm -rf /*. So the rule is to quote every expansion — "$var", "$@", "${array[@]}" — unless you specifically want splitting, which is rare and should be deliberate. "$@" versus "$*" is the related distinction: "$@" preserves each argument separately, "$*" joins them into one string. Passing arguments through a wrapper script requires "$@", and getting it wrong breaks any argument containing a space. shellcheck flags all of this, which is why running it is worth more than remembering the rules.

grep filters lines by pattern. Use it to select. cut extracts fields by delimiter or character position. Simple and fast, but it cannot handle repeated delimiters or quoted fields, so it breaks on real CSV and on whitespace-aligned output. sed edits streams — substitution, deletion, insertion — with substitution being the overwhelmingly common use. Good for a targeted replacement across a stream or file. awk processes structured text by fields and can do arithmetic, conditionals and aggregation. It splits on whitespace by default handling repeated separators correctly, which is exactly where cut fails. The rough decision: grep to filter, cut for simple field extraction with a single-character delimiter, sed for substitution, awk when you need logic, arithmetic, or column handling that cut cannot do. awk is the one worth investing in — summing a column, counting by key, printing conditionally on a field value — because it replaces a surprising number of small scripts with one line. The caution is that all of these are line-oriented and none understand structure, so for JSON use jq and for CSV with quoting use a real parser. Parsing JSON with grep is a recurring mistake.

The standard pipeline is filter, extract, count, sort. grep to narrow to the relevant lines. awk or cut to pull the field you care about. sort then uniq -c to count occurrences. sort -rn to rank them. That combination answers most questions: which endpoints error most, which IPs are hitting you hardest, which status codes appear. For a live view, tail -f piped through grep, or less +F which lets you stop following and search. For time-bounded analysis, awk comparing on the timestamp field is usually easier than trying to express it in grep. The things that trip people up: uniq only collapses adjacent duplicates, so it must be preceded by sort — forgetting that gives wrong counts. And grep -c counts lines, not occurrences, so multiple matches on one line count once. For structured logs, jq is the right tool and treating JSON as text is not — jq can select, filter and aggregate properly. And for anything you do more than twice, it is worth putting in a script, because the pipeline you reconstruct under incident pressure is the one you get wrong.

Single quotes are literal: nothing inside is expanded — no variables, no command substitution, no escapes except that you cannot include a single quote. Double quotes allow variable expansion, command substitution and backslash escapes, while still preventing word splitting and globbing. So you use double quotes when you want the value of a variable, and single quotes when you want the text exactly as written. The practical cases. A regex or an awk program should be in single quotes, because $1 in awk means a field and would otherwise be expanded by the shell into a positional parameter — usually empty, which makes the program silently wrong. A password or a string containing dollar signs or backticks must be single-quoted, or the shell interprets them. A path containing a variable needs double quotes. The rule that follows: default to double quotes around variables, and single quotes around anything meant to be passed through literally. And unquoted is almost never correct — the cases where you want word splitting are rare enough to deserve a comment when you rely on it.

$1 through $9 are positional parameters, $0 is the script name, $# is the count, and "$@" is all of them preserving separation. For anything beyond two or three arguments, getopts handles flags properly — parsing short options, handling arguments to options, and reporting unknown flags. Hand-rolled parsing with a case statement in a while loop is common and works for long options, which getopts does not support in bash. The practices that matter: validate that required arguments are present and fail with a usage message rather than proceeding with empty values. Provide -h. Use "$@" rather than $* when passing arguments through, or anything containing a space is split. Set defaults with "${1:-default}", which substitutes when unset or empty. And shift consumes arguments as you process them, which is what makes the while loop pattern work. The broader judgement: once a script has several flags, subcommands, and validation, it has outgrown bash. Rewriting it in Python with argparse is usually the right call, because error handling and data structures in shell become unmaintainable quickly.

They differ in which startup files are read, which is why an environment variable set in one place is missing in another. A login shell — SSH, or a console login — reads /etc/profile and then the first of ~/.bash_profile, ~/.bash_login or ~/.profile. An interactive non-login shell — opening a new terminal in a desktop session — reads ~/.bashrc. A non-interactive shell running a script reads neither by default; it only reads whatever BASH_ENV points at, which is usually nothing. That is why a PATH addition in .bashrc works in your terminal and not in a cron job or a systemd service, which is one of the most common environment confusions. The convention that avoids most of it is to put environment setup in .profile and have .bash_profile source .bashrc, so both paths converge. The practical rule for anything automated: never rely on shell startup files. A cron entry, a systemd unit or a CI job should set its environment explicitly, use absolute paths for binaries, and not assume PATH contains anything beyond the defaults. Debugging "it works when I run it manually" almost always comes down to this.

cron for simple recurring jobs, with a crontab entry specifying minute, hour, day of month, month and day of week. The things that catch people: cron runs with a minimal environment and a nearly empty PATH, so a script that works in your shell fails under cron. Use absolute paths and set any needed variables explicitly. Output is emailed by default, which usually goes nowhere — so redirect stdout and stderr to a log or you lose all diagnostics. There is no built-in locking, so a job that takes longer than its interval overlaps with itself. flock is the standard fix. And cron has no retry, no alerting on failure, and no record of whether a run succeeded. systemd timers are the better modern option: they log to the journal, support dependencies, have accuracy and randomised delay options, and can be monitored like any unit. They are more verbose to define but far more observable. For anything important, a job scheduler with retries and alerting is the right answer, and the critical addition regardless is dead-man monitoring — alerting when a job has not run, since silent non-execution is the failure nobody notices.

; runs the next command unconditionally. && runs it only if the previous succeeded — exit status zero. || runs it only if the previous failed. They chain, and the combination cmd && echo ok || echo failed is a common idiom, though it has a subtlety: if the echo ok itself fails, the failure branch also runs. For anything beyond trivial use, an if statement is clearer and correct. The practical uses: && for dependent steps, so a build only runs if the previous step succeeded — which is why Dockerfile RUN commands chain with && rather than semicolons, so a failure fails the layer. || true is the idiom for allowing a command to fail without triggering set -e, which is occasionally necessary and should be commented when used. The distinction from & is worth noting since it looks similar: a single ampersand backgrounds the command rather than chaining. And in a Makefile or a CI step, each line often runs in its own shell, so chaining with && is required for state to carry — a directory change on one line does not affect the next.

Stream it. Every standard tool reads line by line and writes as it goes, so a pipeline over a hundred-gigabyte file uses constant memory. The mistakes that break that: reading the file into a variable, using a command that must buffer everything such as sort without limits, or looping in bash with a per-line subprocess call — which spawns a process per line and is thousands of times slower than a single awk invocation. So the rule is one pass with one tool rather than a loop calling tools. sort is the one that genuinely needs memory, but it spills to disk and -S controls the buffer, so it handles files larger than RAM. For a subset, head, tail and sed with a line range avoid reading the whole file — sed with a quit command stops early rather than scanning to the end, which matters on a huge file. Use LC_ALL=C for sort and grep when you do not need locale-aware collation; it is substantially faster. And parallel or xargs -P can split work across cores when the operation is independent per chunk, which is the shell equivalent of a thread pool.

xargs builds command lines from standard input, so you can feed the output of one command as arguments to another. It is needed because many commands take arguments rather than reading stdin — rm, cp, and most others — so piping a list of filenames to them does nothing. It also solves the argument list length limit: a command with a hundred thousand filenames exceeds the kernel limit and fails, while xargs batches them into multiple invocations automatically. The flags that matter: -0 with find -print0 handles filenames containing spaces and newlines, which the default whitespace splitting breaks on. -n limits arguments per invocation. -P runs invocations in parallel, which is a simple way to use several cores. -I replaces a placeholder so the argument can go somewhere other than the end. The safety practice is to echo first — pipe to xargs echo rm — and inspect before running destructive commands, because a mistake here deletes a lot quickly. find -exec is the alternative and avoids the quoting issues entirely; with a trailing plus it batches like xargs.

Check state before acting, and prefer operations that are naturally idempotent. Creating a directory with mkdir -p succeeds whether or not it exists. Adding a line to a config only if absent, rather than appending unconditionally, avoids duplication on the second run. Using a declarative tool — Ansible, Terraform — rather than imperative commands gets this by construction. Guard destructive operations: check that a variable is set and non-empty before using it in a path, and validate that a target looks like what you expect before removing it. Use a lock so two copies cannot run simultaneously — flock is the standard mechanism, and without it a slow run overlapping with the next produces races. Write to a temporary file and move it into place atomically rather than editing in place, so a failure mid-write does not leave a corrupt file. Use trap to clean up temporary files on exit including on error. And make the script report clearly what it did, so a re-run that changed nothing is distinguishable from one that failed to run — silence is ambiguous.

Work up the stack, because each tool tests a different layer and the failure tells you where the problem is. ping tests ICMP reachability only. A successful ping means the host is up and routing works; it says nothing about your service. A failed ping often just means ICMP is blocked, so it is weak evidence either way. nc -zv host port or telnet tests TCP connectivity to a specific port. This is the right tool for "can I reach the service at all", and it distinguishes refused — nothing listening — from timeout, which means packets are being dropped, usually by a firewall or security group. curl -v tests the full application path: DNS, TCP, TLS and HTTP, showing each stage so you can see where it fails. curl -w with a timing format breaks down DNS, connect, TLS and transfer time, which is excellent for finding which phase is slow. openssl s_client sits between, testing TLS specifically and showing the certificate chain. The method is to go up until something fails, because that identifies the layer without guessing.

ss shows socket statistics, replacing netstat and being much faster because it reads netlink rather than parsing /proc. The common invocations: ss -tlnp for listening TCP sockets with owning processes, ss -tanp for all TCP sockets including established, and ss -s for a summary of counts by state. The states worth recognising. Many in SYN_SENT means outgoing connections are not completing — a firewall dropping packets or an unreachable destination. Many in TIME_WAIT is normal for a busy client that closes connections, and only a problem near ephemeral port exhaustion. Many in CLOSE_WAIT is an application bug: the peer closed, the kernel acknowledged, and your application has not called close. Those accumulate and leak descriptors, and only the application can clear them. The Recv-Q and Send-Q columns matter too. On a listening socket they show the accept queue depth and its maximum, so a full accept queue means the application is not accepting fast enough. On an established socket, a growing Send-Q means data is not being acknowledged and a growing Recv-Q means the application is not reading.

Refused means the packet reached the host and was actively rejected with a TCP RST, usually because nothing is listening on that port. It is fast, because you get an explicit answer. Timeout means no response at all: the SYN was sent, retransmitted several times, and nothing came back. That takes the full timeout, often tens of seconds. The diagnostic value is in what each implies. Refused means routing works, the host is up, and the port is closed — so check whether the service is running and bound to the right interface. A service bound to 127.0.0.1 rather than 0.0.0.0 gives exactly this from another machine, and it is one of the most common causes. Timeout means packets are being silently dropped, which almost always means a firewall, a security group, or a routing problem. That is why firewalls prefer DROP over REJECT — dropping produces a timeout that slows port scanning, whereas rejecting answers immediately. In cloud environments the shorthand is: timeout means security group, refused means the application.

dig, because it queries a resolver directly and bypasses application and OS caches. dig name shows the answer with its TTL, which tells you how long a cache will hold it. The status line distinguishes NOERROR, NXDOMAIN for a name that does not exist, and SERVFAIL for a resolver failure — three quite different problems. dig @8.8.8.8 name queries a specific resolver, so comparing a public resolver against your local one immediately shows whether the problem is local configuration or the record itself. dig +trace walks the delegation from the root, which is how you find a broken delegation. The things to check when resolution fails: /etc/resolv.conf for the configured resolvers, /etc/hosts for an override, and the search domains — a wrong search path resolves a short name to something unexpected. Negative caching is the one people forget: an NXDOMAIN response is cached too, so a name queried before it existed stays unresolvable for a while after creation. In containers there is an extra layer, since the container has its own resolv.conf pointing at the cluster DNS, and Kubernetes ndots settings make external lookups try several suffixes first.

An outgoing connection needs a local port from the ephemeral range — typically about 28,000 ports on Linux. A connection is identified by the four-tuple of source and destination address and port, so the limit is that many concurrent connections to the same destination address and port. Exhaustion happens with high connection churn, because TIME_WAIT holds ports for 60 seconds after close. Two thousand new connections per second to one backend means far more sockets in TIME_WAIT than the range allows. The symptom is intermittent "cannot assign requested address" on the client while the server looks entirely healthy — which is why it is often misdiagnosed as a server problem. ss -s shows the TIME_WAIT count, and comparing it against the range in net.ipv4.ip_local_port_range confirms it. The real fix is connection pooling with keep-alive so connections are reused rather than churned, which addresses the cause. Secondary measures: widen the port range, and enable net.ipv4.tcp_tw_reuse for outgoing connections. Never use tcp_tw_recycle — it breaks clients behind NAT and has been removed from modern kernels.

The Maximum Transmission Unit is the largest payload a link can carry in one frame — conventionally 1500 bytes on Ethernet. A packet larger than the path MTU must be fragmented or dropped. Path MTU Discovery handles this: the sender sets the Don't Fragment bit, and a router that cannot forward replies with an ICMP "fragmentation needed" message carrying the correct size. The failure mode is when ICMP is blocked, which many networks do. The sender never learns, keeps retransmitting oversized packets that are silently dropped, and the connection hangs. The symptom is distinctive and confusing: the TCP handshake succeeds, small requests work, and anything large stalls. A TLS handshake completing followed by everything hanging is the classic signature, because the certificate exchange is the first large transfer. It bites hardest with tunnels and overlays. VPNs, VXLAN and some container network plugins add headers that reduce the effective MTU, which is why container networking so often needs explicit MTU configuration. The test is ping with a large payload and the Don't Fragment flag, decreasing the size until it succeeds, which finds the actual path MTU.

tcpdump, filtered narrowly. tcpdump -i any -n port 443 and host 10.0.0.5 -w capture.pcap is a typical invocation — -n avoids DNS lookups that slow the capture and pollute output, and -w writes a file for analysis in Wireshark. What to look for depends on the symptom. For a connection failure: is the SYN going out at all, and does anything come back? A SYN with no reply means packets are dropped, usually by a firewall. A SYN followed by RST means actively refused. For slowness: retransmissions indicate loss, duplicate ACKs indicate out-of-order delivery, and zero-window advertisements mean the receiver is not reading. For protocol problems, the actual bytes show whether the request was malformed. The cautions: capturing on a busy interface is expensive and can affect the system, so always filter. Capturing writes potentially sensitive data to disk. And with TLS you see the handshake but not the payload, so failures after the handshake need application logging instead. In containers, capture inside the network namespace or you see nothing relevant.

ip route shows the table. Each entry maps a destination prefix to a next hop and an outgoing interface, and selection uses longest prefix match — the most specific matching route wins. The default route, 0.0.0.0/0, matches everything and has the shortest prefix, so it is used only when nothing more specific matches. That is why it is the gateway of last resort. The genuinely useful command is ip route get ADDRESS, which asks the kernel which route it would actually use for a specific destination. That is far more reliable than reading the table and reasoning about it yourself, and it immediately reveals when traffic is going out an unexpected interface — a VPN, a secondary NIC, a container bridge. traceroute or mtr shows the path, with mtr being better because it runs continuously and reveals loss patterns rather than a single sample. The interpretation caution with traceroute: asterisks do not mean a hop is down, since many routers deprioritise or block ICMP responses. And latency at an intermediate hop reflects that router's ICMP generation, not the path quality — only a spike that persists to later hops is meaningful.

Both are the userspace interface to the kernel's packet filtering. nftables is the newer replacement, with a unified syntax across IPv4, IPv6 and bridging, better performance for large rule sets, and atomic rule replacement. Most distributions now use nftables underneath with an iptables compatibility layer, so iptables commands still work. What a backend engineer actually needs: understanding that rules are evaluated in order within a chain and the first match wins, so rule order determines behaviour. Knowing that INPUT filters traffic to the host, OUTPUT from it, and FORWARD through it. And understanding stateful matching — a rule accepting ESTABLISHED and RELATED connections is what allows return traffic without opening ports in both directions. Missing that rule is why a firewall appears to block responses. The practical relevance is usually diagnostic: iptables -L -n -v shows rules with packet counts, and a rule with rising counts on a DROP target explains a timeout. In containers, Docker and Kubernetes manipulate these rules extensively, so a manually added rule can be removed or bypassed, and reading the generated chains is how you understand a networking problem.

net.core.somaxconn caps the accept queue — the fully-established connections waiting for the application to accept. The default is low on older kernels, and overflow means connections are dropped silently and clients time out. net.ipv4.tcp_max_syn_backlog is the separate half-open queue, relevant under a SYN flood or a very high connection rate. net.ipv4.ip_local_port_range widens the ephemeral range, and tcp_tw_reuse allows reusing TIME_WAIT sockets for outgoing connections — both relevant to port exhaustion. net.core.rmem_max and wmem_max, with tcp_rmem and tcp_wmem, size socket buffers. On a high bandwidth-delay path the default caps throughput regardless of link speed. net.ipv4.tcp_congestion_control set to bbr often improves throughput on lossy or long-distance links, and is a one-line change. File descriptor limits at the OS level matter as much as any of these. The honest caveat is that most services never need to touch these, and tuning without measurement is cargo culting. The ones worth checking are somaxconn and the descriptor limit, because their defaults genuinely constrain a busy server.

A network namespace gives a set of processes its own network stack: its own interfaces, routing table, iptables rules, and socket table. So a container has its own loopback, its own eth0, and its own view of what ports are in use — which is why two containers can both bind port 8080 without conflicting, and why a port must be published to be reachable from the host. The usual wiring is a veth pair: one end inside the namespace appearing as eth0, the other on the host attached to a bridge. Traffic leaving the container crosses the pair to the bridge and is NATed out. The practical consequences for debugging. Running ss or tcpdump on the host shows the host namespace, not the container's — so you see nothing relevant. You must enter the namespace with nsenter or run the tool inside the container. A service bound to 127.0.0.1 inside a container is unreachable from the host even with the port published, because loopback is namespace-local. Binding to 0.0.0.0 is required, and forgetting is a very common containerisation mistake. And DNS comes from the container's own resolv.conf.

A Unix domain socket communicates between processes on the same machine through the filesystem namespace rather than the network stack. It is faster than TCP over loopback because it skips the entire network stack — no headers, no checksums, no routing — so throughput is higher and latency lower. It also has stronger access control: the socket is a filesystem object with permissions, so you can restrict which users may connect. A TCP socket on localhost is reachable by any local process. And it can pass file descriptors between processes, which TCP cannot — that is how some servers hand off connections to workers. The practical uses: Docker's daemon socket, PostgreSQL local connections, nginx to an application server on the same host, and systemd socket activation. The trade-off is that both processes must be on the same machine, so it cannot be a general service interface. In a container that means the socket must be on a shared volume, which is how a sidecar communicates with its main container. For a database on the same host, switching from TCP to a Unix socket is a measurable and free improvement.

Enumerate what differs, because the difference is the clue. Network position: different subnet, VPC, security group or firewall rules. In cloud environments this is the most common cause — check both directions, remembering that stateless network ACLs need an explicit rule for return traffic on ephemeral ports. DNS: the two machines may use different resolvers and get different answers. Compare dig output and check /etc/resolv.conf. Routing: ip route get to the destination on each shows which interface and gateway would be used. A VPN or a stale route can send traffic somewhere unexpected. TLS trust: different CA bundles, or a JVM with its own truststore. Source address: if the destination has an IP allowlist, check what address it actually sees. Proxy configuration: HTTP_PROXY environment variables set on one machine and not the other silently change everything, and they are easy to overlook because they are invisible in the command. And time: a large clock skew breaks TLS certificate validation and token validation, which produces errors that look unrelated.

Packet loss somewhere on the path, which TCP recovers from at the cost of latency and throughput. netstat -s or ss -s shows retransmission counters; a rising rate as a fraction of segments sent is the signal. The causes: genuine network congestion, a saturated link, faulty hardware or cabling, an overloaded intermediate device, or an undersized queue dropping packets. It can also be a receiver problem — a full socket buffer causing drops if the application is not reading fast enough, which shows as zero-window advertisements alongside. The consequences matter more than the count. A single lost segment recovers with fast retransmit and a halved congestion window — a modest penalty. A timeout collapses the window and restarts slow start, which craters throughput. So the ratio of timeouts to fast retransmits tells you how bad it is. The application sees none of this directly, only latency — which is why unexplained tail latency is often loss on the path rather than anything in the service. mtr over a sustained period is the tool for locating where loss occurs, since a single traceroute sample proves nothing.

Work through the resources systematically rather than guessing. The USE method — utilisation, saturation and errors for each resource — is a good structure to name. Start with uptime or top for load average and CPU. High user CPU means application computation; high system CPU means excessive syscalls or context switching; high iowait means blocked on storage. Memory: free -h, reading available rather than free, and vmstat's si and so columns to check for swapping. Sustained swap activity explains almost any slowness by itself. Disk: iostat -x for per-device utilisation and await. A device with rising await and a growing queue is saturated. Network: ss -s for connection counts and states, and retransmission counters. The pattern that catches people is low CPU with high latency, which almost always means waiting — on I/O, on a lock, or on a downstream service. That is when you stop looking at system metrics and go to application-level tracing or a thread dump. And check the obvious first: disk full, a recent deployment, and whether it is actually this machine.

iowait is the percentage of time the CPU was idle while at least one I/O request was outstanding. It means the CPU had nothing to run because everything was blocked on I/O. The crucial subtlety is that it is a form of idle time, not busy time. High iowait means the CPU is available and work is blocked — it is a symptom, not a problem in itself. And it misleads in both directions. A machine with a slow disk but plenty of other work shows low iowait, because the CPU is never idle — so low iowait does not mean storage is healthy. Conversely a mostly idle machine with one slow I/O shows high iowait without any real issue. It is also averaged across cores, which dilutes it further. So it should never be read alone. Pair it with iostat, which measures the device directly — await, and queue depth — rather than inferring from CPU idleness. The useful conclusion from high iowait is "look at the storage layer", not "the disk is the bottleneck". And in a VM or cloud instance, it may reflect a noisy neighbour or a volume throughput limit rather than anything local.

vmstat 1 shows si and so columns — swap in and swap out. Sustained non-zero values mean active swapping. free shows swap used, but used swap alone is not a problem; pages swapped out long ago and never touched cost nothing. It is the rate that matters, not the amount. Why it matters: swapping means memory accesses become disk accesses, which is orders of magnitude slower. A process that was fast becomes unusably slow, and because the system is thrashing, adding load makes it worse. The signature is high disk I/O with low CPU utilisation and terrible latency — the machine appears busy and idle simultaneously. The fixes are to reduce memory demand or add memory. Tuning the page replacement policy does not help, because the problem is capacity. vm.swappiness controls how eagerly the kernel swaps anonymous memory versus dropping page cache; lowering it favours keeping process memory. Most container deployments disable swap entirely, preferring a fast OOM kill to unpredictable latency — and Kubernetes historically required it off. That trade is deliberate: a dead pod that restarts is better than a live one taking ten seconds per request.

iostat -x 1 gives extended per-device statistics, and the columns that matter are await, utilisation, and the queue size. await is the average time a request spends including queueing — this is the number that reflects what the application experiences. r_await and w_await split it by direction, which matters because reads and writes often behave very differently. %util is the percentage of time the device had at least one request outstanding. It is frequently misread as saturation, and on modern SSDs and RAID it is not: those devices handle many requests in parallel, so 100% utilisation simply means never idle, not overloaded. On a single spinning disk it was a reasonable proxy; today it is not. aqu-sz, the average queue size, is the better saturation signal — a growing queue means requests are arriving faster than they complete. r/s and w/s with rkB/s and wkB/s show whether the load is IOPS-bound or throughput-bound, which determines whether the fix is a faster device or fewer, larger requests. And pair it with iotop or pidstat -d to attribute the I/O to a process, since iostat is per-device only.

iotop shows per-process read and write rates interactively, which iostat cannot since it is per-device. It needs privileges and kernel support for per-task I/O accounting. pidstat -d 1 is the non-interactive equivalent and is better for scripting or capturing over time. /proc/PID/io gives cumulative counters per process — read_bytes and write_bytes are the ones reflecting actual device I/O rather than page cache hits, which is an important distinction because a process reading from cache shows large rchar but no real disk traffic. That distinction is what tells you whether the I/O is genuinely hitting the device. For deeper analysis, biosnoop and biolatency from bcc or bpftrace attribute individual block operations to processes with latency distributions, which is how you find that one query is issuing enormous random reads. The practical sequence during an incident: iostat to confirm the device is saturated, iotop to find the process, then application-level tools to find what within it. Skipping the first step leads to blaming the wrong process, since the busiest I/O consumer may be perfectly reasonable.

VSZ is virtual size — the total address space the process has reserved, including memory never touched, files mapped but not read, and shared libraries. It is almost always far larger than real usage and is a poor metric to alarm on. RSS is resident set size — physical memory currently in use. Closer to reality, but it counts shared pages in full for every process sharing them, so summing RSS across processes massively overcounts. PSS, proportional set size, divides each shared page's cost among its sharers, so it sums correctly. It is available in /proc/PID/smaps and via smem. The practical guidance: RSS for a single process, PSS for total system usage across many. The JVM case illustrates why VSZ misleads — a heap maximum reserves address space immediately, so VSZ is large from startup while RSS grows only as pages are touched. Alarming on VSZ produces constant false positives. And in containers, the number that triggers an OOM kill is the cgroup memory usage, which includes page cache attributed to the cgroup — so RSS alone can understate what the limit sees.

Identify the thread, then find what it is executing. top -H -p PID lists threads with individual CPU usage, which immediately narrows from the process to one or two threads. For a JVM, convert the thread ID to hexadecimal and search a jstack dump for that nid to get the exact Java stack. Taking three dumps a few seconds apart and comparing shows whether the thread is stuck in one place or looping through a range. For Python, py-spy dump attaches without meaningfully disturbing the process and prints stacks. For native code, perf top gives a live profile by symbol and perf record with a flame graph gives the full picture. The distinction to establish is whether it is doing useful work or spinning. A retry loop, a busy-wait, or a regular expression with catastrophic backtracking look identical to legitimate computation from outside — only the stack distinguishes them. One specific pattern worth knowing: an unsynchronised HashMap corrupted by concurrent writes can form a cycle in a bucket, causing get to loop forever at full CPU. The stack points straight at HashMap.get with no obvious cause.

perf is the kernel's profiling framework, using hardware performance counters and sampling. perf top gives a live view of which symbols are consuming CPU, across the whole system or one process. It is the fastest way to see where time is going. perf record and perf report capture a profile for offline analysis, and piping the output through a flame graph generator produces the visualisation that makes a profile immediately readable. perf stat gives counters for a command — instructions, cache misses, branch mispredictions, context switches — which is how you distinguish "slow because of cache behaviour" from "slow because of instruction count". perf trace is a lower-overhead alternative to strace. The advantages over language-specific profilers: it sees everything including kernel time and native libraries, and its sampling overhead is low enough for production. The practical caveats: it needs symbols to produce readable output, so stripped binaries give addresses. For a JVM you need a symbol map agent; for Python, py-spy is easier. And it requires privileges, with perf_event_paranoid controlling what unprivileged users may do — which is why it often does not work in a container without extra capabilities.

eBPF runs sandboxed programs in the kernel, attached to hooks — syscalls, function entry and exit, network events, tracepoints — without modifying kernel code or loading modules. A verifier proves the program terminates and is memory-safe before it loads, which is what makes running arbitrary code in the kernel acceptable. For observability it means measuring things that previously required either heavy instrumentation or kernel patches, at very low overhead — safe to run on production systems where strace is not. The practical tools: bcc and bpftrace provide ready-made scripts. biolatency gives a histogram of block I/O latency. execsnoop shows every process execution. tcpconnect and tcpretrans show connection attempts and retransmissions. opensnoop shows file opens. And profile gives stack sampling for flame graphs. bpftrace lets you write one-liners for ad hoc questions, which is where the real power is — you can answer "which files is this process opening, and how long does each take" without a purpose-built tool. It also underpins modern networking and security tooling — Cilium, Falco — which is why it has become central to the container ecosystem.

Control groups constrain what a set of processes may use — CPU, memory, block I/O, PIDs — and account for their usage. Memory: exceeding the limit triggers a cgroup-scoped OOM kill. The process is SIGKILLed with no chance to log, which is why a container that "just disappeared" leaves nothing in its own logs — the evidence is in the kernel log on the node. Exit code 137 in Kubernetes is this. CPU: two mechanisms. Shares are a relative weight applied only under contention. A quota is a hard cap — so many microseconds per period — and exceeding it causes throttling: the process is descheduled entirely until the next period. Throttling is the one that surprises people, because it is not gentle slowing. A container limited to one CPU gets 100ms per 100ms period; four threads can burn that in 25ms and then sit idle for 75ms, producing latency spikes while average CPU usage looks low. That is why container_cpu_cfs_throttled_seconds is a metric worth alarming on, and why many teams set CPU requests without limits for latency-sensitive services.

137 is 128 plus 9, meaning the process was terminated by SIGKILL. In a container that almost always means the cgroup memory limit was exceeded and the kernel OOM-killed it. The diagnosis is not in the application logs, because SIGKILL cannot be handled — the process gets no chance to write anything. The evidence is in the kernel log on the node: dmesg or journalctl -k shows the OOM message with the process name and memory figures. The usual causes: the limit is genuinely too low; a memory leak; a workload spike; or — very commonly — the limit was set from the application's heap size without accounting for everything else. That last one is worth stressing. A JVM consumes heap plus metaspace plus thread stacks plus the code cache plus native buffers, so total RSS routinely exceeds the heap by 25 to 50 percent. Setting the container limit equal to -Xmx guarantees an eventual kill. MaxRAMPercentage is the better control. The cgroup also charges page cache to the container, so heavy file writing can push usage toward the limit even though that memory is reclaimable. And 137 can also mean a failed graceful shutdown followed by SIGKILL.

Real time is wall clock — how long it actually took. User time is CPU time spent executing application code. System time is CPU time spent in the kernel on the process's behalf. The relationships tell you a lot. If user plus system is far less than real, the process was waiting — on I/O, on a lock, or on a downstream service. That is the signature of a blocked rather than a busy process. If user plus system exceeds real, the process used several cores in parallel, which confirms it is genuinely parallel. High system time relative to user means excessive syscalls or context switching — usually too many small I/O operations, which buffering fixes, or lock contention causing futex churn. High user time means the work is in application code, so profiling the application is the next step. The time command gives all three for a command, and /proc/PID/stat exposes cumulative values for a running process. The practical value is that this three-way split immediately directs the investigation — toward I/O, toward syscalls, or toward the algorithm — before any deeper tooling.

First establish where the growth is: system-wide or one process. free over time, or vmstat, shows whether available memory is declining. If it is, find the process with ps sorted by RSS, or smem for PSS, sampled over time — a steadily growing RSS is the signature. Then the tools depend on the runtime. For a JVM, a heap dump and a dominator tree in Eclipse MAT shows what is retaining memory. For Python, tracemalloc snapshots compared over time show which allocation sites grow. For native code, valgrind or an ASan build finds unfreed allocations. /proc/PID/smaps shows the memory map broken down by region, which distinguishes heap growth from mapped files from thread stacks — and a growing number of stacks means a thread leak rather than a memory leak. The non-leak explanations worth ruling out first: the page cache growing is normal and reclaimable. Memory fragmentation can keep RSS high after a workload without a leak. And a slab cache growing — dentries from millions of file operations — is kernel memory, visible in slabtop rather than in any process. That last one is easy to misattribute.

Establish what kind of unresponsive, because the causes differ. If SSH connects but commands are slow, the system is thrashing or a resource is saturated. Check load, memory and iowait. If the shell hangs on tab completion, it is usually disk. If SSH refuses, the network or sshd is the problem. If it times out, packets are being dropped. If you have console access — a cloud serial console or IPMI — that bypasses sshd and the network, and is often the only way in when memory pressure has made forking impossible. The common causes: memory exhaustion where the OOM killer is thrashing; a full disk, particularly on the root filesystem, where logging and even login can fail; a runaway process consuming all CPU; a hung NFS mount putting processes in D state; and fork exhaustion from a process or PID limit. The kernel log is where the evidence usually is, so dmesg first once you are in. The preventive measures worth naming: reserving root-only space on the root filesystem, memory limits on services, and monitoring that alerts before saturation rather than after.

Measure the same thing under the same conditions, and be honest about variance. The mistakes that make comparisons meaningless: running once, when a single sample tells you nothing about a noisy system. Comparing across different hardware or a different time of day. Not warming caches, so the first run pays for cold page cache and JIT compilation. And measuring the mean when the tail is what matters. For a command, hyperfine handles the repetition, warmup and statistics properly, and reports the distribution rather than a single number. For a service, a load generator at a fixed rate — not a fixed concurrency — with latency percentiles reported. Fixed-concurrency tools produce coordinated omission, which understates latency badly. Control for the environment: same instance type, same data volume, same cache state, and ideally interleave the runs rather than doing all of A then all of B, so drift affects both equally. And state the effect size relative to variance. A 3% improvement with 10% run-to-run variance is not a result, and reporting it as one is how performance work loses credibility.

Throughput is work completed per unit time. Latency is how long one unit takes. They are related but not the same, and optimising one frequently damages the other. Batching is the clearest example: grouping requests amortises per-operation overhead and raises throughput, while adding waiting time and therefore latency. Nagle's algorithm does exactly this at the TCP level, and disabling it trades throughput for latency deliberately. Queueing is the other. A deep queue keeps a resource busy and maximises throughput, but every queued item waits — so latency rises with queue depth. Bufferbloat is this at the network level: enormous router buffers give high throughput and dreadful latency. Little's law connects them: concurrency equals throughput times latency. So at a fixed concurrency, reducing latency raises throughput and vice versa. The practical implication is that you must decide which the service is optimising for. An interactive API cares about p99 latency and should keep queues shallow and shed load. A batch pipeline cares about throughput and should batch aggressively. Measuring only throughput while users experience latency is a common mismatch.

systemd is the init system and service manager — PID 1 on most modern distributions. It starts the system, manages services, handles dependencies and ordering, and supervises processes. A unit is a configuration object describing something systemd manages. Service units describe daemons. Socket units enable socket activation. Timer units replace cron. Mount units describe filesystems. Target units group others, replacing runlevels. The advantages over the old init scripts: parallel startup based on declared dependencies rather than sequential numbering, automatic restart on failure, proper supervision using cgroups so all child processes are tracked and cleaned up, integrated logging, and resource limits per service. That cgroup tracking is a genuine improvement — an init script could lose track of forked children, while systemd reliably stops everything a service started. The practical commands: systemctl status for state and recent logs, start, stop, restart, enable for start-at-boot, and daemon-reload after editing a unit file — which people forget, and then wonder why their change had no effect. journalctl -u reads a service's logs.

ExecStart with the absolute path to the binary, since PATH is minimal. User and Group so it does not run as root. Restart=on-failure or always, with RestartSec to avoid a tight restart loop, and StartLimitBurst with StartLimitIntervalSec so a persistently failing service stops rather than restarting forever. After and Requires or Wants for dependencies — After controls ordering, Requires makes the dependency mandatory, and Wants is a soft dependency. Confusing ordering with requirement is a common mistake, since After alone does not ensure the dependency is started. Environment or EnvironmentFile for configuration, remembering that the service does not read shell startup files. LimitNOFILE for the descriptor limit, since limits.conf does not apply to systemd services — this is the single most common "my ulimit change did nothing" cause. WorkingDirectory if the process assumes one. And for hardening: ProtectSystem, PrivateTmp, NoNewPrivileges and ReadWritePaths, which sandbox the service cheaply. Type matters too — simple for a foreground process, notify if the service signals readiness, which gives accurate dependency ordering.

journalctl -u SERVICE for one unit's logs, which is the most common use. -f follows in real time. -n limits to the last N lines. --since and --until take human-readable times such as "10 minutes ago" or an absolute timestamp, which is far more convenient than filtering text logs. -p err filters by priority, which is how you find errors quickly in a noisy service. -b shows the current boot, and -b -1 the previous one — invaluable after an unexpected reboot. -k shows kernel messages, which is where OOM kills and hardware errors appear. -o json gives structured output for processing, since the journal stores structured fields rather than plain text. The operational details: the journal has size limits configured in journald.conf, and an unbounded journal filling /var/log is a real oversight. Persistence across reboots requires /var/log/journal to exist, otherwise logs are memory-only and lost on restart — which surprises people investigating a crash. And in a container the journal is usually not used at all; logs go to stdout and the platform collects them.

systemctl status gives the state, the exit code, and the last few log lines, which often names the problem directly. journalctl -u SERVICE -n 50 gives more context, and adding --since narrows to the attempt. The common causes and their signatures. A non-absolute path in ExecStart, since PATH is minimal — the error is "No such file or directory" for something you can clearly run in your shell. Permissions, if User cannot read a config or write a directory. A missing environment variable, because the service does not read shell startup files. A port already in use. And a working directory the process assumes but is not set. A forgotten daemon-reload after editing the unit means systemd is running the old definition, which produces the confusing situation where the file is obviously correct. The technique that isolates it: run the exact ExecStart command manually as the configured user with a cleared environment — env -i — which reproduces the service's conditions rather than your shell's. systemd-analyze verify checks the unit file, and systemctl cat shows the effective configuration including drop-ins.

systemd creates and listens on the socket, and starts the service only when a connection arrives — passing the already-open socket to it. The benefits. Services start on demand rather than at boot, so boot is faster and idle services consume nothing. Dependency ordering becomes simpler, because a client can connect immediately and the connection is queued while the service starts. And restarts can be seamless: connections queue in the socket while the service restarts, so clients see a delay rather than a refusal. That last property is the genuinely valuable one for zero-downtime restarts of a local service. The implementation requires the service to accept a pre-opened descriptor rather than binding itself, using the sd_listen_fds protocol. Many daemons support it; many do not. It is also how systemd can run a service as a non-root user while listening on a privileged port, since systemd binds the socket before dropping privileges. In a container world it is less relevant, since the orchestrator handles lifecycle — but the socket-passing idea reappears in graceful restart schemes for web servers.

A timer unit triggers a service unit on a schedule, replacing a crontab entry. The advantages. Logging goes to the journal automatically, so a failed run is visible rather than lost to a discarded email. The triggered job is a normal service, so it gets resource limits, dependencies and restart policy. systemctl list-timers shows the next and last run of everything, which cron has no equivalent for. Persistent=true runs a missed job after downtime, which cron cannot. And RandomizedDelaySec spreads load across machines, avoiding the thundering herd of everything firing at midnight. OnCalendar syntax is more readable than cron's five fields, and monotonic timers can fire relative to boot. The disadvantages: two files rather than one line, and less familiarity. The shared problem is that neither retries on failure or alerts, so anything important still needs monitoring — and the most valuable addition is dead-man alerting on a job not running, since silent non-execution is the failure nobody notices. In containers, neither applies; the orchestrator's CronJob is the equivalent.

After controls ordering only — start this unit after that one has started. It says nothing about whether the other unit is required or even present. Requires declares a hard dependency: the other unit is started too, and if it fails or is stopped, this unit is stopped as well. It does not imply ordering, which is the part people miss. Wants is a soft dependency: the other unit is started, but its failure does not affect this one. So the combination matters. Requires without After starts both simultaneously and this unit may run before its dependency is ready. The correct pattern for a service needing a database is Requires plus After, or more commonly Wants plus After — because a hard Requires means a database restart takes your service down with it, which is usually not what you want. Wants plus After is the pragmatic default: start after it, prefer it running, but survive independently. The deeper caveat is that "started" does not mean "ready". A service with Type=simple is considered started immediately, so ordering does not guarantee availability — Type=notify with readiness signalling is what makes ordering meaningful.

Drop-in files. systemctl edit SERVICE creates an override under /etc/systemd/system/SERVICE.d/ containing only the settings you want to change, which are merged over the vendor unit. That is the correct approach because editing the file shipped by the package means a package update overwrites your change, silently reverting it — a genuinely confusing failure weeks later. The merge semantics have one trap: for list-valued settings such as ExecStart, the override appends rather than replaces. To replace, you must first clear it with an empty assignment then set the new value. Forgetting produces a unit with two ExecStart lines, which fails. systemctl edit --full copies the whole unit for editing if you need wholesale changes. systemctl cat SERVICE shows the effective configuration including all drop-ins, which is how you confirm what is actually in force — and is the first thing to check when a setting appears to be ignored. daemon-reload is required after any change. The same mechanism applies across the precedence directories, with /etc taking priority over /usr.

Restart=on-failure restarts when the process exits non-zero or is killed by a signal; Restart=always restarts even on clean exit. RestartSec sets the delay between attempts, which prevents a tight loop hammering the system. The rate limiting is the part that matters: StartLimitBurst and StartLimitIntervalSec define how many starts are allowed within a window. Exceeding it puts the unit in a failed state and stops trying, which is what prevents an unfixable service from restarting thousands of times and filling the logs. Without those limits, a service that fails immediately on a configuration error restarts continuously, generating enormous log volume and masking the original error. The defaults are five starts in ten seconds, which is often too aggressive for a service with slow startup — so raising the interval is common. systemctl reset-failed clears the state after fixing the problem, which is needed before it will start again. The complementary practice is alerting on the failed state, since a service that gave up is silent — and silence is exactly what you must not rely on someone noticing.

systemd provides sandboxing options that cost nothing to enable and remove large classes of risk. User and Group so it does not run as root — the single most important one. NoNewPrivileges prevents the process gaining privileges through setuid binaries. ProtectSystem=strict makes the whole filesystem read-only except explicitly listed paths, and ProtectHome hides user directories. ReadWritePaths then grants only what the service needs. PrivateTmp gives an isolated /tmp, preventing temp file races and cross-service interference. PrivateDevices restricts device access. ProtectKernelTunables and ProtectKernelModules prevent modifying kernel state. RestrictAddressFamilies limits which socket types are available, so a service that only needs TCP cannot open raw sockets. SystemCallFilter restricts syscalls to a set, which is a meaningful reduction in kernel attack surface. systemd-analyze security SERVICE scores a unit and lists what is not enabled, which is a good way to find easy improvements — and running it against your services usually reveals that almost none of this is configured. The caution is testing, since over-restricting breaks the service in ways whose errors are not obvious.

Logical Volume Management sits between physical devices and filesystems. Physical volumes are grouped into a volume group, from which logical volumes are carved and formatted. The benefits are flexibility. A logical volume can be extended online, drawing from free space in the group, so growing a filesystem does not require repartitioning or downtime. Volumes can span several physical disks. And snapshots allow a consistent point-in-time copy for backup. That online resize capability is the main reason production servers use it — running out of space on a fixed partition otherwise means a maintenance window. The practical commands: pvs, vgs and lvs to inspect; lvextend to grow a volume, followed by resize2fs or xfs_growfs to grow the filesystem, which is a separate step people forget — extending the volume alone changes nothing visible. The caveats: shrinking is riskier than growing and XFS cannot shrink at all. Snapshots consume space as the origin changes and fill up silently if undersized, at which point they are invalidated. And a volume group spanning disks means losing one disk affects everything on it unless there is RAID underneath.

lsblk to identify the new device. Partition it if you want partitions — often unnecessary for a data disk in a cloud instance, where using the whole device is fine. Create a filesystem with mkfs, typically ext4 or xfs. Create a mount point directory, mount it to test, then add an entry to /etc/fstab so it mounts at boot. The fstab detail that matters: use a UUID rather than a device name. Device names such as /dev/sdb are not stable across reboots or hardware changes, so a name-based entry can mount the wrong disk or fail. blkid gives the UUID. The danger with fstab is that a bad entry can prevent boot entirely, dropping the machine into emergency mode. Testing with mount -a before rebooting catches it, and the nofail option prevents a missing device blocking boot — which is worth using for any non-essential mount. The mount options to consider: noatime reduces write traffic by not updating access times, which is a free improvement for most workloads. noexec and nosuid on data mounts are cheap hardening. And remember to set ownership on the mount point after mounting, not before, since the mount hides what was there.

Both are mature journaling filesystems and either is a reasonable default. ext4 is the long-standing Linux default. It handles small files well, can be shrunk as well as grown, and has extensive tooling and recovery experience behind it. XFS was designed for large files and high parallelism. It scales better to very large filesystems and high concurrency, and its allocation behaviour suits large sequential I/O — which is why it is the default on RHEL and is common for database and media storage. The practical difference that catches people is that XFS cannot be shrunk. It can grow online but never shrink, so oversizing a volume is a one-way decision. ext4 can shrink, though only offline. Other considerations: XFS has better performance with many parallel writers; ext4 has slightly better behaviour with huge numbers of small files in one directory, though both handle it far better than older filesystems. For most server workloads the choice is not performance-critical, and matching the distribution default is a defensible reason — it is the configuration most tested and most documented. Btrfs and ZFS are the copy-on-write alternatives, offering snapshots and checksums at the cost of fragmentation with random writes.

RAID 0 stripes across disks for speed and capacity with no redundancy — losing one disk loses everything. It is only appropriate for data you can regenerate. RAID 1 mirrors, giving redundancy and fast reads at half the usable capacity. Simple and safe. RAID 5 stripes with distributed parity, tolerating one disk failure with only one disk of overhead. Its weakness is rebuild: with large modern disks, rebuilding takes many hours under heavy read load, during which a second failure — or an unrecoverable read error — loses the array. That risk is why RAID 5 is discouraged for large drives. RAID 6 uses two parity blocks, tolerating two failures, which addresses the rebuild window. It costs more capacity and write performance. RAID 10 mirrors then stripes, giving good performance and rebuild characteristics at 50% capacity. It is the usual choice for databases. The point worth making: RAID is availability, not backup. It protects against disk failure, not against deletion, corruption, ransomware or a mistaken command — all of which are faithfully mirrored.

Start with iostat -x to confirm the device is the problem and see which one — high await with a growing queue means requests are backing up. Distinguish read from write latency, since they often differ substantially and point at different causes. Then attribute it: iotop or pidstat -d finds the process. For finer detail, biolatency gives a histogram of I/O latency and biosnoop shows individual operations with their originating process, which reveals patterns an average hides — such as a small number of very slow operations dragging the mean. The causes to consider. A saturated device, where the fix is fewer or larger requests, better caching, or faster storage. Random versus sequential access, since random I/O on a spinning disk is orders of magnitude slower. A noisy neighbour on shared cloud storage. Hitting a provisioned IOPS or throughput limit, which produces a very characteristic plateau. And a failing disk, visible in dmesg and SMART data. The application-level causes are often the real answer: a missing index causing full scans, unbatched writes, or fsync per operation where batching would do.

Block storage presents raw fixed-size blocks with no structure — a disk. You put a filesystem on it and mount it. It gives the lowest latency and is what databases want. It is typically attached to one machine at a time. File storage presents a filesystem over a network — NFS, SMB — so several machines can mount the same tree and see the same files with POSIX semantics. Convenient for shared state, but network latency and locking semantics make it a poor fit for databases, and a hung NFS mount puts processes in unkillable D state. Object storage — S3 and equivalents — stores whole objects addressed by key, accessed over HTTP. There is no partial update: you replace an object rather than modifying it. It scales effectively without limit, is cheap, and is durable across failures. The practical mapping: block for databases and anything latency-sensitive, file for shared configuration or legacy applications expecting a filesystem, object for user uploads, backups, logs and static assets. The common mistake is putting uploads on block storage attached to one instance, which prevents horizontal scaling.

Growing is straightforward and usually online: extend the underlying volume first — lvextend for LVM, or resize the cloud volume — then grow the filesystem with resize2fs for ext4 or xfs_growfs for XFS. The step people miss is that extending the volume does nothing visible until the filesystem is grown; df still shows the old size, which causes confusion. For a partition rather than LVM, the partition table must be updated first, which is riskier and often requires a reboot or a partition rescan. Shrinking is the dangerous direction. XFS cannot shrink at all. ext4 can, but only unmounted, and it requires a filesystem check first — and if the data extends beyond the new boundary, shrinking destroys it. So it needs a backup and downtime. The practical guidance is to size conservatively and grow as needed, rather than oversizing and shrinking later. With LVM that is easy, since you can leave free space in the volume group and extend on demand. And always confirm with df afterwards, and check that the filesystem is actually the one you resized — resizing the wrong volume is a memorable mistake.

A container's writable layer is ephemeral: anything written that is not in a volume disappears when the container is removed. That is by design, and writing application data to the container filesystem is the mistake. The writable layer uses copy-on-write through the storage driver — overlayfs typically — so modifying a file from the image copies the whole file into the writable layer first. Editing one line of a large file consumes its full size. That matters for write-heavy workloads: the copy-on-write layer is slower than a volume, so databases and anything doing sustained I/O should use a volume. Disk exhaustion on the node is a common operational problem, caused by accumulated images, stopped containers, unused volumes and build cache. docker system df shows the breakdown and prune reclaims it. In Kubernetes, kubelet garbage collection handles images but disk pressure still evicts pods. Log files are the other frequent cause — a container writing logs to its filesystem with no rotation fills the node, which is another reason to log to stdout. And ephemeral storage limits can be set per pod to prevent one container filling a node.

sudo runs a command as another user, usually root, after checking /etc/sudoers for authorisation and logging the invocation. The logging is a significant part of its value: every sudo invocation is recorded with the user, the command and the working directory, which gives an audit trail that sharing a root password does not. The configuration guidance: grant specific commands rather than blanket ALL where practical, edit with visudo which validates the syntax before saving — a malformed sudoers file can lock everyone out of privilege escalation — and prefer group-based rules over per-user entries. The subtlety worth knowing is that granting a specific command is often not the restriction it appears. Allowing a user to run an editor, or any program that can execute a shell or write arbitrary files, is equivalent to granting full root. The same applies to commands accepting a configuration file path, or anything with a shell escape. GTFOBins catalogues these, and reviewing a sudoers entry against that list is worthwhile. NOPASSWD removes the password prompt, which is convenient for automation and removes a barrier for an attacker with a shell. And sudo -i versus sudo -s differ in whether the target user's environment is loaded.

The setuid bit makes a binary run as its owner rather than as the invoking user. On a root-owned binary, that means any user running it gets root privileges for its duration. It exists because some operations genuinely require privilege — passwd must write to the shadow file, ping historically needed raw sockets. It is dangerous because the binary becomes a privilege boundary, and any vulnerability in it is a privilege escalation. A buffer overflow in a setuid program is not a crash, it is a root shell. That is why setuid binaries have historically been a major source of local privilege escalation. The practices that follow: minimise the number of setuid binaries on a system, and audit them — find with -perm -4000 lists them. Question any that are not from the distribution. Mount data filesystems with nosuid so a setuid binary placed there is ineffective, which blocks a common escalation path when an attacker can write files. The modern alternative is capabilities, which grant a specific privilege rather than full root — CAP_NET_BIND_SERVICE lets a process bind a low port without any other root power, which is far narrower than setuid.

Capabilities split root's powers into distinct privileges that can be granted individually, rather than the all-or-nothing root model. CAP_NET_BIND_SERVICE allows binding ports below 1024. CAP_NET_RAW allows raw sockets, which ping needs. CAP_SYS_ADMIN is an enormous catch-all that is close to root and should be treated as such. CAP_CHOWN, CAP_SETUID and CAP_DAC_OVERRIDE are similarly broad. The practical use is running a service as a non-root user while still allowing the one privileged thing it needs. A web server binding port 80 can have CAP_NET_BIND_SERVICE rather than starting as root — though systemd's socket activation or a reverse proxy usually removes the need entirely. In containers this matters more. Docker drops most capabilities by default, and adding them back should be deliberate and minimal — --cap-add=NET_ADMIN rather than --privileged, which grants everything and effectively removes the container boundary. The caveat is that some capabilities are effectively equivalent to root because they allow acquiring the rest. CAP_SYS_ADMIN, CAP_SYS_MODULE and CAP_DAC_READ_SEARCH are in that category, so granting them is not the reduction it appears.

Disable password authentication entirely and use keys — PasswordAuthentication no. That removes brute forcing as an attack, which is the overwhelming majority of SSH attack traffic. Disable direct root login with PermitRootLogin no, so administrative actions go through a named user and sudo, giving attribution. Restrict which users may connect with AllowUsers or AllowGroups, so a new account is not automatically remotely accessible. Use modern key types — ed25519 rather than RSA — and protect private keys with a passphrase plus an agent. Changing the port reduces log noise from automated scanning but is not meaningful security; it is worth doing for the noise reduction and not worth relying on. fail2ban or equivalent rate-limits repeated failures. The better architectural answer for a fleet is to remove standing SSH access entirely: a bastion or an agent-based system such as SSM Session Manager gives auditable, credential-free access with no open port and no key distribution problem. And on the client side, use an agent with forwarding disabled by default, since agent forwarding to an untrusted host lets that host use your keys.

Both are mandatory access control systems. Ordinary Unix permissions are discretionary — the owner decides — whereas MAC enforces a system-wide policy that even root cannot override, confining each process to what its profile allows. The value is containment. A compromised web server confined by policy cannot read arbitrary files or open arbitrary sockets, even running as root, because the policy denies it regardless of file permissions. SELinux uses labels on every file and process with detailed policy; AppArmor uses path-based profiles and is simpler to write and reason about. People disable them because a denial produces a failure that does not look like a permissions problem — a service that cannot write a file it clearly owns, with a confusing error — and turning it off makes the problem disappear. That is understandable and it removes a real defence. The better approach is permissive mode, which logs denials without enforcing, so you can see what the policy would block and adjust. audit2allow generates policy from those logs, and semanage handles common cases such as allowing a service to use a non-standard port. On a server running one known application, the profile is usually small.

Automate security updates. unattended-upgrades on Debian and Ubuntu, or dnf-automatic on RHEL derivatives, applies security patches without intervention — configured to security updates only so functional changes do not arrive unannounced. The argument for automation is that the alternative is a manual process nobody performs consistently, and unpatched known vulnerabilities are the most common route into a server. The complication is reboots. Kernel and libc updates require a restart to take effect, and needrestart or checkrestart identifies services still running against replaced libraries — a patched library on disk does nothing for a process that already loaded the old one, which is a frequently missed detail. Livepatch avoids kernel reboots for some updates. The better model for a fleet is immutable infrastructure: build a new image with patches applied, test it, and replace instances rather than updating in place. That makes patching a deployment rather than a maintenance operation, gives a tested artefact, and eliminates configuration drift between machines. Containers follow the same logic — rebuild the image and redeploy rather than patching a running container, and scan images for known vulnerabilities in CI.

Preserve evidence before changing anything, and prefer to isolate the machine rather than reboot — a reboot destroys memory-resident evidence and may be what an attacker wants. The things to look at: unexpected listening ports and established connections with ss. Unfamiliar processes, particularly ones with deleted binaries, which /proc/PID/exe reveals as a dangling link. Recently modified files, found with find and -mtime. New or modified accounts in /etc/passwd and unexpected entries in authorized_keys, which is a common persistence mechanism. Cron entries and systemd timers for scheduled persistence. And auth logs for successful logins. The important caveat is that a root-level compromise means the tools you are running may be modified — a rootkit replaces ps and ls to hide itself. So findings from the machine itself are not trustworthy, and analysis from outside — a disk image, network flow logs, a separate monitoring system — is what you can rely on. The honest conclusion is usually that a confirmed compromise means rebuild rather than clean, because proving a machine is clean is much harder than rebuilding it from a known-good image.

Every process and person gets exactly the access needed, and nothing more. Concretely: services run as dedicated non-root users, never as root. Each service has its own user, so a compromise of one does not give access to another's files. File permissions are as narrow as they can be, with configuration containing secrets readable only by its service user. sudo grants specific commands rather than blanket access. Capabilities replace root where a single privilege is needed. Containers run as non-root with a read-only filesystem and dropped capabilities. Network access is restricted so a service can only reach what it needs, which contains lateral movement. Credentials are scoped: a database user for the application has only the permissions that application needs, not superuser. The practical difficulty is that least privilege is inconvenient, so the pressure is always toward granting more. The countermeasure is making the narrow path easy — templates for service units with hardening already set, base images that run as non-root — because a secure default is followed and a secure procedure is not. And the test is asking what an attacker gains from compromising each component.

Not in the source repository, not in environment variables visible in a process listing, and not in world-readable files. The practical hierarchy. A secret manager — Vault, AWS Secrets Manager — with short-lived credentials fetched at runtime is best, because a leaked credential expires and rotation does not require redeploying. Failing that, a file readable only by the service user, ideally mounted from the platform rather than baked into the image. Kubernetes secrets mounted as files and systemd's LoadCredential both work this way. Environment variables are common and weaker: they appear in /proc/PID/environ readable by the same user, are inherited by child processes, and frequently end up in crash dumps, error reports and logs. The operational practices matter as much as the mechanism. Rotate on a schedule and have a tested rotation procedure, because a rotation that has never been exercised does not work when you need it. Scan for committed secrets in CI. And treat any secret that reached a repository or a log as compromised and rotate it, since removal does not undo exposure. And audit who and what can read each secret.

The honest answer is a specific subset, and the deep sysadmin material mostly does not. What matters constantly: reading /proc to inspect a running process when the convenient tool is missing. Understanding signals and graceful shutdown, because that determines whether deployments drop requests. Knowing what cgroup limits do, because that is what kills your container and throttles it in ways that look like application problems. File descriptors and ulimits, since exhaustion is a common slow-burn failure. The diagnostic sequence for a slow system — CPU, memory, disk, network — and which tool answers which question. Enough networking to distinguish a firewall from a DNS problem from an application problem. Enough shell to write a safe script and to analyse a log file. What matters less day to day: filesystem internals, RAID configuration, kernel tuning, and package management minutiae. The framing worth giving is that this knowledge earns its keep during incidents. When the application logs look clean and the service is still failing, the answer is almost always one layer down — and you need the vocabulary and the tools to go and look.