Essential Linux Commands for DevOps Engineers
🔵 Basic File Operations
Basic File Operations
ls -lah — List directory contents with full detail
$ ls -lah /var/log/nginx/ total 1.4G drwxr-xr-x 2 www-data adm 4.0K Jun 29 09:15 . drwxrwxr-x 16 root syslog 4.0K Jun 28 00:00 .. -rw-r--r-- 1 www-data adm 1.1G Jun 29 09:15 access.log -rw-r--r-- 1 www-data adm 312M Jun 29 09:15 error.log -rw-r--r-- 1 www-data adm 2.4K Jun 28 23:59 access.log.1
Lists files with permissions, owner, group, size, and timestamp. -l long format, -a includes dotfiles, -h prints sizes in human-readable K/M/G notation.
access.log exceeds 1 GB, review your logrotate configuration.find — Search for files matching conditions
$ find /var/log -name "*.log" -mtime -7 -size +100M /var/log/nginx/access.log /var/log/postgresql/postgresql-2026-06-29_0000.log$ find /opt/app -name “.conf” -not -path “/node_modules/*” /opt/app/config/app.conf /opt/app/config/db.conf
Recursively locates files matching the given predicates. -mtime -7 matches files modified within the last 7 days, -size +100M filters by size, -not -path skips specific subtrees like node_modules.
DiskPressure state, identify log files modified in the past 7 days that exceed 100 MB. Then use truncate -s 0 /path/to/file.log to zero them out without restarting the service.tail -f — Stream a log file in real time
$ tail -f /var/log/nginx/access.log | grep -v "health\|readiness" 10.0.1.45 - - [29/Jun/2026:09:22:14 +0000] "POST /api/deploy HTTP/1.1" 200 348 10.0.1.12 - - [29/Jun/2026:09:22:15 +0000] "GET /api/status HTTP/1.1" 503 27 10.0.1.45 - - [29/Jun/2026:09:22:16 +0000] "GET /metrics HTTP/1.1" 200 1842
The -f flag streams new lines as they are appended. Piping through grep -v filters out Kubernetes health and readiness probe requests — without this, probes every 10 seconds drown the signal.
| grep " 503 " to isolate error lines, or | grep -c " 503 " to count them per second.rsync — Sync files and directories, locally or remotely
$ rsync -avz --progress --exclude="*.tmp" \
/opt/app/data/ backup@10.0.0.5:/backups/app-$(date +%Y%m%d)/
sending incremental file list
config/app.conf
1,024 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/3)
sent 1,234 bytes received 35 bytes 1,381.07 bytes/sec
total size is 1,024 speedup is 0.80
Transfers only changed blocks (delta transfer), making it far faster than cp -r for incremental backups. -a archive mode preserves permissions and timestamps, -z compresses in transit, --progress shows per-file transfer status.
--delete to mirror deletions from the source, and --checksum instead of --update when you need byte-level verification.grep -rn — Recursively search file contents
$ grep -rn "DB_PASSWORD" /etc/app/ --include="*.conf" /etc/app/config/db.conf:7:DB_PASSWORD=s3cr3t!$ grep -rn “error|critical” /var/log/ —include=“*.log” -l /var/log/nginx/error.log /var/log/app/app.log
-r descends recursively, -n prints the line number, --include limits the file glob, -l lists only filenames of matching files. Add -i for case-insensitive matching.
grep -rn "password\|secret\|token" /etc/ --include="*.conf" 2>/dev/null is the standard sweep.tar — Archive and extract files or directories
$ tar -czf config-backup-$(date +%Y%m%d).tar.gz /opt/app/config/ tar: Removing leading '/' from member names$ tar -tzf config-backup-20260629.tar.gz | head -4 opt/app/config/ opt/app/config/app.conf opt/app/config/db.conf
$ tar -xzf config-backup-20260629.tar.gz -C /tmp/restore/
-c creates an archive, -x extracts, -t lists contents without extracting, -z gzip compression, -f specifies the filename. Always verify with -t before extracting to confirm the archive is intact.
tar -xzf config-backup-YYYYMMDD.tar.gz -C /opt/app/ restores the previous state in seconds.🟡 Disk Management
Disk Management
df -hT — Show disk usage including filesystem type
$ df -hT Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext4 40G 31G 6.8G 82% / /dev/sdb1 xfs 500G 423G 77G 85% /data tmpfs tmpfs 7.8G 1.2G 6.6G 16% /dev/shm overlay overlay 40G 31G 6.8G 82% /var/lib/docker/overlay2/a1b2c3
-h prints human-readable sizes, -T shows the filesystem type (ext4, xfs, tmpfs, overlay). Overlay entries are Docker container layers — a large number of them means it’s time for docker system prune.
DiskPressure state. If /data is above 85%, intervene before kubelet starts evicting pods.du -sh * | sort -rh — Sort directories by size, largest first
$ du -sh /var/log/* | sort -rh | head -8 1.1G /var/log/nginx 423M /var/log/postgresql 188M /var/log/journal 42M /var/log/audit 18M /var/log/syslog 3.2M /var/log/kern.log
Reports the total disk usage of each path. -s summarises (no per-subdirectory recursion), -h human-readable sizes. sort -rh orders by human-readable size, descending.
journalctl --vacuum-size=500M for the systemd journal, or logrotate -f /etc/logrotate.conf for everything else.lsblk — List block devices and their mount points
$ lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE NAME SIZE TYPE MOUNTPOINT FSTYPE sda 100G disk ├─sda1 50G part / ext4 └─sda2 50G part /data xfs sdb 500G disk └─sdb1 500G part /backups ext4 sr0 1024M rom
Unlike fdisk -l, this requires no root privileges and renders the device tree with mount points in a single view. It’s the first command to run after attaching a new disk or EBS volume to identify its device path.
/dev/xvdf). Then format it with mkfs.xfs /dev/xvdf and mount it at the desired path.ncdu — Interactive disk usage navigator
$ ncdu /var --exclude=/var/lib/docker -x
ncdu 1.19 ~ Use the arrow keys to navigate, press ? for help
--- /var -----------------------------------------------------------------------
423.1 MiB [##########] /log
88.4 MiB [## ] /cache
12.3 MiB [ ] /spool
1.2 MiB [ ] /tmp
The interactive, navigable version of du. Arrow keys to browse, d to delete directly. -x stays within the current filesystem boundary (won’t cross into NFS or Docker overlay mounts), --exclude skips the specified path entirely.
sudo ncdu / -x --exclude=/proc --exclude=/sys is the safe invocation on live systems.iotop -o — Identify processes generating disk I/O
$ sudo iotop -o -d 2 Total DISK READ: 84.23 M/s | Total DISK WRITE: 12.44 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 4821 be/4 postgres 82.11 M/s 0.00 B/s 0.00 % 98.12 % postgres: autovacuum 723 be/3 root 1.44 M/s 12.44 M/s 0.00 % 4.21 % jbd2/sda1-8
-o shows only processes with active I/O (keeps the output clean), -d 2 refreshes every 2 seconds. Think of it as top for storage — it immediately surfaces which process is saturating the disk rather than the CPU.
iostat reports high I/O wait but the culprit is unclear, run sudo iotop -o. If PostgreSQL autovacuum is reading 80 MB/s, increase autovacuum_vacuum_cost_delay to spread its I/O load over time.🟢 Network & Troubleshooting
Network & Troubleshooting
ss -tulpn — List listening ports with owning processes
$ ss -tulpn | grep LISTEN
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1234))
tcp LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=4521))
tcp LISTEN 0 511 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=4521))
tcp LISTEN 0 128 127.0.0.1:5432 0.0.0.0:* users:(("postgres",pid=7823))
The modern replacement for netstat. -t TCP, -u UDP, -l listening only, -p process name and PID, -n skip DNS resolution (faster). Notice PostgreSQL is bound only to localhost — that’s correct; Nginx binds all interfaces.
ss -tulpn | grep 8080 returns nothing, the process either failed to start or bound to a different port.curl — Send HTTP requests with timing breakdown
$ curl -sI -o /dev/null \
-w "HTTP %{http_code} DNS %{time_namelookup}s Connect %{time_connect}s Total %{time_total}s\n" \
https://api.kubeatlas.com/health
HTTP 200 DNS 0.003s Connect 0.021s Total 0.143s
-s silent mode, -I HEAD request (no body download), -o /dev/null discards the body, -w custom output format. The timing breakdown isolates DNS resolution, TCP connect, and total time individually — you know exactly where latency is coming from.
-k to skip certificate validation on staging environments with self-signed certs.tcpdump — Capture network packets
$ tcpdump -i eth0 -n -s 0 'port 5432 and host 10.0.1.20' -w /tmp/pg-capture.pcap tcpdump: listening on eth0, link-type EN10MB (Ethernet) ^C 847 packets captured 847 packets received by filter 0 packets dropped by kernel
-i interface, -n skip DNS resolution (speed), -s 0 full packet capture, -w write to file for Wireshark analysis. The BPF filter 'port 5432 and host IP' limits capture to the relevant traffic.
pgsql display filter and inspect query/response timing.dig — Query DNS resolution
$ dig +short @8.8.8.8 kubeatlas.com A 93.184.216.34$ dig +trace kubeatlas.com | tail -4 kubeatlas.com. 300 IN A 93.184.216.34 ;; Received 56 bytes from 205.251.196.1#53(ns-1.awsdns-00.com) in 8 ms
+short returns just the IP address. +trace walks the full resolution chain from root nameservers — you see exactly which authoritative server served the final answer. Specifying @8.8.8.8 bypasses your ISP’s cache.
@1.1.1.1, @8.8.8.8, and @208.67.222.222 (OpenDNS). Differing answers mean the TTL hasn't expired on some resolvers yet.mtr — Analyse the network path in real time
$ mtr --report --report-cycles 20 8.8.8.8 HOST: prod-node1 Loss% Snt Last Avg Best Wrst StDev 1.|-- 10.0.0.1 0.0% 20 0.4 0.5 0.3 0.8 0.1 2.|-- 203.0.113.1 0.0% 20 1.2 1.4 1.1 2.1 0.3 3.|-- 198.51.100.45 5.0% 20 11.3 12.1 10.8 18.4 2.1 4.|-- 8.8.8.8 0.0% 20 12.8 13.2 12.4 15.1 0.7
Combines traceroute and ping in a single tool — every hop shows packet loss percentage and latency statistics. --report produces a one-shot summary, --report-cycles 20 controls the sample count. Strictly superior to plain traceroute: you get quality data, not just the route.
nc -zv — Test TCP port reachability
$ nc -zv 10.0.1.20 5432 Connection to 10.0.1.20 5432 port [tcp/postgresql] succeeded!$ nc -zv 10.0.1.20 6379 nc: connect to 10.0.1.20 port 6379 (tcp) failed: Connection refused
$ nc -zvw 3 10.0.1.20 443 Connection to 10.0.1.20 443 port [tcp/https] succeeded!
-z connects and immediately closes (no data sent), -v verbose output, -w 3 three-second timeout. The fastest way to test port reachability without telnet or nmap — available inside most containers too.
nc -zv postgres-svc 5432 tests Service DNS resolution. From the node, nc -zv <pod-ip> 5432 tests network policy separately.🟣 Process & System Monitoring
Process & System Monitoring
ps aux --sort=-%mem — Rank processes by memory consumption
$ ps aux --sort=-%mem | head -8 USER PID %CPU %MEM VSZ RSS STAT COMMAND postgres 4821 1.2 18.4 2145236 752184 Sl postgres: autovacuum worker node 7342 0.4 8.7 982340 356472 Sl /usr/bin/node server.js java 3201 2.1 7.2 4256789 294120 Sl java -jar app.jar root 1234 0.0 0.1 72348 4096 Ss /usr/sbin/sshd -D
a all users, u user-friendly format, x includes non-terminal processes. --sort=-%mem orders by descending memory. RSS is actual physical memory; VSZ is virtual address space — focus on RSS.
autovacuum_work_mem in your PostgreSQL configuration.journalctl — Query the systemd service journal
$ journalctl -u nginx.service -f --since "30 minutes ago" -p err Jun 29 09:14:22 prod-node1 nginx[4521]: [error] upstream timed out (110: Connection timed out) Jun 29 09:14:23 prod-node1 nginx[4521]: [error] connect() failed (111: Connection refused) Jun 29 09:14:25 prod-node1 nginx[4521]: [error] no live upstreams while connecting to upstream
-u filters by service unit, -f tails live, --since sets a time window, -p err shows only error priority and above. Use --output=json-pretty when you need structured log data for programmatic parsing.
kubectl get endpoints in a separate pane — if the endpoint list is empty, no healthy pods are registered.lsof -i — Identify which process owns a port
$ lsof -i :8080 -n -P COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME node 7342 deploy 22u IPv4 123456 0t0 TCP *:8080 (LISTEN) node 7342 deploy 24u IPv4 123457 0t0 TCP 10.0.1.5:8080->10.0.1.20:54321 (ESTABLISHED)
-i :PORT filters by port, -n suppresses hostname lookups, -P suppresses port-to-service-name translation. You see both the listening socket and all established connections with their remote endpoints.
lsof -i :8080 to find the stale process, then kill -15 PID for a graceful shutdown. Wait 5 seconds before restarting — TIME_WAIT sockets clear on their own.strace -p — Trace system calls of a running process
$ strace -p 7342 -e trace=network,file -T 2>&1 | head -10
strace: Process 7342 attached
read(28, "", 4096) = 0 <0.000012>
connect(29, {sa_family=AF_INET, sin_port=htons(5432)}, 16) = 0 <0.021453>
read(29, "\x00\x00\x00...", 4096) = 256 <0.143211>
write(29, "SELECT ...", 512) = 512 <0.000089>
Attaches to a running process and reports system calls in real time. -e trace=network,file limits output to network and file calls (reduces noise significantly), -T appends the wall-clock duration of each call. Introduces minor overhead — use on production with awareness.
-p PID -e trace=network -T and watch the read() call on the PostgreSQL socket. A 143 ms read indicates a slow query, not application logic.free -h — Snapshot current memory usage
$ free -h
total used free shared buff/cache available
Mem: 15Gi 8.2Gi 1.1Gi 312Mi 5.9Gi 6.8Gi
Swap: 2.0Gi 0.0Gi 2.0Gi
$ watch -n 2 free -h
used is active process consumption; buff/cache is the kernel’s page cache — the OS reclaims it under pressure. The column to watch is available, not free. Any non-zero swap usage signals active memory pressure that needs immediate attention.
watch -n 2 free -h tracks it every 2 seconds. If available drops below 10% of total RAM, reduce JVM heap size or PostgreSQL shared_buffers before OOMKiller fires.vmstat 1 — CPU, memory, and I/O in one view
$ vmstat 1 5 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 1124352 204288 6029312 0 0 182 124 1842 3241 18 4 72 6 0 1 0 0 1123840 204288 6029312 0 0 0 856 2103 3891 22 5 71 2 0
r is the run queue (processes waiting for CPU), b is blocked on I/O, wa is the percentage of CPU time wasted waiting for disk. si/so are swap-in and swap-out — non-zero values signal serious memory pressure. Arguments: interval in seconds, then sample count.
systemctl — Manage and inspect services
$ systemctl status nginx.service ● nginx.service - A high performance web server Loaded: loaded (/lib/systemd/system/nginx.service; enabled) Active: active (running) since Mon 2026-06-29 08:00:01 UTC; 1h 22min ago Main PID: 4521 (nginx)$ systemctl list-units —state=failed UNIT LOAD ACTIVE SUB DESCRIPTION app.service loaded failed failed Application Service
status shows service state, last log lines, and the main PID in one view. list-units --state=failed scans every unit for failures at a glance. is-active and is-enabled return exit codes usable in shell conditionals without parsing output.
journalctl -u SERVICE_NAME --since boot -p err to read the exact failure reason.🔴 File Permissions & Ownership
File Permissions & Ownership
chmod — Set file permissions
$ chmod 640 /etc/app/secrets/db.conf $ ls -la /etc/app/secrets/db.conf -rw-r----- 1 app app 512 Jun 29 09:00 /etc/app/secrets/db.conf$ chmod 755 /opt/app/bin/start.sh $ chmod 444 /etc/nginx/nginx.conf
Octal notation: owner / group / others with r=4, w=2, x=1. 640 → owner reads and writes, group reads, others have no access. Secrets: 640 or 600. Executable scripts: 755. Static configs: 644. Read-only enforcement: 444.
chown -R — Change file ownership recursively
$ chown -R app:app /opt/myapp/ $ ls -la /opt/myapp/ total 16 drwxr-xr-x 4 app app 4096 Jun 29 09:00 . drwxr-xr-x 8 root root 4096 Jun 28 00:00 .. drwxr-xr-x 2 app app 4096 Jun 29 09:00 config drwxr-xr-x 2 app app 4096 Jun 29 09:00 logs
-R applies the change recursively to all files and subdirectories. The user:group syntax sets both owner and group in a single call. Application directories should never be owned by root — always transfer ownership to a dedicated service account.
systemctl restart myapp.find security audit — Locate world-writable files
$ find /opt/app -type f -perm /o+w /opt/app/config/app.conf /opt/app/tmp/upload-cache$ find /etc -name “*.conf” -perm -0002 2>/dev/null (no output = clean)
-perm /o+w (equivalent to -perm /002) finds files where the “others” write bit is set. Any such file under /etc is a security finding. Embed this check in your CI pipeline to catch permission regressions on every deploy.
chmod o-w and encode the correct mode in your configuration management (Ansible/Chef) to prevent regression.stat — Show full file metadata
$ stat /etc/nginx/nginx.conf File: /etc/nginx/nginx.conf Size: 1742 Blocks: 8 IO Block: 4096 regular file Device: 8,1 Inode: 131074 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-29 08:14:33.000000000 +0000 Modify: 2026-06-15 11:22:07.000000000 +0000 Change: 2026-06-15 11:22:07.000000000 +0000
Unlike ls -l, stat exposes the inode number and separates atime (last accessed), mtime (content modified), and ctime (metadata changed). Use -c "%n %a %U:%G" to get name, octal mode, and owner in a script-friendly format.
mtime and ctime differ, only metadata (permissions or ownership) was changed — the content itself is unchanged. Critical for change audits.getfacl / setfacl — Fine-grained access control lists
$ getfacl /opt/app/logs/ # file: opt/app/logs/ # owner: app # group: app user::rwx group::r-x other::--- user:monitoring:r-x$ setfacl -m u:monitoring:rx /opt/app/logs/ $ setfacl -Rm d:u:monitoring:rx /opt/app/logs/
Grants per-user permissions beyond the standard owner/group/other Unix model. The d: (default) ACL applies automatically to all files and subdirectories created within the directory. No ownership change needed — ideal for granting read access to a monitoring agent or log shipper without touching the application’s user.
app user's ownership. setfacl -Rm u:prometheus:rx /opt/app/logs/ covers both existing and future files in one pass.