Operating systems for backend engineers series: 1. Kernel, processes and threads · 2. Virtual memory and the page cache · 3. I/O models and epoll · 4. File systems and the shell
A lot of operational incidents touch the file system: the disk is still full after you deleted the logs, a container can’t write to its volume, a deploy script replaces the wrong string in a manifest. After this post you can explain inodes, links and permissions, and write safe shell scripts through three exercises I actually ran. File descriptors and I/O are covered in part 3.
At a glance:
flowchart TD
DIR["Directory: name → inode"] --> IN["Inode + data"]
HL[Hard link] -->|adds a name| IN
SL[Soft link] -->|holds a path| DIR
RM[rm] -->|removes one name| DIR
IN --> FR{"Links = 0 and nobody has it open?"}
FR -->|yes| OK[Freed]
FR -->|no| DF["df still full: lsof +L1"]
PERM["Permissions rwx = 4/2/1"] --> DEL["Delete: parent directory permission"]
SH["Script: set -euo pipefail"]
1. The file system and “everything is a file”
Key idea: the file system manages persistent data: naming, directories, allocating blocks, permissions. “Everything is a file” means one set of system calls (open, read, write, close) works on files, sockets, pipes, devices and even kernel state.
For example, /dev/null and /dev/urandom are devices, and cat /proc/<pid>/status reads process state. That’s why ls /proc/<pid>/fd | wc -l is enough to count a JVM’s open files and connections.
The FHS tree: directories a backend engineer touches most
Linux follows a standard directory tree (FHS, the Filesystem Hierarchy Standard: a convention for which directory holds what):
| Directory | Contains | Why a backend cares |
|---|---|---|
/etc |
system configuration | /etc/hosts, /etc/resolv.conf (DNS inside containers), /etc/security/limits.conf |
/var |
constantly changing data | /var/log, /var/lib/mysql, /var/lib/docker (a classic disk filler) |
/tmp |
temporary files, sticky bit set | default java.io.tmpdir, Spring’s temporary uploads |
/proc |
virtual file system generated by the kernel | /proc/<pid>/fd, /proc/<pid>/limits, /proc/meminfo |
/sys |
virtual file system for devices and cgroups | a container’s CPU/RAM limits live under /sys/fs/cgroup |
2. Inodes, directory entries, hard links and soft links
Key idea: a file’s name is not stored in the file. The inode holds metadata and pointers to the data; the directory holds a “name → inode number” table. A hard link adds a name for the same inode, a soft link is a separate file holding a path.
- Inode: the structure describing a file. It has an inode number unique within the file system, size, owner, permissions, timestamps, link count and pointers to the data blocks. No name.
- Directory entry (dentry): one “name → inode number” row in a directory. A path like
/home/ronin/app.logis resolved by walking each directory and looking up the name.
Hard link (ln a b) |
Soft/symbolic link (ln -s a b) |
|
|---|---|---|
| What it is | another name pointing to the same inode | a separate file (own inode) storing the path to the target |
| Delete the original | data survives, the inode still has a name | link breaks (dangling) |
| Rename or move the original | no effect | breaks, the stored path is now wrong |
| Across file systems, to a directory | not allowed | allowed |
| Used for | rsync --link-dest style backups |
current -> releases/2026-11-18 for deploys and rollbacks |
Trying it on macOS (ls -li shows the inode number and link count):
$ echo hello > a.txt; ln a.txt hard.txt; ln -s a.txt soft.txt; ls -li
253700484 -rw-r--r-- 2 vinh wheel 6 a.txt
253700484 -rw-r--r-- 2 vinh wheel 6 hard.txt
253700487 lrwxr-xr-x 1 vinh wheel 5 soft.txt -> a.txt
$ rm a.txt; cat hard.txt; cat soft.txt
hello
cat: soft.txt: No such file or directory
a.txt and hard.txt share an inode number and the link count is 2. After deleting a.txt the count drops to 1 and the data is still readable through hard.txt; soft.txt is broken.
Log rotation: create vs copytruncate
Key idea: create renames the old file and creates a new one, so the process must be told to reopen it. copytruncate copies and then truncates the original to zero; the process needn’t know, but lines can be lost.
Log rotation means splitting a log into several files so none grows forever. logrotate can do it two ways:
create(the default): renamesapp.logtoapp.log.1and creates a new file. The process still holds an fd to the old inode, so it keeps writing intoapp.log.1until it’s told to reopen (apostrotatesignal, ornginx -s reopen).copytruncate: copies toapp.log.1and truncatesapp.logto zero bytes. Lines written between the copy and the truncate are lost; if the file wasn’t opened withO_APPEND(always write at the end) the start of the file becomes a run of zero bytes.
3. Permissions
Key idea: every file has an owner, a group and three rwx sets for user, group and other, with bit values 4, 2 and 1. The three sets become three octal digits, such as 750.
| Permission | Pattern | Value | On a file | On a directory |
|---|---|---|---|---|
| read | r-- |
4 | read contents | list the names inside |
| write | -w- |
2 | modify contents | create, delete, rename entries inside |
| execute | --x |
1 | run as a program | enter it (cd), access files inside |
Why 4, 2, 1? They are the three bits of a three-digit binary number (100, 010, 001). Every combination adds up to a unique number from 0 to 7 that you can decode back: 5 = 4 + 1 = r-x, 6 = rw-, 7 = rwx.
- Remove all permissions:
chmod 000 file(root can still read it; permissions don’t stop root). - The course’s example, owner full access, group read and execute, others nothing:
chmod 750 deploy.sh(7 =rwx, 5 =r-x, 0 =---), orchmod u=rwx,g=rx,o= deploy.sh.
Reading ls -l, umask, the sticky bit and setuid
Reading ls -l: -rwxr-x--- 1 vinh wheel 2 p.sh. The first character is the file type (- regular, d directory, l symlink, s socket, p pipe, c/b device). Then come three rwx triplets for user, group and other, then link count, owner, group and size.
umask (a permission mask) sets the default permissions of new files: files start from 666 and directories from 777, then the umask bits are switched off. A umask of 022 gives 644 files and 755 directories; 077 gives 600 files (owner read/write only), which suits files holding secrets.
Two special bits you’ll meet:
- Sticky bit on a directory (
drwxrwxrwton/tmp): anyone can create files, but only a file’s owner can delete it. - setuid on an executable (
-rwsr-xr-xon/usr/bin/passwd): the program runs with the file owner’s privileges (root), not the caller’s. Linux ignores setuid on scripts.
Don’t run containers as root
Key idea: by default a container process runs as UID 0 (root). If the app is exploited, the attacker is root, so run as an ordinary user and allow writes only where they’re needed.
Think Log4Shell: the attacker is root inside the container, and closer to the host when the setup is loose (a mounted docker.sock, privileged).
FROM eclipse-temurin:21-jre
RUN groupadd --system app && useradd --system --gid app --no-create-home app
COPY --chown=app:app target/app.jar /app/app.jar
USER app
ENTRYPOINT ["java", "-jar", "/app/app.jar"]
In Kubernetes add securityContext: { runAsNonRoot: true, readOnlyRootFilesystem: true } and mount an emptyDir on /tmp (the JVM and Spring need to write temporary files). A complete non-root Dockerfile, plus PID 1 and healthchecks, is in Docker for Spring Boot.
4. Everyday commands
Key idea: read logs with tail/grep/sed/awk, call and read APIs with curl/jq, and inspect connections and open files with ss/lsof. To follow a log, use tail -F, not tail -f.
Everyday commands with real-world examples
| Command | Used for | Real-world example |
|---|---|---|
head, tail |
first/last lines of a file | tail -n 200 app.log, tail -F app.log to follow a log |
grep |
filter lines by pattern | grep -c 'ERROR' app.log, grep -B2 -A10 'Exception' app.log |
sed |
stream editor: substitute, delete lines | sed -n '100,120p' app.log, sed 's/old/new/g' |
awk |
a small language for column-oriented data | awk '{print $9}' access.log | sort | uniq -c counts by status code |
curl |
make HTTP requests | curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' localhost:8080/actuator/health |
wget |
download over HTTP, HTTPS, FTP | wget -q <url> |
jq |
read and filter JSON | curl -s .../actuator/metrics/jvm.threads.live | jq '.measurements[0].value' |
ss |
inspect sockets | ss -tanp state established '( dport = :5432 )' counts connections to Postgres |
lsof |
open files and sockets | lsof -p <pid>, lsof -i :8080 to see who holds a port |
5. Shells and shell scripts
Key idea: zsh is fine for typing commands every day, but scripts that run on servers and in CI should target bash or POSIX sh with an explicit shebang. Start every script with set -euo pipefail and quote your variables.
Common shells: sh (POSIX; on Debian/Ubuntu it’s dash), bash (the default on most Linux distributions), csh (rarely used), zsh (the default on macOS). The slides recommend zsh, which is right for your interactive shell. But Alpine images only ship BusyBox ash, and macOS still bundles bash 3.2 (no mapfile).
A safe script skeleton (the shebang is the #! first line that picks the interpreter):
#!/usr/bin/env bash
set -euo pipefail # -e: stop on errors; -u: unset variables are errors; pipefail: don't swallow errors mid-pipe
IFS=$'\n\t' # split words on newlines and tabs, not spaces
main() {
local dir="${1:?usage: $0 <dir>}" # required argument with a clear error
[[ -d "$dir" ]] || { echo "not a directory: $dir" >&2; exit 1; }
# ...
}
main "$@"
When to drop shell for a real language
Exercises
I ran these scripts on macOS (bash, BSD sed/date) in a scratch directory outside the repo; differences from Linux (GNU) are noted.
Exercise 1: Rename images with a date prefix
Task: in a directory, rename every image to YYYY-MM-DD_<old name>. I picked the file’s modification time (mtime); using the EXIF capture date would need exiftool. Extra requirements: names with spaces, upper-case extensions, re-running must not prefix twice, and a dry-run mode.
Solution 1: an idempotent image-rename script
#!/usr/bin/env bash
# Usage: ./rename-images.sh [-n] [dir] (-n = dry run, print only)
set -euo pipefail
dry_run=false
if [[ "${1:-}" == "-n" ]]; then dry_run=true; shift; fi
dir="${1:-.}"
shopt -s nullglob nocaseglob # no match means an empty loop; .JPG matches too
count=0
for f in "$dir"/*.{jpg,jpeg,png,gif,webp,heic}; do
name=$(basename -- "$f")
[[ "$name" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}_ ]] && continue # already prefixed
day=$(date -r "$f" +%Y-%m-%d) # date -r FILE works on both GNU and BSD
target="$dir/${day}_$name"
if [[ -e "$target" ]]; then echo "skip (exists): $target" >&2; continue; fi
if $dry_run; then
echo "would rename: $f -> $target"
else
mv -n -- "$f" "$target"
echo "renamed: $f -> $target"
fi
count=$((count + 1))
done
echo "$count file(s) $($dry_run && echo 'to rename' || echo 'renamed')"
A run with beach day.jpg (mtime 2024-05-01), xmas.PNG (2023-12-24), notes.txt and 2024-01-01_old.jpg:
$ ./rename-images.sh -n photos
would rename: photos/beach day.jpg -> photos/2024-05-01_beach day.jpg
would rename: photos/xmas.PNG -> photos/2023-12-24_xmas.PNG
2 file(s) to rename
$ ./rename-images.sh photos # renames for real, prints 2 "renamed" lines
$ ./rename-images.sh photos
0 file(s) renamed
Key points: quoted variables keep names with spaces intact; -- protects against names starting with -; mv -n never overwrites; the prefix regex makes the script idempotent (running it again changes nothing).
Exercise 2: Replace $IMAGE_TAG in a manifest with sed and awk
Task: deployment.yaml has a line image: $IMAGE_TAG; replace it with ronin:v0.0.1, once with sed and once with awk. I added a trap line to the test file: value: "do not touch $IMAGE_TAG_SUFFIX".
Solution 2: sed, awk and the sed -i trap between macOS and Linux
IMAGE_TAG='ronin:v0.0.1'
# sed: | as delimiter since values may contain '/'; \$ matches a literal $; anchored at end of line
sed "s|image: \\\$IMAGE_TAG\$|image: ${IMAGE_TAG}|" deployment.yaml > deployment.out.yaml
# awk: compare fields instead of running a regex over the whole line
awk -v tag="$IMAGE_TAG" '$1 == "image:" && $2 == "$IMAGE_TAG" { sub(/\$IMAGE_TAG$/, tag) } { print }' \
deployment.yaml > deployment.out.yaml
# in place: -i.bak (no space) works with both GNU sed and BSD sed
sed -i.bak "s|image: \\\$IMAGE_TAG\$|image: ${IMAGE_TAG}|" deployment.yaml
Result: diff shows only line 10 changed to image: ronin:v0.0.1; the sed and awk outputs are identical; the trap line is untouched. The naive sed "s|\$IMAGE_TAG|$IMAGE_TAG|g" turns the trap line into do not touch ronin:v0.0.1_SUFFIX.
In a real pipeline sed is fragile: it can match the wrong thing, it doesn’t understand YAML, and it stays silent when nothing matches (exit code 0). Better options:
envsubst '$IMAGE_TAG': substitutes only the listed variables and leaves$IMAGE_TAG_SUFFIXalone (I ran it; the trap line survived).yq: edits by YAML path, e.g.yq -i '.spec.template.spec.containers[0].image = strenv(IMAGE_TAG)' deployment.yaml.- kustomize (
kustomize edit set image app=ronin:v0.0.1) or Helm (--set image.tag=v0.0.1): the standard Kubernetes tools, with diffs and rollbacks.
Exercise 3: A script to open your workspace
Task: a script that opens the editor, the browser and Postman and starts the containers you need; for example ./container.sh mysql up.
Solution 3: container.sh with a whitelist and DRY_RUN
#!/usr/bin/env bash
# Usage: ./container.sh <service> <up|down|logs|status> e.g. ./container.sh mysql up
# ./container.sh workspace editor, browser, Postman, mysql + redis
set -euo pipefail
COMPOSE_FILE="${COMPOSE_FILE:-$HOME/workspace/docker-compose.yml}"
SERVICES=(mysql redis kafka)
DRY_RUN="${DRY_RUN:-false}"
run() { echo "+ $*"; if [[ "$DRY_RUN" != "true" ]]; then "$@"; fi; }
usage() {
local IFS='|'
echo "usage: $0 <${SERVICES[*]}> <up|down|logs|status> | $0 workspace" >&2
exit 2
}
open_app() { # macOS: open -a "App"; Linux: run the command if installed
case "$(uname -s)" in
Darwin) run open -a "$1" ;;
Linux) if command -v "$2" >/dev/null; then run "$2" >/dev/null 2>&1 & else echo "skip: $2" >&2; fi ;;
esac
}
open_url() { if [[ "$(uname -s)" == Darwin ]]; then run open "$1"; else run xdg-open "$1"; fi; }
compose() { run docker compose -f "$COMPOSE_FILE" "$@"; }
[[ $# -ge 1 ]] || usage
if [[ "$1" == "workspace" ]]; then
open_app "Visual Studio Code" code
open_app "Postman" postman
open_url "http://localhost:8080/swagger-ui.html"
compose up -d mysql redis
exit 0
fi
[[ $# -eq 2 ]] || usage
service="$1"; action="$2"
[[ " ${SERVICES[*]} " == *" $service "* ]] || { echo "unknown service: $service" >&2; usage; }
case "$action" in
up) compose up -d "$service" ;;
down) compose stop "$service" ;;
logs) compose logs -f --tail=100 "$service" ;;
status) compose ps "$service" ;;
*) usage ;;
esac
A run with DRY_RUN=true (print the commands only):
$ DRY_RUN=true ./container.sh mysql up
+ docker compose -f /Users/vinh/workspace/docker-compose.yml up -d mysql
$ DRY_RUN=true ./container.sh workspace
+ open -a Visual Studio Code
+ open -a Postman
+ open http://localhost:8080/swagger-ui.html
+ docker compose -f /Users/vinh/workspace/docker-compose.yml up -d mysql redis
$ ./container.sh mongo up; echo "exit=$?"
unknown service: mongo
usage: ./container.sh <mysql|redis|kafka> <up|down|logs|status> | ./container.sh workspace
exit=2
I did not run a real docker compose and did not test the Linux branch (xdg-open).
Design choices:
- Services are a whitelist, so typos are rejected.
downusesstopto keep data (down -vwould delete volumes).DRY_RUNlets CI check the script without Docker.- Exit code 2 signals a usage error.
Remember
- Explain where a file’s name lives, and how hard and soft links behave when the original is deleted or renamed.
- Know why
rm-ing a log that’s being written doesn’t free the disk, how to find it and how to fix it. - Read
chmod 750,ls -land a 022 umask; explain why the right to delete a file lives on the directory. - Know how to run a container as non-root and fix volume write errors without
chmod 777. - List what
set -euo pipefaildoesn’t catch, and whysedis fragile in a deploy pipeline.
Interview questions
Junior
What is an inode? Where is the file name stored?
An inode is the structure describing a file; the name lives in a directory entry of the containing directory.
The inode holds the inode number, size, owner, permissions, timestamps, link count and pointers to the data blocks. The directory maps name → inode number, so one inode can have several names (hard links).
How do hard links and soft links differ?
A hard link is another name for the same inode; a soft link is a separate file holding the target’s path.
Hard link: deleting the original name keeps the data, but it can’t cross file systems or point to a directory. Soft link: it can cross file systems and point to directories, but it breaks when the target is deleted, renamed or moved.
What does chmod 750 mean, and why are permissions 4, 2, 1?
User rwx (7 = 4 + 2 + 1), group r-x (5 = 4 + 1), others nothing (0).
4, 2 and 1 are the three bits of a three-digit binary number, so every combination sums to a unique value from 0 to 7 that decodes back unambiguously.
Mid
You deleted a huge log file but the disk is still full. Why, and what do you do?
The service still holds an fd to the file, so the inode and its blocks haven’t been freed.
rm only removes a name; data is freed only when the link count is zero and no process has the file open. So du can’t see it but df is still full. Find it with lsof +L1 and reclaim the space by restarting the process or running : > /proc/<pid>/fd/<n>.
What’s the difference between logrotate’s create and copytruncate?
create renames the old file and creates a new one; copytruncate copies the content elsewhere and truncates the original to zero.
With create, the process keeps writing to the old inode until it’s told to reopen. With copytruncate, the process needn’t know, but lines written between the copy and the truncate can be lost, and without O_APPEND you get a zero-filled region.
Who can delete a read-only (444) file?
Anyone with w and x on its directory, because deleting modifies the directory entry, not the file.
Shared directories like /tmp set the sticky bit so only a file’s owner (and root) can delete it.
Why shouldn’t containers run as root, and how do you fix volume write errors?
If the app is exploited the attacker is root in the container; fix volume write errors with fsGroup or an init container, not by going back to root.
Root inside the container can write anywhere in the image and gets closer to the host when the setup is loose. Use USER in the Dockerfile and runAsNonRoot plus readOnlyRootFilesystem in Kubernetes. Volumes keep their on-disk UID/GID, so a non-root user may not be able to write; never “fix” that with chmod 777.
Senior
A deploy script uses sed to replace the image tag. How do you review it?
sed can match the wrong thing, doesn’t understand YAML and stays silent when nothing matches; at minimum anchor and verify, better use yq, kustomize or Helm.
- Risks: matching other strings (
$IMAGE_TAG_SUFFIX), no YAML awareness (several containers, indentation), exit code 0 when nothing was replaced, andsed -idiffers between macOS and Linux. - At minimum: anchor the pattern and verify with
grep -q, failing if nothing was replaced. - Better:
envsubstwith an explicit variable list,yq, or kustomize/Helm; pin images by digest so deploys are reproducible.
Is set -euo pipefail enough to make a script safe?
No. -e ignores failures inside if conditions and &&/||, and local x=$(cmd) swallows cmd’s exit code.
You still need to quote every variable, use ${VAR:?} for required variables before dangerous commands like rm -rf, put -- before file names, run shellcheck in CI, and provide a dry-run mode.
How do you safely replace a binary or config file that’s in use?
Write the new version to a temp file on the same file system, then mv it into place, because rename is atomic.
Readers see either the old or the new version, never a half-written file, and running processes keep the old inode. Overwriting in place can cause Text file busy or mixed content. Deploying into versioned directories and switching a current symlink follows the same principle; rollback is just pointing the symlink back.
Useful phrases for interviews
- “The name lives in the directory entry; the inode holds the metadata and the block pointers.”
- “Disk space only comes back when the link count is zero and no process holds the file open.”
- “
lsof +L1shows files that were deleted but are still held open.” - “Deleting a file is a write to its directory, so the directory’s permissions decide.”
- “We run the container as a non-root user and fix volume ownership with
fsGroup, notchmod 777.” - “I’d use
tail -Frather thantail -fso it survives log rotation.” - “For manifests I’d rather use
yqor kustomize thansed, because sed silently succeeds when nothing matches.” - “The script is idempotent and has a dry-run mode, so it’s safe to run twice or in CI.”
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.