Skip to content
notes.vinhnq.com

Search

Git, DevOps & ToolingMid22 min read

Linux for backend engineers: inodes, links, permissions and shell scripts

Review 19 cards
On this page

Operating systems for backend engineers series: 1. Kernel, processes and threads · 2. Virtual memory and the page cache · 3. I/O models and epoll · 4. File systems and the shell

A lot of operational incidents touch the file system: the disk is still full after you deleted the logs, a container can’t write to its volume, a deploy script replaces the wrong string in a manifest. After this post you can explain inodes, links and permissions, and write safe shell scripts through three exercises I actually ran. File descriptors and I/O are covered in part 3.

At a glance:

flowchart TD
  DIR["Directory: name → inode"] --> IN["Inode + data"]
  HL[Hard link] -->|adds a name| IN
  SL[Soft link] -->|holds a path| DIR
  RM[rm] -->|removes one name| DIR
  IN --> FR{"Links = 0 and nobody has it open?"}
  FR -->|yes| OK[Freed]
  FR -->|no| DF["df still full: lsof +L1"]
  PERM["Permissions rwx = 4/2/1"] --> DEL["Delete: parent directory permission"]
  SH["Script: set -euo pipefail"]

1. The file system and “everything is a file”

Key idea: the file system manages persistent data: naming, directories, allocating blocks, permissions. “Everything is a file” means one set of system calls (open, read, write, close) works on files, sockets, pipes, devices and even kernel state.

For example, /dev/null and /dev/urandom are devices, and cat /proc/<pid>/status reads process state. That’s why ls /proc/<pid>/fd | wc -l is enough to count a JVM’s open files and connections.

The FHS tree: directories a backend engineer touches most

Linux follows a standard directory tree (FHS, the Filesystem Hierarchy Standard: a convention for which directory holds what):

Directory Contains Why a backend cares
/etc system configuration /etc/hosts, /etc/resolv.conf (DNS inside containers), /etc/security/limits.conf
/var constantly changing data /var/log, /var/lib/mysql, /var/lib/docker (a classic disk filler)
/tmp temporary files, sticky bit set default java.io.tmpdir, Spring’s temporary uploads
/proc virtual file system generated by the kernel /proc/<pid>/fd, /proc/<pid>/limits, /proc/meminfo
/sys virtual file system for devices and cgroups a container’s CPU/RAM limits live under /sys/fs/cgroup

Key idea: a file’s name is not stored in the file. The inode holds metadata and pointers to the data; the directory holds a “name → inode number” table. A hard link adds a name for the same inode, a soft link is a separate file holding a path.

  • Inode: the structure describing a file. It has an inode number unique within the file system, size, owner, permissions, timestamps, link count and pointers to the data blocks. No name.
  • Directory entry (dentry): one “name → inode number” row in a directory. A path like /home/ronin/app.log is resolved by walking each directory and looking up the name.
Hard link (ln a b) Soft/symbolic link (ln -s a b)
What it is another name pointing to the same inode a separate file (own inode) storing the path to the target
Delete the original data survives, the inode still has a name link breaks (dangling)
Rename or move the original no effect breaks, the stored path is now wrong
Across file systems, to a directory not allowed allowed
Used for rsync --link-dest style backups current -> releases/2026-11-18 for deploys and rollbacks

Trying it on macOS (ls -li shows the inode number and link count):

$ echo hello > a.txt; ln a.txt hard.txt; ln -s a.txt soft.txt; ls -li
253700484 -rw-r--r--  2 vinh  wheel  6 a.txt
253700484 -rw-r--r--  2 vinh  wheel  6 hard.txt
253700487 lrwxr-xr-x  1 vinh  wheel  5 soft.txt -> a.txt
$ rm a.txt; cat hard.txt; cat soft.txt
hello
cat: soft.txt: No such file or directory

a.txt and hard.txt share an inode number and the link count is 2. After deleting a.txt the count drops to 1 and the data is still readable through hard.txt; soft.txt is broken.

Log rotation: create vs copytruncate

Key idea: create renames the old file and creates a new one, so the process must be told to reopen it. copytruncate copies and then truncates the original to zero; the process needn’t know, but lines can be lost.

Log rotation means splitting a log into several files so none grows forever. logrotate can do it two ways:

  • create (the default): renames app.log to app.log.1 and creates a new file. The process still holds an fd to the old inode, so it keeps writing into app.log.1 until it’s told to reopen (a postrotate signal, or nginx -s reopen).
  • copytruncate: copies to app.log.1 and truncates app.log to zero bytes. Lines written between the copy and the truncate are lost; if the file wasn’t opened with O_APPEND (always write at the end) the start of the file becomes a run of zero bytes.

3. Permissions

Key idea: every file has an owner, a group and three rwx sets for user, group and other, with bit values 4, 2 and 1. The three sets become three octal digits, such as 750.

Permission Pattern Value On a file On a directory
read r-- 4 read contents list the names inside
write -w- 2 modify contents create, delete, rename entries inside
execute --x 1 run as a program enter it (cd), access files inside

Why 4, 2, 1? They are the three bits of a three-digit binary number (100, 010, 001). Every combination adds up to a unique number from 0 to 7 that you can decode back: 5 = 4 + 1 = r-x, 6 = rw-, 7 = rwx.

  • Remove all permissions: chmod 000 file (root can still read it; permissions don’t stop root).
  • The course’s example, owner full access, group read and execute, others nothing: chmod 750 deploy.sh (7 = rwx, 5 = r-x, 0 = ---), or chmod u=rwx,g=rx,o= deploy.sh.
Reading ls -l, umask, the sticky bit and setuid

Reading ls -l: -rwxr-x--- 1 vinh wheel 2 p.sh. The first character is the file type (- regular, d directory, l symlink, s socket, p pipe, c/b device). Then come three rwx triplets for user, group and other, then link count, owner, group and size.

umask (a permission mask) sets the default permissions of new files: files start from 666 and directories from 777, then the umask bits are switched off. A umask of 022 gives 644 files and 755 directories; 077 gives 600 files (owner read/write only), which suits files holding secrets.

Two special bits you’ll meet:

  • Sticky bit on a directory (drwxrwxrwt on /tmp): anyone can create files, but only a file’s owner can delete it.
  • setuid on an executable (-rwsr-xr-x on /usr/bin/passwd): the program runs with the file owner’s privileges (root), not the caller’s. Linux ignores setuid on scripts.

Don’t run containers as root

Key idea: by default a container process runs as UID 0 (root). If the app is exploited, the attacker is root, so run as an ordinary user and allow writes only where they’re needed.

Think Log4Shell: the attacker is root inside the container, and closer to the host when the setup is loose (a mounted docker.sock, privileged).

FROM eclipse-temurin:21-jre
RUN groupadd --system app && useradd --system --gid app --no-create-home app
COPY --chown=app:app target/app.jar /app/app.jar
USER app
ENTRYPOINT ["java", "-jar", "/app/app.jar"]

In Kubernetes add securityContext: { runAsNonRoot: true, readOnlyRootFilesystem: true } and mount an emptyDir on /tmp (the JVM and Spring need to write temporary files). A complete non-root Dockerfile, plus PID 1 and healthchecks, is in Docker for Spring Boot.

4. Everyday commands

Key idea: read logs with tail/grep/sed/awk, call and read APIs with curl/jq, and inspect connections and open files with ss/lsof. To follow a log, use tail -F, not tail -f.

Everyday commands with real-world examples
Command Used for Real-world example
head, tail first/last lines of a file tail -n 200 app.log, tail -F app.log to follow a log
grep filter lines by pattern grep -c 'ERROR' app.log, grep -B2 -A10 'Exception' app.log
sed stream editor: substitute, delete lines sed -n '100,120p' app.log, sed 's/old/new/g'
awk a small language for column-oriented data awk '{print $9}' access.log | sort | uniq -c counts by status code
curl make HTTP requests curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' localhost:8080/actuator/health
wget download over HTTP, HTTPS, FTP wget -q <url>
jq read and filter JSON curl -s .../actuator/metrics/jvm.threads.live | jq '.measurements[0].value'
ss inspect sockets ss -tanp state established '( dport = :5432 )' counts connections to Postgres
lsof open files and sockets lsof -p <pid>, lsof -i :8080 to see who holds a port

5. Shells and shell scripts

Key idea: zsh is fine for typing commands every day, but scripts that run on servers and in CI should target bash or POSIX sh with an explicit shebang. Start every script with set -euo pipefail and quote your variables.

Common shells: sh (POSIX; on Debian/Ubuntu it’s dash), bash (the default on most Linux distributions), csh (rarely used), zsh (the default on macOS). The slides recommend zsh, which is right for your interactive shell. But Alpine images only ship BusyBox ash, and macOS still bundles bash 3.2 (no mapfile).

A safe script skeleton (the shebang is the #! first line that picks the interpreter):

#!/usr/bin/env bash
set -euo pipefail          # -e: stop on errors; -u: unset variables are errors; pipefail: don't swallow errors mid-pipe
IFS=$'\n\t'                # split words on newlines and tabs, not spaces

main() {
  local dir="${1:?usage: $0 <dir>}"      # required argument with a clear error
  [[ -d "$dir" ]] || { echo "not a directory: $dir" >&2; exit 1; }
  # ...
}
main "$@"
When to drop shell for a real language

Exercises

I ran these scripts on macOS (bash, BSD sed/date) in a scratch directory outside the repo; differences from Linux (GNU) are noted.

Exercise 1: Rename images with a date prefix

Task: in a directory, rename every image to YYYY-MM-DD_<old name>. I picked the file’s modification time (mtime); using the EXIF capture date would need exiftool. Extra requirements: names with spaces, upper-case extensions, re-running must not prefix twice, and a dry-run mode.

Solution 1: an idempotent image-rename script
#!/usr/bin/env bash
# Usage: ./rename-images.sh [-n] [dir]   (-n = dry run, print only)
set -euo pipefail

dry_run=false
if [[ "${1:-}" == "-n" ]]; then dry_run=true; shift; fi
dir="${1:-.}"

shopt -s nullglob nocaseglob          # no match means an empty loop; .JPG matches too
count=0
for f in "$dir"/*.{jpg,jpeg,png,gif,webp,heic}; do
  name=$(basename -- "$f")
  [[ "$name" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}_ ]] && continue   # already prefixed
  day=$(date -r "$f" +%Y-%m-%d)       # date -r FILE works on both GNU and BSD
  target="$dir/${day}_$name"
  if [[ -e "$target" ]]; then echo "skip (exists): $target" >&2; continue; fi
  if $dry_run; then
    echo "would rename: $f -> $target"
  else
    mv -n -- "$f" "$target"
    echo "renamed: $f -> $target"
  fi
  count=$((count + 1))
done
echo "$count file(s) $($dry_run && echo 'to rename' || echo 'renamed')"

A run with beach day.jpg (mtime 2024-05-01), xmas.PNG (2023-12-24), notes.txt and 2024-01-01_old.jpg:

$ ./rename-images.sh -n photos
would rename: photos/beach day.jpg -> photos/2024-05-01_beach day.jpg
would rename: photos/xmas.PNG -> photos/2023-12-24_xmas.PNG
2 file(s) to rename
$ ./rename-images.sh photos      # renames for real, prints 2 "renamed" lines
$ ./rename-images.sh photos
0 file(s) renamed

Key points: quoted variables keep names with spaces intact; -- protects against names starting with -; mv -n never overwrites; the prefix regex makes the script idempotent (running it again changes nothing).

Exercise 2: Replace $IMAGE_TAG in a manifest with sed and awk

Task: deployment.yaml has a line image: $IMAGE_TAG; replace it with ronin:v0.0.1, once with sed and once with awk. I added a trap line to the test file: value: "do not touch $IMAGE_TAG_SUFFIX".

Solution 2: sed, awk and the sed -i trap between macOS and Linux
IMAGE_TAG='ronin:v0.0.1'

# sed: | as delimiter since values may contain '/'; \$ matches a literal $; anchored at end of line
sed "s|image: \\\$IMAGE_TAG\$|image: ${IMAGE_TAG}|" deployment.yaml > deployment.out.yaml

# awk: compare fields instead of running a regex over the whole line
awk -v tag="$IMAGE_TAG" '$1 == "image:" && $2 == "$IMAGE_TAG" { sub(/\$IMAGE_TAG$/, tag) } { print }' \
  deployment.yaml > deployment.out.yaml

# in place: -i.bak (no space) works with both GNU sed and BSD sed
sed -i.bak "s|image: \\\$IMAGE_TAG\$|image: ${IMAGE_TAG}|" deployment.yaml

Result: diff shows only line 10 changed to image: ronin:v0.0.1; the sed and awk outputs are identical; the trap line is untouched. The naive sed "s|\$IMAGE_TAG|$IMAGE_TAG|g" turns the trap line into do not touch ronin:v0.0.1_SUFFIX.

In a real pipeline sed is fragile: it can match the wrong thing, it doesn’t understand YAML, and it stays silent when nothing matches (exit code 0). Better options:

  • envsubst '$IMAGE_TAG': substitutes only the listed variables and leaves $IMAGE_TAG_SUFFIX alone (I ran it; the trap line survived).
  • yq: edits by YAML path, e.g. yq -i '.spec.template.spec.containers[0].image = strenv(IMAGE_TAG)' deployment.yaml.
  • kustomize (kustomize edit set image app=ronin:v0.0.1) or Helm (--set image.tag=v0.0.1): the standard Kubernetes tools, with diffs and rollbacks.

Exercise 3: A script to open your workspace

Task: a script that opens the editor, the browser and Postman and starts the containers you need; for example ./container.sh mysql up.

Solution 3: container.sh with a whitelist and DRY_RUN
#!/usr/bin/env bash
# Usage: ./container.sh <service> <up|down|logs|status>   e.g. ./container.sh mysql up
#        ./container.sh workspace                         editor, browser, Postman, mysql + redis
set -euo pipefail

COMPOSE_FILE="${COMPOSE_FILE:-$HOME/workspace/docker-compose.yml}"
SERVICES=(mysql redis kafka)
DRY_RUN="${DRY_RUN:-false}"

run() { echo "+ $*"; if [[ "$DRY_RUN" != "true" ]]; then "$@"; fi; }

usage() {
  local IFS='|'
  echo "usage: $0 <${SERVICES[*]}> <up|down|logs|status> | $0 workspace" >&2
  exit 2
}

open_app() {                      # macOS: open -a "App"; Linux: run the command if installed
  case "$(uname -s)" in
    Darwin) run open -a "$1" ;;
    Linux)  if command -v "$2" >/dev/null; then run "$2" >/dev/null 2>&1 & else echo "skip: $2" >&2; fi ;;
  esac
}

open_url() { if [[ "$(uname -s)" == Darwin ]]; then run open "$1"; else run xdg-open "$1"; fi; }

compose() { run docker compose -f "$COMPOSE_FILE" "$@"; }

[[ $# -ge 1 ]] || usage
if [[ "$1" == "workspace" ]]; then
  open_app "Visual Studio Code" code
  open_app "Postman" postman
  open_url "http://localhost:8080/swagger-ui.html"
  compose up -d mysql redis
  exit 0
fi

[[ $# -eq 2 ]] || usage
service="$1"; action="$2"
[[ " ${SERVICES[*]} " == *" $service "* ]] || { echo "unknown service: $service" >&2; usage; }

case "$action" in
  up)     compose up -d "$service" ;;
  down)   compose stop "$service" ;;
  logs)   compose logs -f --tail=100 "$service" ;;
  status) compose ps "$service" ;;
  *)      usage ;;
esac

A run with DRY_RUN=true (print the commands only):

$ DRY_RUN=true ./container.sh mysql up
+ docker compose -f /Users/vinh/workspace/docker-compose.yml up -d mysql
$ DRY_RUN=true ./container.sh workspace
+ open -a Visual Studio Code
+ open -a Postman
+ open http://localhost:8080/swagger-ui.html
+ docker compose -f /Users/vinh/workspace/docker-compose.yml up -d mysql redis
$ ./container.sh mongo up; echo "exit=$?"
unknown service: mongo
usage: ./container.sh <mysql|redis|kafka> <up|down|logs|status> | ./container.sh workspace
exit=2

I did not run a real docker compose and did not test the Linux branch (xdg-open).

Design choices:

  • Services are a whitelist, so typos are rejected.
  • down uses stop to keep data (down -v would delete volumes).
  • DRY_RUN lets CI check the script without Docker.
  • Exit code 2 signals a usage error.

Remember

  • Explain where a file’s name lives, and how hard and soft links behave when the original is deleted or renamed.
  • Know why rm-ing a log that’s being written doesn’t free the disk, how to find it and how to fix it.
  • Read chmod 750, ls -l and a 022 umask; explain why the right to delete a file lives on the directory.
  • Know how to run a container as non-root and fix volume write errors without chmod 777.
  • List what set -euo pipefail doesn’t catch, and why sed is fragile in a deploy pipeline.

Interview questions

Junior

What is an inode? Where is the file name stored?

An inode is the structure describing a file; the name lives in a directory entry of the containing directory.

The inode holds the inode number, size, owner, permissions, timestamps, link count and pointers to the data blocks. The directory maps name → inode number, so one inode can have several names (hard links).

What does chmod 750 mean, and why are permissions 4, 2, 1?

User rwx (7 = 4 + 2 + 1), group r-x (5 = 4 + 1), others nothing (0).

4, 2 and 1 are the three bits of a three-digit binary number, so every combination sums to a unique value from 0 to 7 that decodes back unambiguously.

Mid

You deleted a huge log file but the disk is still full. Why, and what do you do?

The service still holds an fd to the file, so the inode and its blocks haven’t been freed.

rm only removes a name; data is freed only when the link count is zero and no process has the file open. So du can’t see it but df is still full. Find it with lsof +L1 and reclaim the space by restarting the process or running : > /proc/<pid>/fd/<n>.

What’s the difference between logrotate’s create and copytruncate?

create renames the old file and creates a new one; copytruncate copies the content elsewhere and truncates the original to zero.

With create, the process keeps writing to the old inode until it’s told to reopen. With copytruncate, the process needn’t know, but lines written between the copy and the truncate can be lost, and without O_APPEND you get a zero-filled region.

Who can delete a read-only (444) file?

Anyone with w and x on its directory, because deleting modifies the directory entry, not the file.

Shared directories like /tmp set the sticky bit so only a file’s owner (and root) can delete it.

Why shouldn’t containers run as root, and how do you fix volume write errors?

If the app is exploited the attacker is root in the container; fix volume write errors with fsGroup or an init container, not by going back to root.

Root inside the container can write anywhere in the image and gets closer to the host when the setup is loose. Use USER in the Dockerfile and runAsNonRoot plus readOnlyRootFilesystem in Kubernetes. Volumes keep their on-disk UID/GID, so a non-root user may not be able to write; never “fix” that with chmod 777.

Senior

A deploy script uses sed to replace the image tag. How do you review it?

sed can match the wrong thing, doesn’t understand YAML and stays silent when nothing matches; at minimum anchor and verify, better use yq, kustomize or Helm.

  • Risks: matching other strings ($IMAGE_TAG_SUFFIX), no YAML awareness (several containers, indentation), exit code 0 when nothing was replaced, and sed -i differs between macOS and Linux.
  • At minimum: anchor the pattern and verify with grep -q, failing if nothing was replaced.
  • Better: envsubst with an explicit variable list, yq, or kustomize/Helm; pin images by digest so deploys are reproducible.
Is set -euo pipefail enough to make a script safe?

No. -e ignores failures inside if conditions and &&/||, and local x=$(cmd) swallows cmd’s exit code.

You still need to quote every variable, use ${VAR:?} for required variables before dangerous commands like rm -rf, put -- before file names, run shellcheck in CI, and provide a dry-run mode.

How do you safely replace a binary or config file that’s in use?

Write the new version to a temp file on the same file system, then mv it into place, because rename is atomic.

Readers see either the old or the new version, never a half-written file, and running processes keep the old inode. Overwriting in place can cause Text file busy or mixed content. Deploying into versioned directories and switching a current symlink follows the same principle; rollback is just pointing the symlink back.

Useful phrases for interviews

  • “The name lives in the directory entry; the inode holds the metadata and the block pointers.”
  • “Disk space only comes back when the link count is zero and no process holds the file open.”
  • “lsof +L1 shows files that were deleted but are still held open.”
  • “Deleting a file is a write to its directory, so the directory’s permissions decide.”
  • “We run the container as a non-root user and fix volume ownership with fsGroup, not chmod 777.”
  • “I’d use tail -F rather than tail -f so it survives log rotation.”
  • “For manifests I’d rather use yq or kustomize than sed, because sed silently succeeds when nothing matches.”
  • “The script is idempotent and has a dry-run mode, so it’s safe to run twice or in CI.”

Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.