strace: A Practical Troubleshooting Guide¶

strace intercepts and records the system calls a process makes and the signals it receives. When something is broken and you don't know why — a process hangs, a file "doesn't exist" when it clearly does, a service is slow for no obvious reason — strace tells you what the process actually asked the kernel to do, and what the kernel said back.

1. The Mental Model¶

Every meaningful interaction a program has with the outside world — opening a file, reading a socket, forking a child, allocating memory via mmap, sleeping, locking a mutex via futex — goes through a syscall. Application-level logs tell you what the program thinks happened. strace tells you what the kernel actually did. When those two disagree, strace wins.

Basic anatomy of a traced line:

openat(AT_FDCWD, "/etc/nginx/[ hostname ]", O_RDONLY) = 3

• openat — the syscall
• everything in parens — the arguments, decoded into human-readable form
• = 3 — the return value (here, a file descriptor)

Errors look like this:

openat(AT_FDCWD, "/etc/app/[ hostname ]", O_RDONLY) = -1 ENOENT (No such file or directory)

-1 plus an errno name plus a human description. This single line format is 80% of what you need for "why is this failing" investigations.

2. Getting Started¶

# Trace a command from launch
strace ./myapp

# Attach to a running process
strace -p 12345

# Attach to a process and all its threads
strace -f -p 12345

-f (follow forks) is essential for anything that forks or spawns threads — most real services do. Without it you'll only see the parent and wonder why nothing interesting shows up.

For containerized workloads, -p requires ptrace capability on the target. In Kubernetes this typically means SYS_PTRACE capability and often running the trace from the same PID namespace (e.g., kubectl debug with a node-level debug pod, or nsenter into the container's namespace, since strace isn't usually in minimal images).

3. Cutting the Noise: Filtering¶

Raw strace output is a firehose. Filter by syscall class immediately.

# Only file-related calls
strace -e trace=file ./myapp

# Only network calls
strace -e trace=network ./myapp

# Only process/signal related calls
strace -e trace=process,signal ./myapp

# Specific syscalls only
strace -e trace=open,openat,read,write ./myapp

# Everything except noisy ones
strace -e trace=all -e trace=\!futex,epoll_wait ./myapp

Built-in classes worth knowing: file, network, process, signal, ipc, memory, desc (file descriptor ops).

Filter by return status — extremely useful for "something is failing silently":

strace -Z ./myapp     # only calls that failed (returned error)
strace -z ./myapp     # only calls that succeeded

4. Timing: Finding Where Time Goes¶

This is where strace earns its keep for performance troubleshooting.

# Timestamp on every line (wall clock)
strace -t ./myapp

# Microsecond precision timestamps
strace -tt ./myapp

# Time delta between the *end* of one syscall and the *start* of the next
strace -T ./myapp

# Summary: syscall counts, total time, errors — no per-call output
strace -c ./myapp

# Same as -c but shows the live trace too
strace -C ./myapp

-c output looks like:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 45.23    0.021340         106       201           read
 30.11    0.014200         710        20           connect
 12.04    0.005678          28       201           write

If connect is eating 30% of wall time with 20 calls at 710µs each, you have a DNS resolution or upstream latency problem, not an application bug. This is usually the first thing to run when someone says "the service feels slow" and you don't yet know if it's CPU, I/O, or network bound.

-T (per-call latency) is better than -c when you need to find the one slow call rather than the aggregate — e.g., a single read() that blocked for 4 seconds in the middle of an otherwise fast trace.

5. Common Troubleshooting Patterns¶

"The config file isn't being found" / permission errors¶

strace -e trace=file -f ./myapp 2>&1 | grep -E 'ENOENT|EACCES|EPERM'

Walk the openat calls in order — many programs probe several candidate paths before succeeding or giving up. You'll often find it's reading /etc/app/[ hostname ] from an unexpected working directory, or hitting a symlink it doesn't have permission to traverse, or an SELinux/AppArmor denial that manifests as a plain EACCES (cross-check with dmesg / ausearch -m avc if the underlying cause isn't obvious from strace alone — strace shows the denial, not always the reason).

"The process just hangs"¶

Attach directly and watch the last call:

strace -p <pid> -f -tt

If it stops printing entirely and the process shows as D state in ps, it's blocked in uninterruptible sleep on I/O — strace itself won't show more because the process hasn't returned from the syscall yet. Common culprits: futex (lock contention), read/recvfrom on a socket with no data and no timeout, connect to a host that's blackholing packets (hangs until TCP timeout, ~2 minutes by default), or NFS/network filesystem stalls.

# Check what state the syscall is stuck in
cat /proc/<pid>/stack   # kernel stack, if you have permissions
cat /proc/<pid>/wchan   # what it's waiting on

"Intermittent failures under load"¶

Run with -f against the whole tree and log to a file rather than the terminal — you need to catch the failure when it happens, and terminal scrollback isn't reliable for that:

strace -f -tt -o /tmp/[ hostname ] -p <pid>

Then grep the log for the failing syscall/errno once you reproduce the issue, and look at what happened in the few lines immediately before it across all threads (sorted by timestamp).

Network connection issues¶

strace -f -e trace=network -s 200 ./myapp

-s 200 widens the string-argument truncation (default is 32 chars) so you can actually read full IP:port tuples, HTTP request lines inside sendto, etc. Watch for:
- connect(...) = -1 ECONNREFUSED — nothing listening on the target port
- connect(...) = -1 ETIMEDOUT — firewall/network path issue, packets going nowhere
- connect(...) = -1 EINPROGRESS followed by poll/epoll_wait — normal for non-blocking connects, not itself a problem

File descriptor leaks¶

strace -f -e trace=open,openat,close -o /tmp/[ hostname ] ./myapp
# then compare open() vs close() counts per fd over time
awk '/openat|open\(/{opens++} /close\(/{closes++} END{print opens, closes}' /tmp/[ hostname ]

If opens vastly outpace closes over a long run, you're leaking descriptors — this is how "too many open files" (EMFILE) surfaces days later in production.

6. Reducing Overhead¶

strace slows the traced process down substantially (syscall-heavy workloads can see 2–10x+ slowdown) because every syscall now round-trips through the tracer. For production troubleshooting:

• Filter aggressively (-e trace=...) — tracing everything is rarely necessary and multiplies overhead.
• Prefer short, targeted attach-and-detach windows over long-running traces.
• For a lower-overhead alternative when you just need aggregate syscall latency without full argument decoding, consider perf trace or eBPF-based tools (bpftrace, execsnoop, opensnoop from bcc-tools) — they impose far less overhead and are generally the better choice for latency-sensitive production services. Reach for strace first because it's near-universally available and needs no special kernel tooling; reach for eBPF tools when strace's overhead itself would change the behavior you're trying to observe.
• -p on a live process only needs ptrace permission — check /proc/sys/kernel/yama/ptrace_scope; a value of 1 or higher restricts attaching to non-child processes unless you have CAP_SYS_PTRACE (root, or the capability explicitly granted to the container).

7. Useful Flag Reference¶

8. A Real Workflow¶

1. Reproduce with -c first. Get the aggregate picture — is this CPU-bound (barely any syscalls), I/O-bound (dominated by read/write), or network-bound (dominated by connect/recv/send)?
2. Narrow with -e trace= to the class implicated by step 1.
3. Add -f -tt once you're looking at a specific subsystem, so you can correlate timing across threads.
4. Log to a file with -o if the issue is intermittent, then grep for errno strings or unusually large -T deltas after reproduction.
5. Cross-reference with application logs using the timestamps from -tt — strace tells you what happened at the kernel boundary; your app logs tell you what the code thought it was doing at the same moment. The gap between those two is usually where the bug lives.

9. strace vs. Alternatives¶

• ltrace — traces library calls (libc functions) instead of syscalls. Useful when the problem is in library-level logic rather than kernel interaction, but largely unmaintained upstream and slower; reach for it rarely.
• perf trace — lower overhead, syscall-level like strace, integrates with the rest of the perf toolchain, but less readable argument decoding.
• bpftrace / bcc-tools — eBPF-based, minimal overhead, best for production and for aggregating across many processes at once (e.g., opensnoop across every container on a node) rather than deep-diving a single process's argument values.

Use strace when you need full argument visibility on a specific process and can tolerate the overhead. Use eBPF tooling when overhead matters or you need a system-wide view.