blktrace records what the kernel's block layer does with each I/O request: when it was queued, merged, dispatched, and completed. Use it when iostat shows a problem and you need to know which I/O pattern causes it.

sudo dnf -y install blktrace        # or apt-get install blktrace

# Live trace of a device, decoded on the fly
sudo blktrace -d /dev/sda -o - | blkparse -i -

# Record for 30 seconds to files, analyse later
sudo blktrace -d /dev/sda -w 30 -o trace
blkparse -i trace | less
blkparse -i trace -d trace.bin && btt -i trace.bin      # latency breakdown

Event letters in blkparse output include Q (queued), G (request allocated), M (merged), D (dispatched to the driver), and C (completed). The gap between D and C is device time. The gap between Q and D is time in the kernel queue.

Alternative using perf: sudo perf record -e block:block_rq_issue -ag records the same tracepoints with call graphs, showing which code path issues the I/O.