DTrace cookbook: filesystem and disk I/O

Scope: Oracle Solaris 11.4 and, for LDoms, compatible SPARC hardware with Oracle VM Server for SPARC. Examples are educational: verify release, privileges, target names and backups before running changes.

How it works

Separate filesystem from disk

An application write can be buffered, compressed or cached; syscall duration is not identical to physical disk latency.

Choose the right layer

Start with iostat and zpool iostat for device pressure, then DTrace when you need per-process attribution or deeper kernel evidence.

Probe availability

The io provider and ZFS-related probes depend on the running system; always list actual probes first.

Worked examples

Each example explains the question it answers, a reproducible starting point, what to look for, and operational precautions.

1. List I/O probes

Confirm the tracing surface.

dtrace -l -P io
dtrace -l | grep -i zfs | head -40

Interpretation: Discover exact names instead of assuming a script from another release works unchanged.

Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.

2. Count I/O starts

Spot heavy block-I/O sources.

io:::start
{
  @starts[execname] = count();
}

Interpretation: Counts requests started in the I/O provider; attribution may reflect issuing context.

Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.

3. Kernel stack for I/O submission

Understand the kernel path to storage.

io:::start
{
  @[stack()] = count();
}

Interpretation: Collect for a brief interval; frequent stacks may reveal filesystem or device paths.

Operational note: Stack capture has more overhead than simple counters.

4. Correlate with pool metrics

Validate physical symptoms alongside trace data.

iostat -xn 2
zpool iostat -v 2
zpool status -v

Interpretation: Compare queue/latency signals with trace counts, and watch for device faults.

Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.

Before you finish

Do not equate a high `io:::start` count with slow disks.

Primary documentation

Read the relevant Oracle documentation for exact version-specific flags, supported platforms and limitations: Oracle primary reference ↗.

This field manual is independent, not affiliated with or endorsed by Oracle. Command examples should be adapted to your environment.

DTrace foundations: probes, clauses and safety →DTrace cookbook: syscalls and latency →DTrace cookbook: CPU profiling and hot stacks →DTrace cookbook: network diagnosis →