solaris.pro / Field manuals / DTrace

DTrace cookbook: syscalls and latency

Investigate slow system calls, high syscall rates, failed opens and unexpected process activity.

Scope: Oracle Solaris 11.4 and, for LDoms, compatible SPARC hardware with Oracle VM Server for SPARC. Examples are educational: verify release, privileges, target names and backups before running changes.

How it works

Counts before stacks

Start with low-overhead counts grouped by executable and syscall. Narrow to a process or function before enabling stack traces.

Latency is a distribution

Averages hide outliers. quantize() gives a power-of-two histogram, while avg() gives a compact summary.

Return values and errno

Probe argument semantics differ by provider and function. Verify args[], arg0, errno and relevant syscall names for your release before interpreting errors.

Worked examples

Each example explains the question it answers, a reproducible starting point, what to look for, and operational precautions.

1. Most frequent syscalls by executable

Find a process generating kernel traffic.

syscall:::entry
{
  @calls[execname, probefunc] = count();
}

Interpretation: Run `dtrace -s calls.d` for a short interval; identify high-volume process/function pairs.

Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.

2. Write latency distribution

Find whether writes have a long tail.

syscall::write:entry
{
  self->start = timestamp;
}
syscall::write:return
/self->start/
{
  @ns[execname] = quantize(timestamp - self->start);
  self->start = 0;
}

Interpretation: Histograms are nanoseconds; compare tails across executables and workload windows.

Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.

3. Read latency average

Compare per-process read cost.

syscall::read:entry { self->t = timestamp; }
syscall::read:return /self->t/
{
  @avg_ns[execname] = avg(timestamp - self->t);
  self->t = 0;
}

Interpretation: Averages do not identify which files or disks are responsible.

Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.

4. Count open-family calls

See which programs frequently access file paths.

syscall::open*:entry
{
  @opens[execname] = count();
}

Interpretation: Probe wildcard matches available open-family probes; inspect actual names with `dtrace -l`.

Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.

5. Trace exec activity

Find process creation and command launches.

proc:::exec-success
{
  printf("%Y pid=%d %s\n", walltimestamp, pid, execname);
}

Interpretation: Use `dtrace -l -P proc` to verify probe availability and adjust names if necessary.

Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.

Before you finish

  • High syscall rate is not automatically a problem; compare to useful work.

Primary documentation

Read the relevant Oracle documentation for exact version-specific flags, supported platforms and limitations: Oracle primary reference ↗.

This field manual is independent, not affiliated with or endorsed by Oracle. Command examples should be adapted to your environment.