How it works
Investigation 1: service CPU spike
Baseline with prstat, sample on-CPU executables, narrow to one PID, then collect user stacks. Compare the stack distribution before and after a change.
Investigation 2: slow application writes
Check zpool iostat, then measure write syscall latency distributions. If writes are slow but disks are not busy, examine locks, filesystem sync behavior and application semantics.
Investigation 3: intermittent network delays
Validate routes and interface errors, then histogram connect syscall durations for the affected executable. Correlate timestamps with network and DNS telemetry.
Investigation 4: restart loop
Inspect svcs -xv and logs, count proc:::exec-success for the service executable, and look for frequent short-lived processes.
Investigation 5: unexplained system calls
Aggregate syscall:::entry by process and function. Focus on outliers rather than dumping every event.
Investigation 6: one guest slow, others healthy
Collect prstat, iostat and DTrace inside the guest, then correlate with service-domain zpool iostat and LDom bindings.
Worked examples
Each example explains the question it answers, a reproducible starting point, what to look for, and operational precautions.
1. Scenario: CPU spike
First capture top consumers, then trace.
prstat 1
# Save the following as cpu.d:
profile-97 { @on_cpu[execname] = count(); }
# dtrace -s cpu.dInterpretation: If one executable dominates, switch to per-PID `ustack()` sampling.
Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.
2. Scenario: high write latency
Compare syscall and physical device evidence.
zpool iostat -v 2
# Save as write-lat.d:
syscall::write:entry { self->t = timestamp; }
syscall::write:return /self->t/ { @ns[execname] = quantize(timestamp-self->t); self->t=0; }Interpretation: If tail latency rises without device load, investigate synchronization or application locking.
Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.
3. Scenario: connect stalls
Find which process sees delays.
syscall::connect:entry { self->t=timestamp; }
syscall::connect:return /self->t/ { @ns[execname]=quantize(timestamp-self->t); self->t=0; }Interpretation: Correlate with `netstat -s`, DNS response times and server logs.
Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.
4. Scenario: repeated process launch
Confirm a restart loop.
svcs -xv
proc:::exec-success { @launches[execname] = count(); }Interpretation: Compare counts with SMF restart counters and service logs.
Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.
5. Scenario: noisy system calls
Locate one high-rate source.
syscall:::entry { @syscalls[execname,probefunc] = count(); }Interpretation: Filter the top executable and syscall in the next run.
Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.
6. Scenario: guest-only I/O stall
Compare guest and service domain.
ldm list-bindings app01
# In guest: iostat -xn 2
# In service domain: zpool iostat -v 2Interpretation: Determine whether a virtual disk backend or guest workload is responsible.
Operational note: Run commands in a test environment first; confirm device names, privileges and release-specific options.
Before you finish
- Record workload, sampling window, and before/after conditions.
Primary documentation
Read the relevant Oracle documentation for exact version-specific flags, supported platforms and limitations: Oracle primary reference ↗.
This field manual is independent, not affiliated with or endorsed by Oracle. Command examples should be adapted to your environment.