illumos Dynamic Tracing Guide

https://illumos.org/books/dtrace
Admin
DTrace is a comprehensive dynamic tracing framework for the illumos Operating System, allowing...
Tokens:116,683
Snippets:508
Trust Score:9.5
Update:11 months ago
Show doc for...
Tokens:
Raw
### DTrace Script Execution Example

Source: https://illumos.org/books/dtrace/chp-intro

This is an example of how to execute a DTrace script from the command line. It shows the command to run a script named 'rw.d' and the expected output indicating matched probes.

```bash
# dtrace -s rw.d
dtrace: script 'rw.d' matched 2 probes
CPU     ID                    FUNCTION:NAME
	0     34                      write:entry
	0     32                       read:entry
	0     34                      write:entry
	0     32                       read:entry
	0     34                      write:entry
	0     32                       read:entry
	0     34                      write:entry
	0     32                       read:entry
...
```

--------------------------------

### DTrace Countdown Execution Example

Source: https://illumos.org/books/dtrace/chp-intro

Example output demonstrating the execution of the DTrace countdown timer script. It shows the sequence of numbers traced during the countdown and the final 'blastoff!' message.

```shell
# dtrace -s countdown.d
dtrace: script 'countdown.d' matched 3 probes
CPU     ID                    FUNCTION:NAME
	0  25499                       :tick-1sec        10
	0  25499                       :tick-1sec         9
	0  25499                       :tick-1sec         8
	0  25499                       :tick-1sec         7
	0  25499                       :tick-1sec         6
	0  25499                       :tick-1sec         5
	0  25499                       :tick-1sec         4
	0  25499                       :tick-1sec         3
	0  25499                       :tick-1sec         2
	0  25499                       :tick-1sec         1
	0  25499                       :tick-1sec   blastoff!
```

--------------------------------

### pid Provider Probe Naming Examples

Source: https://illumos.org/books/dtrace/chp-pid

These examples demonstrate various ways to name probes for the pid provider, including using full library names, common suffixes, and the 'a.out' alias for executables. The output shows how DTrace resolves these to a canonical form.

```dtrace
pid123:libc.so.1:strcpy:entry
pid123:libc.so:strcpy:entry
pid123:libc:strcpy:entry

pid123:csh:main:return
pid123:a.out:main:return
```

--------------------------------

### Basic truss(1) command example

Source: https://illumos.org/books/dtrace/chp-intro

This command demonstrates the basic usage of the `truss` utility to trace all system calls executed by the `date` command and displays its output.

```shell
`$ truss date`
```

--------------------------------

### DTrace Manual Buffer Resizing Failure Example

Source: https://illumos.org/books/dtrace/chp-buf

Illustrates the `manual` buffer resize policy, where DTrace fails to start if it cannot allocate the requested buffer size. This requires manual intervention to resolve memory issues.

```shell
# dtrace -P syscall -x bufsize=1g -x bufresize=manual
dtrace: description 'syscall' matched 430 probes
dtrace: could not enable tracing: Not enough space
#

```

--------------------------------

### DTrace trace Action Examples

Source: https://illumos.org/books/dtrace/chp-actsub

Examples of the basic DTrace `trace` action, which takes a D expression as an argument and traces its result to the directed buffer.

```dtrace
trace(execname);
tracemem(curlwpsinfo->pr_pri);
tracemem(timestamp / 1000);
tracemem(`lbolt);
tracemem("somehow managed to get here");
```

--------------------------------

### DTrace Macro Argument Usage Examples

Source: https://illumos.org/books/dtrace/chp-script

Demonstrates various ways macro arguments can be used within D programs. These examples show how to compare with strings, add to integers, and use as identifiers, assuming appropriate macro argument values are provided.

```d
execname == $1    /* with a string macro argument */
x += $1           /* with an integer macro argument */
trace(x->$1)      /* with an identifier macro argument */
```

--------------------------------

### DTrace Aggregation Example with printa Formatting

Source: https://illumos.org/books/dtrace/chp-fmt

Shows how to aggregate data using a tuple and then use printa with a format string to display specific elements of the aggregated data along with the aggregation result.

```d
@a["hello", 123] = count();
@a["goodbye", 456] = count();

printa(`_format-string_`, @a);
```

--------------------------------

### DTrace Execution Example and Output

Source: https://illumos.org/books/dtrace/chp-proc

This section shows how to execute the DTrace script and provides a sample of the output generated. The output displays a distribution of program runtimes, categorized by the executable name, using DTrace's 'quantize' function.

```shell
`# dtrace -s ./progtime.d`
dtrace: script './progtime.d' matched 2 probes
`^C`

  ir2hf
           value  ------------- Distribution -------------
         4194304 |
         8388608 |
        16777216 |@@@@@@@@@@@@@@@@                         14
        33554432 |@@@@@@@@@@                               9
        67108864 |@@@                                      3
       134217728 |@                                        1
       268435456 |@@@@                                     4
       536870912 |@                                        1
      1073741824 |

  ube
           value  ------------- Distribution -------------
        16777216 |
        33554432 |@@@@@@@                                  6
        67108864 |@@@                                      3
       134217728 |@@                                       2
       268435456 |@@@@                                     4
       536870912 |@@@@@@@@@@@@                             10
      1073741824 |@@@@@@@                                  6
      2147483648 |@@                                       2
      4294967296 |

  acomp
           value  ------------- Distribution -------------
         8388608 |
        16777216 |@@                                       2
        33554432 |
        67108864 |@                                        1
       134217728 |@@@                                      3
       268435456 |
       536870912 |@@@@@                                    5
      1073741824 |@@@@@@@@@@@@@@@@@@@@@@@@@                22
      2147483648 |@                                        1
      4294967296 |

  cc
           value  ------------- Distribution -------------
        33554432 |
        67108864 |@@@                                      3
       134217728 |@                                        1
       268435456 |
       536870912 |@@@@                                     4
      1073741824 |@@@@@@@@@@@@@@                           13
      2147483648 |@@@@@@@@@@@@                             11
      4294967296 |@@@                                      3
      8589934592 |

  sh
           value  ------------- Distribution -------------
          262144 |
          524288 |@                                        5
         1048576 |@@@@@@@                                  29
         2097152 |
         4194304 |
         8388608 |@@@                                      12
        16777216 |@@                                       9
        33554432 |@@                                       9
        67108864 |@@                                       8
       134217728 |@                                        7
       268435456 |@@@@@                                    20
       536870912 |@@@@@@                                   26
      1073741824 |@@@                                      14
      2147483648 |@@                                       11
      4294967296 |
      8589934592 |
     17179869184 |

  make.bin
           value  ------------- Distribution -------------
        16777216 |
        33554432 |@                                        1
        67108864 |@                                        1
       134217728 |@@                                       2
       268435456 |
       536870912 |@@                                       2
      1073741824 |@@@@@@@@@                                9
      2147483648 |@@@@@@@@@@@@@@@                          14
      4294967296 |@@@@@@                                   6
      8589934592 |@@                                       2
     17179869184 |
```

--------------------------------

### Display sys Kernel Statistics using kstat

Source: https://illumos.org/books/dtrace/chp-sysinfo

This example demonstrates how to use the kstat command to display both the names and current values of the sys named kernel statistic. It requires the kstat utility to be available.

```shell
`$ kstat -n sys`
module: cpu                             instance: 0
name:   sys                             class:    misc
	bawrite                         123
	bread                           2899
	bwrite                          17995
...
```

--------------------------------

### Example DTrace iotime.d Output

Source: https://illumos.org/books/dtrace/chp-io

This is an example of the output generated by the 'iotime.d' DTrace script when a USB storage device is hot-plugged. It shows detailed I/O operations, including the device (cmdk0), the associated file or kernel module, the operation type (R for read, W for write), and the time taken in milliseconds. Observations from this output can reveal performance characteristics, driver loading, and system events.

```text
    DEVICE                                                 FILE RW      MS
     cmdk0                                 /kernel/drv/scsa2usb  R  24.781
     cmdk0                                 /kernel/drv/scsa2usb  R  25.208
     cmdk0                                    /var/adm/messages  W  25.981
     cmdk0                                 /kernel/drv/scsa2usb  R   5.448
     cmdk0                                               <none>  W   4.172
     cmdk0                                 /kernel/drv/scsa2usb  R   2.620
     cmdk0                                    /var/adm/messages  W   0.252
     cmdk0                                            <unknown>  R   3.213
     cmdk0                                               <none>  W   3.011
     cmdk0                                            <unknown>  R   2.197
     cmdk0                                    /var/adm/messages  W   2.680
     cmdk0                                               <none>  W   0.436
     cmdk0                                    /var/adm/messages  W   0.542
     cmdk0                                               <none>  W   0.339
     cmdk0                                    /var/adm/messages  W   0.414
     cmdk0                                               <none>  W   0.344
     cmdk0                                    /var/adm/messages  W   0.361
     cmdk0                                               <none>  W   0.315
     cmdk0                                    /var/adm/messages  W   0.421
     cmdk0                                               <none>  W   0.349
     cmdk0                                               <none>  R   1.524
     cmdk0                                            <unknown>  R   3.648
     cmdk0                                 /usr/lib/librcm.so.1  R   2.553
     cmdk0                                 /usr/lib/librcm.so.1  R   1.332
     cmdk0                                 /usr/lib/librcm.so.1  R   0.222
     cmdk0                                 /usr/lib/librcm.so.1  R   0.228
     cmdk0                                 /usr/lib/librcm.so.1  R   0.927
     cmdk0                                               <none>  R   1.189
       ...
     cmdk0                            /usr/lib/devfsadm/linkmod  R   1.110
     cmdk0         /usr/lib/devfsadm/linkmod/SUNW_audio_link.so  R   1.763
     cmdk0         /usr/lib/devfsadm/linkmod/SUNW_audio_link.so  R   0.161
     cmdk0           /usr/lib/devfsadm/linkmod/SUNW_cfg_link.so  R   0.819
     cmdk0           /usr/lib/devfsadm/linkmod/SUNW_cfg_link.so  R   0.168
     cmdk0          /usr/lib/devfsadm/linkmod/SUNW_disk_link.so  R   0.886
     cmdk0          /usr/lib/devfsadm/linkmod/SUNW_disk_link.so  R   0.185
     cmdk0        /usr/lib/devfsadm/linkmod/SUNW_fssnap_link.so  R   0.778
     cmdk0        /usr/lib/devfsadm/linkmod/SUNW_fssnap_link.so  R   0.166
     cmdk0          /usr/lib/devfsadm/linkmod/SUNW_lofi_link.so  R   1.634
     cmdk0          /usr/lib/devfsadm/linkmod/SUNW_lofi_link.so  R   0.163
     cmdk0            /usr/lib/devfsadm/linkmod/SUNW_md_link.so  R   0.477
     cmdk0            /usr/lib/devfsadm/linkmod/SUNW_md_link.so  R   0.161
     cmdk0          /usr/lib/devfsadm/linkmod/SUNW_misc_link.so  R   0.198
     cmdk0          /usr/lib/devfsadm/linkmod/SUNW_misc_link.so  R   0.168
     cmdk0          /usr/lib/devfsadm/linkmod/SUNW_misc_link.so  R   0.247
     cmdk0     /usr/lib/devfsadm/linkmod/SUNW_misc_link_i386.so  R   1.735
       ...
```

--------------------------------

### DTrace print() with other providers (fbt::zio_done:entry)

Source: https://illumos.org/books/dtrace/chp-fmt

An example of using the print() action with the fbt provider to print the first argument of the 'zio_done:entry' function. It demonstrates how DTrace can inspect kernel data structures.

```dtrace
# dtrace -n 'fbt::zio_done:entry{ print(*args[0]); }'
dtrace: description 'fbt::zio_done:entry' matched 1 probe
CPU     ID                    FUNCTION:NAME
 10  53680                   zio_done:entry zio_t {
    zbookmark_t io_bookmark = {
        uint64_t zb_objset = 0
        uint64_t zb_object = 0
        int64_t zb_level = 0
        uint64_t zb_blkid = 0x14
    }
    zio_prop_t io_prop = {
        enum zio_checksum zp_checksum = ZIO_CHECKSUM_INHERIT
        enum zio_compress zp_compress = ZIO_COMPRESS_INHERIT
        dmu_object_type_t zp_type = DMU_OT_NONE
        uint8_t zp_level = 0
        uint8_t zp_copies = 0
        uint8_t zp_dedup = 0
        uint8_t zp_dedup_verify = 0
    }
    zio_type_t io_type = ZIO_TYPE_NULL
    enum zio_child io_child_type = ZIO_CHILD_VDEV
    int io_cmd = 0
    uint8_t io_priority = 0x4
    uint8_t io_reexecute = 0
    uint8_t [2] io_state = [ 0x1, 0 ]
    uint64_t io_txg = 0x8dfb3a
    spa_t *io_spa = 0xffffff0d37f3a500
    blkptr_t *io_bp = 0xffffff0d39581a00
    blkptr_t *io_bp_override = 0
    blkptr_t io_bp_copy = {
        dva_t [3] blk_dva = [
            dva_t {
                uint64_t [2] dva_word = [ 0x5, 0x394c7972 ]
            },
...

```

--------------------------------

### DTrace Default Action Example

Source: https://illumos.org/books/dtrace/chp-actsub

Demonstrates enabling all probes in the 'TS' timeshare scheduling module using the default action, which traces the enabled probe identifier (EPID) to the principal buffer.

```dtrace
# dtrace -m TS
```

--------------------------------

### DTrace Execution: Error Path Tracing Example

Source: https://illumos.org/books/dtrace/chp-user

This example shows how to execute a DTrace script named `errorpath.d` with arguments to trace a specific function (e.g., `_chdir`) in a process. The output demonstrates the probes hit, including instruction offsets.

```shell
# ./errorpath.d 100461 _chdir
dtrace: script './errorpath.d' matched 19 probes
CPU     ID                    FUNCTION:NAME
  0  25253                     _chdir:entry 81e08 6d140 ffbfcb20 656c73 0
  0  25253                     _chdir:entry
  0  25269                         _chdir:0
  0  25270                         _chdir:4
  0  25271                         _chdir:8
  0  25272                         _chdir:c
  0  25273                        _chdir:10
  0  25274                        _chdir:14
  0  25275                        _chdir:18
  0  25276                        _chdir:1c
  0  25277                        _chdir:20
  0  25278                        _chdir:24
  0  25279                        _chdir:28
  0  25280                        _chdir:2c
  0  25268                    _chdir:return
```

--------------------------------

### DTrace `ustack` Example for Brk System Call

Source: https://illumos.org/books/dtrace/chp-actsub

This D program demonstrates the use of the `ustack` action to count stack frames for the `brk` system call, filtering by executable name. It shows how to collect and display stack traces, including hexadecimal frames when symbol translation fails. The example command line and output illustrate its usage with the Netscape browser.

```dtrace
syscall::brk:entry
/execname == $$1/
{
	@[ustack(40)] = count();
}
```

```shell
# dtrace -s brk.d .netscape.bin
dtrace: description 'syscall::brk:entry' matched 1 probe
`^C`
               libc.so.1`_brk_unlocked+0xc
               88143f6
               88146cd
               .netscape.bin`unlocked_malloc+0x3e
               .netscape.bin`unlocked_calloc+0x22
               .netscape.bin`calloc+0x26
               .netscape.bin`_IMGCB_NewPixmap+0x149
               .netscape.bin`il_size+0x2f7
               .netscape.bin`il_jpeg_write+0xde
               8440c19
               .netscape.bin`il_first_write+0x16b
               8394670
               83928e5
               .netscape.bin`NET_ProcessHTTP+0xa6
               .netscape.bin`NET_ProcessNet+0x49a
               827b323
               libXt.so.4`XtAppProcessEvent+0x38f
               .netscape.bin`fe_EventLoop+0x190
               .netscape.bin`main+0x1875
                  1

               libc.so.1`_brk_unlocked+0xc
               libc.so.1`sbrk+0x29
               88143df
               88146cd
               .netscape.bin`unlocked_malloc+0x3e
               .netscape.bin`unlocked_calloc+0x22
               .netscape.bin`calloc+0x26
               .netscape.bin`_IMGCB_NewPixmap+0x149
               .netscape.bin`il_size+0x2f7
               .netscape.bin`il_jpeg_write+0xde
               8440c19
               .netscape.bin`il_first_write+0x16b
               8394670
               83928e5
               .netscape.bin`NET_ProcessHTTP+0xa6
               .netscape.bin`NET_ProcessNet+0x49a
               827b323
               libXt.so.4`XtAppProcessEvent+0x38f
               .netscape.bin`fe_EventLoop+0x190
               .netscape.bin`main+0x1875
                  1
    ...
```

--------------------------------

### DTrace printa Aggregation Formatting Example

Source: https://illumos.org/books/dtrace/chp-fmt

This D program demonstrates the use of the printa function to format aggregation results. It samples the caller function and then prints the count of calls using a custom format string. This requires the 'profile' provider.

```d
profile:::profile-997
{
	@a[caller] = count();
}

END
{
	printa("%@8u %a\n", @a);
}
```

--------------------------------

### DTrace Command Execution Example

Source: https://illumos.org/books/dtrace/chp-structs

This command demonstrates how to execute a DTrace script ('kstat.d') against a running process identified by its PID, obtained using `pgrep mpstat`. The `-q` flag suppresses DTrace's default output.

```bash
# dtrace -q -s kstat.d `pgrep mpstat`
```

--------------------------------

### DTrace Default Buffer Ordering Example

Source: https://illumos.org/books/dtrace/chp-buf

Shows the default buffer ordering policy in DTrace, where output is ordered first by CPU and then by the order within each CPU's buffer. This can result in timestamps not being strictly sequential.

```shell
syscall:::entry
{
	trace(timestamp);
}
```

```text
CPU     ID                    FUNCTION:NAME
 23     24                      close:entry  3302220933052713
 23     24                      close:entry  3302220933064286
 23     24                      close:entry  3302220933066326
 23     16                      rexit:entry  3302220933111500
  1     20                      write:entry  3302220705802875
  1     20                      write:entry  3302220705807694
  1     20                      write:entry  3302220705812112
  1    106                      ioctl:entry  3302220705815463
```

--------------------------------

### DTrace BEGIN and END Probes for Measuring Trace Duration

Source: https://illumos.org/books/dtrace/chp-dtrace

This example shows how to use the BEGIN and END probes in DTrace to measure the total time spent tracing. The BEGIN probe records the start timestamp, and the END probe calculates and prints the duration.

```dtrace
BEGIN
{
	start = timestamp;
}

/*
 * ... other tracing actions...
 */

END
{
	printf("total time: %d secs", (timestamp - start) / 1000000000);
}
```

--------------------------------

### DTrace: Get Offset of Struct/Union Member

Source: https://illumos.org/books/dtrace/chp-structs

The `offsetof` operator in DTrace calculates the byte offset of a specific member within a struct or union, relative to the start of the object. It returns a `size_t` value and can be used as an integer constant in D programs. This is helpful for memory address calculations and data structure analysis.

```d
size_t offset = offsetof (struct s, member_name);
```

--------------------------------

### DTrace Script with BEGIN Probe

Source: https://illumos.org/books/dtrace/chp-intro

A simple DTrace script that uses the BEGIN probe to execute code at the start of a tracing session. It prints 'hello, world' and then exits. This demonstrates the usage of the BEGIN probe, which is part of the DTrace framework itself.

```dtrace
dtrace:::BEGIN
{
	trace("hello, world");
	exit(0);
}
```

--------------------------------

### DTrace Script for Program Start and Exit Timing

Source: https://illumos.org/books/dtrace/chp-proc

This DTrace script utilizes the 'proc:::start' and 'proc:::exit' probes to record the execution time of programs. It stores the start time in a per-thread variable and quantifies the duration upon program exit, aggregating results by executable name. No external libraries are required.

```dtrace
proc:::start
{
	self->start = timestamp;
}

proc:::exit
/self->start/
{
	@[execname] = quantize(timestamp - self->start);
	self->start = 0;
}
```

--------------------------------

### DTrace 'Hello, World' Program

Source: https://illumos.org/books/dtrace/chp-intro

This D program defines actions for the BEGIN probe to print 'hello, world' and then exit. It demonstrates basic DTrace syntax, including probe names, actions within braces, function calls like trace() and exit(), and statement termination with semicolons.

```d
BEGIN
{
	trace("hello, world");
	exit(0);
}
```

--------------------------------

### D Trace Translate Operator Example

Source: https://illumos.org/books/dtrace/chp-xlate

Provides a concrete example of invoking a FILE struct translator and accessing its 'file_fd' member using the 'xlate' operator. Assumes 'f' is a D variable of type FILE *.

```d
xlate <struct file_info *>(f)->file_fd;
```

--------------------------------

### Basic DTrace Script Example

Source: https://illumos.org/books/dtrace/chp-script

A simple DTrace script that prints 'hello' upon execution and then exits. This script demonstrates the 'BEGIN' probe, which triggers before any other probes, and the 'trace' and 'exit' actions. The '#!' directive at the beginning makes it executable.

```dtrace
#!/usr/sbin/dtrace -s
BEGIN
{
	trace("hello");
	exit(0);
}
```

--------------------------------

### DTrace Translator Example for FILE to file_info

Source: https://illumos.org/books/dtrace/chp-xlate

An example of a DTrace translator that maps a C FILE pointer to a D structure 'struct file_info'. It demonstrates how to access members of the internal 'struct file_impl' to populate the target structure.

```dtrace
translator struct file_info < FILE *F > {
	file_fd = ((struct file_impl *)F)->fd;
	file_eof = ((struct file_impl *)F)->eof;
};
```

--------------------------------

### Enable BEGIN Probe with dtrace Utility

Source: https://illumos.org/books/dtrace/chp-intro

This command enables the `BEGIN` probe using the `dtrace` utility. The `BEGIN` probe fires once when a tracing request starts. The `-n` option specifies the probe name. Output includes probe description and the probe firing event.

```bash
# dtrace -n BEGIN
dtrace: description 'BEGIN' matched 1 probe
CPU     ID		      FUNCTION:NAME
  0      1                  :BEGIN
`^C`
#
```

--------------------------------

### Explore Available DTrace Probes

Source: https://illumos.org/books/dtrace/chp-vers

Lists the available DTrace providers and probes on the current illumos system. This command-line option is essential for understanding what tracing capabilities are accessible.

```bash
$ dtrace -l
```

--------------------------------

### DTrace lwpsinfo_t Translator Example

Source: https://illumos.org/books/dtrace/chp-xlate

An example of an inline D variable declaration using the `xlate` operator to translate from a private `curthread` type to a stable `lwpsinfo_t` type. It also demonstrates the use of a pragma to explicitly set stability attributes.

```d
inline lwpsinfo_t *curlwpsinfo = xlate <lwpsinfo_t *> (curthread);
#pragma D attributes Stable/Stable/Common curlwpsinfo
```

--------------------------------

### Measure Thread Execution Time with DTrace

Source: https://illumos.org/books/dtrace/chp-proc

This D script utilizes `proc:::lwp-start` and `proc:::lwp-exit` probes to record the start and end times of Lightweight Processes (LWPs), effectively measuring individual thread execution durations. It filters out the initial kernel thread (tid 1) and aggregates the results using `quantize` based on the process's executable name. The `self->start` variable stores the timestamp of thread start, and it's reset after calculating the duration.

```dtrace
proc:::lwp-start
/tid != 1/
{
	self->start = timestamp;
}

proc:::lwp-exit
/self->start/
{
	@[execname] = quantize(timestamp - self->start);
	self->start = 0;
}
```

--------------------------------

### Iostat Extended Device Statistics Output

Source: https://illumos.org/books/dtrace/chp-io

This is an example of extended device statistics provided by the 'iostat' command. It offers a summary of I/O performance metrics for various devices, including read/write rates (r/s, w/s), kilobytes per second (kr/s, kw/s), wait times, active times, service times, and utilization percentages (%w, %b). This output is essential for identifying I/O bottlenecks and understanding overall disk and network throughput.

```text
extended device statistics
device       r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b
cmdk0        8.0    0.0  399.8    0.0  0.0  0.0    0.8   0   1
sd0          0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
sd2          0.0  109.0    0.0  435.9  0.0  1.0    8.9   0  97
nfs1         0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
nfs2         0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
```

--------------------------------

### Extract Substring (DTrace)

Source: https://illumos.org/books/dtrace/chp-actsub

The `substr` function extracts a portion of a string. The first form returns a new string starting from `_start_` to the end. The second form returns a substring of at most `_length_` characters starting from `_start_`. Both handle boundary conditions gracefully.

```d
string substr(char *`_str_`, int `_start_`)
```

```d
string substr(char *`_str_`, int `_start_`, int `_length_`)
```

--------------------------------

### Listing All Available DTrace Probes

Source: https://illumos.org/books/dtrace/chp-intro

This command lists all available probes on the system, organized by ID, provider, module, function, and probe name. It's useful for understanding the instrumentation available for tracing. The output includes built-in probes like BEGIN, END, and ERROR, as well as probes from other providers like lockstat.

```bash
# dtrace -l
  ID   PROVIDER            MODULE          FUNCTION NAME
   1     dtrace                                     BEGIN
   2     dtrace                                     END
   3     dtrace                                     ERROR
   4   lockstat           genunix       mutex_enter adaptive-acquire
   5   lockstat           genunix       mutex_enter adaptive-block
   6   lockstat           genunix       mutex_enter adaptive-spin
   7   lockstat           genunix       mutex_exit  adaptive-release

   ... many lines of output omitted ...
```

--------------------------------

### DTrace Inline Directive Syntax and Example

Source: https://illumos.org/books/dtrace/chp-types

Demonstrates the syntax for defining named constants using DTrace's `inline` directive and provides an example of tracing string and integer values defined as inlines. The `inline` directive allows for type-safe lexical replacement.

```d
inline string hello = "hello";
inline int number = 100 + 23;

BEGIN
{
	trace(hello);
	trace(number);
}
```

--------------------------------

### DTrace Destructive Actions Warning Example

Source: https://illumos.org/books/dtrace/chp-actsub

This example shows the error message generated by `dtrace` when a destructive action is attempted without explicit enabling using the `-w` option. Destructive actions modify system state and require specific privileges or explicit user consent.

```shell
dtrace: failed to enable 'syscall': destructive actions not allowed
```

--------------------------------

### Enable BEGIN and END Probes with dtrace Utility

Source: https://illumos.org/books/dtrace/chp-intro

This command enables both the `BEGIN` and `END` probes using the `dtrace` utility. The `BEGIN` probe fires at the start of tracing, and the `END` probe fires upon completion. The `-n` option is used for each probe. The output shows when each probe is matched and fires.

```bash
# dtrace -n BEGIN -n END
dtrace: description 'BEGIN' matched 1 probe
dtrace: description 'END' matched 1 probe
CPU     ID		      FUNCTION:NAME
  0      1                  :BEGIN
`^C`
  0      2                    :END

```

--------------------------------

### Build Application with DTrace Probes

Source: https://illumos.org/books/dtrace/chp-usdt

Illustrates modifying a standard C build process to include DTrace probe definitions. The `dtrace` command is used with the `-G` option to post-process object files, linking probe definitions before the final application binary is created.

```makefile
cc -c src1.c
cc -c src2.c
...
dtrace -G -32 -s myserv.d src1.o src2.o ...
cc -o myserv myserv.o src1.o src2.o ...
```

--------------------------------

### DTrace Failed Process Grab Warning Example

Source: https://illumos.org/books/dtrace/chp-actsub

This example demonstrates a warning message from `dtrace` indicating a failure to grab a process, likely due to the process exiting before symbol translation could occur. This results in raw hexadecimal stack frames being displayed instead of translated symbols.

```shell
dtrace: failed to grab process 100941: no such process
               c7b834d4
               c7bca85d
               c7bca1a4
               c7bd4374
               c7bc2628
               8047efc
```

--------------------------------

### Example Output of DTrace Speculative open() Script

Source: https://illumos.org/books/dtrace/chp-spec

This is an example of the output generated when the DTrace script for speculative open() failure tracing is executed. It shows the hierarchical trace of system calls involved in an `open()` operation, with arrows indicating function entry and return. The output format is typical for DTrace's `-F` option.

```text
`# ./specopen.d`
dtrace: script './specopen.d' matched 24282 probes
CPU FUNCTION
  1  => open                                  /var/ld/ld.config
  1    -> open
  1      -> copen
  1        -> falloc
  1          -> ufalloc
  1            -> fd_find
  1              -> mutex_owned
  1              <- mutex_owned
  1            <- fd_find
  1            -> fd_reserve
  1              -> mutex_owned
  1              <- mutex_owned
  1              -> mutex_owned
  1              <- mutex_owned
  1            <- fd_reserve
  1          <- ufalloc
  1          -> kmem_cache_alloc
  1            -> kmem_cache_alloc_debug
  1              -> verify_and_copy_pattern
  1              <- verify_and_copy_pattern
  1              -> file_cache_constructor
  1                -> mutex_init
  1                <- mutex_init
  1              <- file_cache_constructor
  1              -> tsc_gethrtime
  1              <- tsc_gethrtime
  1              -> getpcstack
  1              <- getpcstack
  1              -> kmem_log_enter
  1              <- kmem_log_enter
  1            <- kmem_cache_alloc_debug
  1          <- kmem_cache_alloc
  1          -> crhold
  1          <- crhold
  1        <- falloc
  1        -> vn_openat
  1          -> lookupnameat
  1            -> copyinstr
  1            <- copyinstr
  1            -> lookuppnat
  1              -> lookuppnvp
  1                -> pn_fixslash
  1                <- pn_fixslash
  1                -> pn_getcomponent
  1                <- pn_getcomponent
  1                -> ufs_lookup
  1                  -> dnlc_lookup
  1                    -> bcmp
  1                    <- bcmp
  1                  <- dnlc_lookup
  1                  -> ufs_iaccess
  1                    -> crgetuid
  1                    <- crgetuid
  1                    -> groupmember
  1                      -> supgroupmember
  1                      <- supgroupmember
  1                    <- groupmember
  1                  <- ufs_iaccess
  1                <- ufs_lookup
  1                -> vn_rele
  1                <- vn_rele
  1                -> pn_getcomponent
  1                <- pn_getcomponent
  1                -> ufs_lookup
  1                  -> dnlc_lookup
  1                    -> bcmp
  1                    <- bcmp
  1                  <- dnlc_lookup
  1                  -> ufs_iaccess
  1                    -> crgetuid
  1                    <- crgetuid
  1                  <- ufs_iaccess
  1                <- ufs_lookup
  1                -> vn_rele
  1                <- vn_rele
  1                -> pn_getcomponent
  1                <- pn_getcomponent
  1                -> ufs_lookup
  1                  -> dnlc_lookup
  1                    -> bcmp
  1                    <- bcmp
  1                  <- dnlc_lookup
  1                  -> ufs_iaccess
  1                    -> crgetuid
  1                    <- crgetuid
  1                  <- ufs_iaccess
  1                  -> vn_rele
  1                  <- vn_rele
  1                <- ufs_lookup
  1                -> vn_rele
  1                <- vn_rele
  1              <- lookuppnvp
  1            <- lookuppnat
  1          <- lookupnameat

```

--------------------------------

### DTrace: Example printf format string and arguments

Source: https://illumos.org/books/dtrace/chp-intro

This code snippet demonstrates the usage of the DTrace printf function to format and print system call information. It showcases how to combine literal strings with format specifiers (%s, %d, %x) to represent probe function names and integer arguments.

```dtrace
printf("%s(%d, 0x%x, %4d)", probefunc, arg0, arg1, arg2);
```

--------------------------------

### DTrace 'switch' Policy Drop Message Example

Source: https://illumos.org/books/dtrace/chp-buf

This example shows the message format displayed by dtrace(8) when data drops occur under the 'switch' buffer policy due to insufficient space. It indicates the number of drops and the specific CPU where they happened. To mitigate drops, users can increase buffer size ('bufsize') or switching rate ('switchrate').

```dtrace
dtrace: 11 drops on CPU 0
```

--------------------------------

### Define DTrace Probes with Arguments

Source: https://illumos.org/books/dtrace/chp-usdt

This example demonstrates defining two probes, 'query__receive' and 'query__respond', within the 'myserv' provider. 'query__receive' accepts two string arguments, while 'query__respond' takes none. DTrace automatically converts double underscores to hyphens in probe names.

```dtrace
provider myserv {
	probe query__receive(string, string);
	probe query__respond();
};
```

--------------------------------

### DTrace Command-Line Probe Listing with Patterns

Source: https://illumos.org/books/dtrace/chp-prog

Shows how to use DTrace's command-line interface with the '-l' option to list probes that match a given pattern. This example specifically lists probes in functions whose names begin with 'kmem_'.

```bash
dtrace -l -f kmem_*
```

--------------------------------

### Get DTrace Version

Source: https://illumos.org/books/dtrace/chp-vers

Retrieves the current D programming interface version supported by the DTrace compiler. This is useful for understanding compatibility with existing D scripts.

```bash
$ dtrace -V
dtrace: Sun D 1.0
$
```

--------------------------------

### Get Minor Device Number (DTrace)

Source: https://illumos.org/books/dtrace/chp-actsub

The `getminor` subroutine extracts the minor number from a device number (`dev_t`). This function is a D analogue for the DDI `getminor(9F)`.

```d
int getminor(dev_t `_dev_`)
```
Tokens:
Raw