DTrace SDT Kernel Probes


SDT stands for Statically Defined Tracing. The SDT provider in DTrace allows a programmer to instrument specific kernel code with his own DTrace probes. As the programmer has more semantic knowledge about kernel code, this allows to add more interesting probes to the kernel, than an automatic code parsing provider (like the fbt provider) can do.

Adding DTrace SDT Probes to the FreeBSD Kernel

Necessary includes

The following headers are needed if you want to add a SDT probe:

#include <sys/param.h>
#include <sys/queue.h>
#include <sys/sdt.h>

On stable/10 or earlier, you will also need:

#include "opt_kdtrace.h"

Defining a static provider

Then you need to define a provider name for your static probes:


The provider is displayed when you list all dtrace providers. As of this writting DTrace in FreeBSD offers "syscall", "vfs", "fbt" and some more providers. The above would add a provider with the name "foobar".

This provider needs to be specified only once, if you have other source files which shall use the same provider, you need only to declare the provider, but not define it again:


Currently, adding SDT probes to KLD modules is only supported on 10-CURRENT as of r233552. It is not supported in 9.0-RELEASE or what will be 8.3-RELEASE. There is nothing in the code that will prevent you from adding probes to a KLD module but it is fairly easy to panic the system if you do.

Defining static probes

Probes need to be defined first (multiple places in the source can fire the same probe, if the same probe shall be fired from within different source files, the probe needs to be defined at one place globally, and declared in all places where it is used).

Defining a probe (you only need to specify arguments, which you want to provide in the probe):

SDT_PROBE_DEFINE3(provider, module, function, probename, "argtype0", "argtype1", "argtype2");

Two consecutive underscores in the probe name are automatically converted to a dash. For example:

SDT_PROBE_DEFINE0(foo, , func, probe__name);

will create a probe called "foo::func:probe-name".

The "3" in the macro name indicates that the probe takes 3 arguments, and the "argtype" fields should be replaced with the actual types of the probe arguments, e.g. "int", or "struct tcphdr *".

Probes can take 0 to 7 arguments.

Declaring a probe:

SDT_PROBE_DECLARE(provider, module, function, probename);


Assumption, you have

and you want to add probes for

See further down the page for the functions where those probes are fired to see what each probe is doing.

 #include <sys/param.h>
 #include <sys/queue.h>
 #include <sys/sdt.h>

 SDT_PROBE_DEFINE2(foobar, source_file1, foo, entry, entry, "int", "char *");
 SDT_PROBE_DEFINE1(foobar, source_file1, foo, return, return, "int");
 SDT_PROBE_DEFINE(foobar, source_file2, bar, entry, entry);
 SDT_PROBE_DEFINE(foobar, source_file2, bar, my_error_condition_name, my-error-condition-name);
 SDT_PROBE_DEFINE(foobar, source_file2, bar, return, return);

Instead of "source_file1" and "source_file2" you can also use different names, e.g. in the same source file you can add different names here. This part of the definition tells about the "module" in which you are. In case you have a source file which contains several logical "modules" (whatever a module is in your opinion) you can use different names for those modules. So far we have no rule what to use here, so if you don't have a better idea what to use in your code, you can use the filename without the .c ending here. When such a probe fires, the developer looking at problems quickly get's an idea where he has to look at things. For subsystems like GEOM, it would make sense to use "geom" as the provider name, the core GEOM functions get "core" as the module name, and specific GEOM providers use their name for the module, e.g. geom:mirror:functionX:entry. This is not set in stone, it's just a suggestion. Maybe it makes even more sense to have a provider in each GEOM provider.

Instrumenting code with probes


foo(int a, const char *b) {
    int c;

    SDT_PROBE2(foobar, source_file1, foo, entry, a, b);

    if (a == 3) {
         SDT_PROBE1(foobar, source_file1, foo, return, 1);
         return 1;

    c = a*a;
    printf("%s\f", b);

    SDT_PROBE1(foobar, source_file1, foo, return, c);
    return (c);

bar(void) {

    SDT_PROBE0(foobar, source_file2, bar, entry);

    if (trigger_error()) {
         * Some kind of "milestone", we want to be able
         * to trigger a dtrace action in case this happens.
        SDT_PROBE0(foobar, source_file2, bar, my_error_condition_name);

    SDT_PROBE0(foobar, source_file2, bar, return);

Corresponding DTrace script

    printf("ERROR: %s (%d) triggered error condition X in %s:%s:%s!\n", execname, pid, probeprov, probemod, probefunc);

CategoryDtrace CategoryHowTo

DTrace/HowToAddSDTProbes (last edited 2022-09-19T02:07:11+0000 by KubilayKocak)