HOWTO add SDT probes to DTrace in the kernel

What is DTrace SDT?

SDT stands for Statically Defined Tracing. The SDT provider in DTrace allows a programmer to instrument specific kernel code with his own DTrace probes. As the programmer has more semantic knowledge about kernel code, this allows to add more interesting probes to the kernel, than an automatic code parsing provider (like the fbt provider) can do.

Necessary includes

The following includes are needed if you want to add a SDT probe:

#include "opt_kdtrace.h"
#include <sys/kernel.h>
#include <sys/sdt.h>

Defining a static provider

Then you need to define a provider name for your static probes:

SDT_PROVIDER_DEFINE(foobar);

The provider is displayed when you list all dtrace providers. As of this writting DTrace in FreeBSD offers "syscall", "vfs", "fbt" and some more providers. The above would add a provider with the name "foobar".

This provider needs to be specified only once, if you have other source files which shall use the same provider, you need only to declare the provider, but not define it again:

SDT_PROVIDER_DECLARE(foobar);

Currently, adding SDT probes to KLD modules is only supported on 10-CURRENT as of r233552. It is not supported in 9.0-RELEASE or what will be 8.3-RELEASE. There is nothing in the code that will prevent you from adding probes to a KLD module but it is fairly easy to panic the system if you do.

Defining static probes

Probes need to be defined first (multiple places in the source can fire the same probe, if the same probe shall be fired from within different source files, the probe needs to be defined at one place globally, and declared in all places where it is used).

Defining a probe (you only need to specify arguments, which you want to provide in the probe):

SDT_PROBE_DEFINE3(provider, module, function, probename, probename, "argtype0", "argtype1", "argtype2");

Note that the probename is repeated. This is not a mistake: the first instance is the internal identifier of the probe, which must be a valid C identifier, and the second is the probe name as it is to be used in scripts. This distinction makes it possible to define probes with dashes in their names; typically the internal identifiers use an underscore instead of a dash in this case. For example:

SDT_PROBE_DEFINE0(foo, , func, probe_name, probe-name);

This can be used as "foo::func:probe-name" in DTrace scripts.

The "3" in the macro name indicates that the probe takes 3 arguments, and the "argtype" fields should be replaced with the actual types of the probe arguments, e.g. "int", or "struct tcphdr *".

Probes can take 0 to 7 arguments.

Declaring a probe:

SDT_PROBE_DECLARE(provider, module, function, probename);

Example

Assumption, you have

and you want to add probes for

See further down the page for the functions where those probes are fired to see what each probe is doing.

 #include "opt_kdtrace.h"
 #include <sys/kernel.h>
 #include <sys/sdt.h>

 SDT_PROVIDER_DEFINE(foobar);
 SDT_PROBE_DEFINE2(foobar, source_file1, foo, entry, entry, "int", "char *");
 SDT_PROBE_DEFINE1(foobar, source_file1, foo, return, return, "int");
 SDT_PROBE_DEFINE(foobar, source_file2, bar, entry, entry);
 SDT_PROBE_DEFINE(foobar, source_file2, bar, my_error_condition_name, my-error-condition-name);
 SDT_PROBE_DEFINE(foobar, source_file2, bar, return, return);

Instead of "source_file1" and "source_file2" you can also use different names, e.g. in the same source file you can add different names here. This part of the definition tells about the "module" in which you are. In case you have a source file which contains several logical "modules" (whatever a module is in your opinion) you can use different names for those modules. So far we have no rule what to use here, so if you don't have a better idea what to use in your code, you can use the filename without the .c ending here. When such a probe fires, the developer looking at problems quickly get's an idea where he has to look at things. For subsystems like GEOM, it would make sense to use "geom" as the provider name, the core GEOM functions get "core" as the module name, and specific GEOM providers use their name for the module, e.g. geom:mirror:functionX:entry. This is not set in stone, it's just a suggestion. Maybe it makes even more sense to have a provider in each GEOM provider.

Instrumenting code with probes

Example:

int
foo(int a, const char *b) {
    int c;

    SDT_PROBE(foobar, source_file1, foo, entry, a, b, 0, 0, 0);

    if (a == 3) {
         SDT_PROBE(foobar, source_file1, foo, return, 1, 0, 0, 0, 0);
         return 1;
    }

    c = a*a;
    printf("%s\f", b);

    SDT_PROBE(foobar, source_file1, foo, return, c, 0, 0, 0, 0);
    return (c);
}

void
bar(void) {

    SDT_PROBE(foobar, source_file2, bar, entry, 0, 0, 0, 0, 0);

    ...
    if (trigger_error()) {
        /*
         * Some kind of "milestone", we want to be able
         * to trigger a dtrace action in case this happens.
         */
        SDT_PROBE(foobar, source_file2, bar, my_error_condition_name, 0, 0, 0, 0, 0);
        return;
    }
    ...

    SDT_PROBE(foobar, source_file2, bar, return, 0, 0, 0, 0, 0);
}

Corresponding DTrace script

foobar:source_file2:bar:my_error_condition_name
{
    printf("ERROR: %s (%d) triggered error condition X in %s:%s:%s!\n", execname, pid, probeprov, probemod, probefunc);
}


CategoryDTrace

DTrace/HowToAddSDTProbes (last edited 2013-09-15 17:27:14 by MarkJohnston)