• WaitChannels

What is a Wait Channel?

Threads in the kernel can wait for resources using various APIs such as tsleep() or cv_wait(). The resource being waited on is associated with a pointer in the kernel's virtual address space. These pointers are known as wait channels. In addition to the raw pointer itself, each thread stores a reference to a short string describing the wait channel. This string is known as a wait message.

Wait Channels of Active Threads

There are various ways of determining which wait channel a thread is waiting on. The raw pointer can be displayed via the "nwchan" keyword for ps(1):

> ps O nwchan
  PID NWCHAN           TT  STAT    TIME COMMAND
 6515 -                v0  Z    0:00.19 <defunct>
 6517 fffff8000deffa18 v0  IW   0:00.00 -tcsh (tcsh)
 6545 fffff8000dbea970 v0  IW+  0:00.00 /bin/sh /usr/local/bin/startx
...

The wait message is available via the "wchan" keyword:

> ps O wchan
  PID WCHAN  TT  STAT    TIME COMMAND
 6515 -      v0  Z    0:00.19 <defunct>
 6517 pause  v0  IW   0:00.00 -tcsh (tcsh)
 6545 wait   v0  IW+  0:00.00 /bin/sh /usr/local/bin/startx
...

The ps(1) command also includes another keyword called "mwchan". This keyword displays the wait message for a thread blocked on a wait channel. For a thread blocked on a mutex, read-mostly lock or read/write lock, the name of the lock is displayed after a "*" prefix. Note that other locks such as lockmanager and shared/exclusive locks are implemented using wait channels, so threads blocked on one of those locks display the lock name as the wait message without a "*" prefix.

In addition, the STATE column in top(1) displays the same data as the "mwchan" keyword for threads blocked on either a wait channel or a lock. Wait messages generally use lower-case letters in contrast to the non-waiting status messages displayed in this column such as CPU x, RUN, or WAIT.

  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   11 root       155 ki31     0K    64K CPU3    3 275.1H  92.09% idle{idle: cpu
   11 root       155 ki31     0K    64K RUN     2 273.6H  89.45% idle{idle: cpu
   11 root       155 ki31     0K    64K RUN     0 273.6H  87.89% idle{idle: cpu
   11 root       155 ki31     0K    64K RUN     1 273.6H  86.77% idle{idle: cpu
 6692 john        39    0   445M   119M CPU1    1 267:50  22.07% akonadi_imap_r
 6637 john        33    0   217M 73660K CPU2    2 158:27  15.19% akonadiserver{
 6638 john        27    0   242M   153M sbwait  0 204:18   7.86% mysqld{mysqld}
 6692 john        24    0   445M   119M select  0   3:22   4.49% akonadi_imap_r
 6559 john        21    0  3330M   101M select  3 127:07   0.88% Xorg
 6622 john        20    0   613M 93880K select  3  85:02   0.29% kwin{kwin}
 6629 john        20    0   849M   154M select  3  49:13   0.10% kdeinit4{kdein
 6638 john        20    0   242M   153M sbwait  2 219:54   0.00% mysqld{mysqld}
 7546 john        20    0   505M 86916K select  3  48:41   0.00% kdeinit4{kdein
   12 root       -96    -     0K   320K WAIT    3  42:18   0.00% intr{irq264: v
 1765 root        20    0 28876K  4180K select  1  37:53   0.00% hald-addon-mou
    8 root        16    -     0K    16K syncer  3  32:32   0.00% syncer
 6692 john        20    0   445M   119M select  3  17:17   0.00% akonadi_imap_r

When a process is sent SIGINFO, the kernel outputs the wait channel of the "most interesting" thread in the process on the terminal. Shells send SIGINFO to the running process when Ctrl+T is pressed:

> cat
load: 0.07  cmd: cat 44115 [ttyin] 0.67r 0.00u 0.00s 0% 1532k

The procstat(1) utility can also be used to query the wait channel of processes either via its default output or via the -t flag:

> procstat -t `pgrep firefox`
  PID    TID COMM             TDNAME           CPU  PRI STATE   WCHAN    
16864 100646 firefox          -                  1  120 sleep   select    
16864 100723 firefox          Gecko_IOThread     0  124 sleep   kqread    
16864 100724 firefox          Socket Thread      1  120 sleep   select    
16864 100725 firefox          JS GC Helper       0  120 sleep   uwait     
16864 100726 firefox          JS Watchdog        0  120 sleep   uwait     
16864 100727 firefox          Hang Monitor       3  122 sleep   uwait     
16864 100728 firefox          Analysis Helper    2  120 sleep   uwait     
...

In the in-kernel debugger, DDB, wait channels for threads are displayed via the ps, show proc, and show thread commands.

In kgdb(1), the wait message of a given thread can be found by the string pointed to by the td_wmesg field of the associated thread structure. The wait channel pointer can be found via the td_wchan field.

Finding Wait Channels

Wait channels are associated with wait messages in the kernel source code. To determine the meaning of an unfamiliar wait channel, use a tool like grep(1) to search for the wait message string in the kernel sources. Once you have found the relevant string, examine the nearby code to determine what the thread is waiting for. For example, the "biord" message is used here:

> grep -4Hr biord /sys
/sys/kern/vfs_bio.c-int
/sys/kern/vfs_bio.c-bufwait(struct buf *bp)
/sys/kern/vfs_bio.c-{
/sys/kern/vfs_bio.c-    if (bp->b_iocmd == BIO_READ)
/sys/kern/vfs_bio.c:            bwait(bp, PRIBIO, "biord");
/sys/kern/vfs_bio.c-    else
/sys/kern/vfs_bio.c-            bwait(bp, PRIBIO, "biowr");
/sys/kern/vfs_bio.c-    if (bp->b_flags & B_EINTR) {
/sys/kern/vfs_bio.c-            bp->b_flags &= ~B_EINTR;

Common Wait Channels

This is a table of some of the common wait channel messages one might see in a system and a brief description of what state the associated thread is in (for example, which resource a thread is waiting on). Note that this list is not exhaustive.