Kernel Debugging with dcons(4)

Contents

Kernel Debugging with dcons(4)
Synopsis
1. Dcons over FireWire
2. Dcons with KVM
How to use dcons over FireWire
1. Target machine
2. Host machine
How to use dcons with KVM
Live core debug
References

Synopsis

Dcons(4) is a very simple console driver that is not directly connected with any physical devices. It just reads/writes characters from/to a buffer in kernel or loader. Because of its simple nature, it is very useful for kernel debugging especially with FireWire device. Currently, we provide two ways to interact with the buffer from outside of the kernel using dconschat(8).

Dcons over FireWire

Most of host controllers of FireWire(IEEE1394) are based the OHCI specification, that supports physical access to the host memory. This means that once the host controller is initialized, we can access the host memory without help of software(kernel). We can exploit this facility for interaction with dcons. Dcons provides similar functionality as serial console. It emulates two serial ports, one for console and DDB, the other for GDB. Because remote memory access is fully handled by the hardware, the dcons buffer is accessible even when system crashes. Only restriction is that physical access is disabled after bus reset on FireWire, that is, if you have a bus rest during a DDB session you will lose the connection.

You should note that supported FireWire devices are not limited to those built in motherborads. Just buy a cheap (maybe $20-$30) PCI card for a desktop and buy a cardbus interface for a laptop. (I don't think loader(8) can support cardbus though.)

Dcons with KVM

We can directly read dcons buffer via /dev/mem for live systems. And we can read the buffer in core dump for crashed system. They will give you similar output with 'dmesg -a' but the dcons buffer include some more information such as a session of ddb and loader as far as dcons in the loader is enabled.

How to use dcons over FireWire

Target machine

Enable FireWire/dcons support in kernel

Make sure your kernel supports dcons/dcons_crom/firewire.
Dcons should be statically linked with kernel. For dcons_crom and firewire, modules should be o.k.
Make sure physical DMA is enabled. You may need to add hw.firewire.phydma_enable=1 in /boot/loader.conf
Add options for debug(KDB and etc).
Add dcons_gdb=1 in /boot/loader.conf if you use gdb over FireWire.
Enable dcons in /etc/ttys.
(optional) Put hw.firewire.dcons_crom.force_console=1 in loader.conf to force dcons to be the high-level console.

Enable FireWire/dcons support in loader(8) (i386/amd64 only)

Put LOADER_FIREWIRE_SUPPORT=YES in /etc/make.conf and rebuild loader.

  # cd /sys/boot/i386 && make clean && make && make install

Put boot_multicons="YES" in /boot/loader.conf to enable dcons as an active low-level console.

Configuration example

Kernel configuration file

   device dcons
   options KDB
   options DDB
   options GDB
   options ALT_BREAK_TO_DEBUGGER

/boot/loader.conf

   dcons_crom_load="YES"
   dcons_gdb=1
   boot_multicons="YES"
   hw.firewire.phydma_enable=1
   hw.firewire.dcons_crom.force_console=1

Tips

To take full advantage of speed of FireWire, disable other slow console drivers.

  # conscontrol delete ttyd0             # serial console
  # conscontrol delete consolectl        # video/keyboard

emacs gdb mode (emacs22, gud) patch screenshot

  (setq gud-gdba-command-name "kgdb -a -a -a -r :12345")
  (setq gdb-many-windows t)
  (xterm-mouse-mode 1)
  M-x gdba

A hack to reset target committed
For DDD(devel/ddd),

   # remote serail protocoal
   LANG=C ddd --debugger kgdb -r :12345 kernel
   # live core debug
   LANG=C ddd --debugger kgdb kernel /dev/fwmem0.2

Host machine

Enable FireWire support in kernel

  # kldload firewire

Find out EUI64 of the target machine

EUI64 is a unique 64bit identifier of FireWire host controller.
Use fwcontrol(8)or inspect dmesg of the target machine to find out EUI64 of the target machine.

Run dconschat(8)

  # dconschat -e \# -br -G 12345 -t 00-11-22-33-44-55-66-77

"CR" + "~" + "."	Disconnect
"CR" + "~" + "^B"	ALT BREAK
"CR" + "~" + "^R"	RESET target
"CR" + "~" + "^Z"	Suspend dconschat

Attach remote gdb

Start kgdb(1) with remote debugging session.

   kgdb -r :12345 kernel

How to use dcons with KVM

Dump dcons buffer of live system.

  # dconschat -1

Dump dcons buffer of a crash dump.

  # dconschat -1 -M vmcore.XX

Live core debug

* Current kvm(3) doesn't support RAW coredump. This patch re-enables RAW coredump support for /dev/fwmem on i386 and amd64. committed

    # fwcontrol -m target_eui64
    # kgdb kernel /dev/fwmem0.2