Kernel Debugging with dcons(4)
NOTE: This page has been migrated from the wiki to http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-dcons.html - any future updates should be made there and not here.
- Kernel Debugging with dcons(4)
- How to use dcons over FireWire
- How to use dcons with KVM
- Live core debug
Dcons(4) is a very simple console driver that is not directly connected with any physical devices. It just reads/writes characters from/to a buffer in kernel or loader. Because of its simple nature, it is very useful for kernel debugging especially with FireWire device. Currently, we provide two ways to interact with the buffer from outside of the kernel using dconschat(8).
Dcons over FireWire
Most of host controllers of FireWire(IEEE1394) are based the OHCI specification, that supports physical access to the host memory. This means that once the host controller is initialized, we can access the host memory without help of software(kernel). We can exploit this facility for interaction with dcons. Dcons provides similar functionality as serial console. It emulates two serial ports, one for console and DDB, the other for GDB. Because remote memory access is fully handled by the hardware, the dcons buffer is accessible even when system crashes. Only restriction is that physical access is disabled after bus reset on FireWire, that is, if you have a bus rest during a DDB session you will lose the connection.
You should note that supported FireWire devices are not limited to those built in motherborads. Just buy a cheap (maybe $20-$30) PCI card for a desktop and buy a cardbus interface for a laptop. (I don't think loader(8) can support cardbus though.)
Dcons with KVM
We can directly read dcons buffer via /dev/mem for live systems. And we can read the buffer in core dump for crashed system. They will give you similar output with 'dmesg -a' but the dcons buffer include some more information such as a session of ddb and loader as far as dcons in the loader is enabled.
How to use dcons over FireWire
Enable FireWire/dcons support in kernel
- Make sure your kernel supports dcons/dcons_crom/firewire.
- Dcons should be statically linked with kernel. For dcons_crom and firewire, modules should be o.k.
- Make sure physical DMA is enabled. You may need to add hw.firewire.phydma_enable=1 in /boot/loader.conf
- Add options for debug(KDB and etc).
Add dcons_gdb=1 in /boot/loader.conf if you use gdb over FireWire.
- Enable dcons in /etc/ttys.
(optional) Put hw.firewire.dcons_crom.force_console=1 in loader.conf to force dcons to be the high-level console.
Enable FireWire/dcons support in loader(8) (i386/amd64 only)
- Put LOADER_FIREWIRE_SUPPORT=YES in /etc/make.conf and rebuild loader.
# cd /sys/boot/i386 && make clean && make && make install
Put boot_multicons="YES" in /boot/loader.conf to enable dcons as an active low-level console.
- Kernel configuration file
device dcons device dcons_crom options KDB options DDB options GDB options ALT_BREAK_TO_DEBUGGER
dcons_crom_load="YES" dcons_gdb=1 boot_multicons="YES" hw.firewire.phydma_enable=1 hw.firewire.dcons_crom.force_console=1
To take full advantage of speed of FireWire, disable other slow console drivers.
# conscontrol delete ttyd0 # serial console # conscontrol delete consolectl # video/keyboard
(setq gud-gdba-command-name "kgdb -a -a -a -r :12345") (setq gdb-many-windows t) (xterm-mouse-mode 1) M-x gdba
A hack to reset target committed
- For DDD(devel/ddd),
# remote serail protocoal LANG=C ddd --debugger kgdb -r :12345 kernel # live core debug LANG=C ddd --debugger kgdb kernel /dev/fwmem0.2
Enable FireWire support in kernel
# kldload firewire
Find out EUI64 of the target machine
EUI64 is a unique 64bit identifier of FireWire host controller.
Use fwcontrol(8)or inspect dmesg of the target machine to find out EUI64 of the target machine.
# dconschat -e \# -br -G 12345 -t 00-11-22-33-44-55-66-77
"CR" + "~" + "."
"CR" + "~" + "^B"
"CR" + "~" + "^R"
"CR" + "~" + "^Z"
Attach remote gdb
Start kgdb(1) with remote debugging session.
kgdb -r :12345 kernel
How to use dcons with KVM
- Dump dcons buffer of live system.
# dconschat -1
- Dump dcons buffer of a crash dump.
# dconschat -1 -M vmcore.XX
Live core debug
* Current kvm(3) doesn't support RAW coredump. This patch re-enables RAW coredump support for /dev/fwmem on i386 and amd64. committed
# fwcontrol -m target_eui64 # kgdb kernel /dev/fwmem0.2