Project name

Student: ShengYiHong (<aokblast AT FreeBSD DOT org>)
Mentor: LiWenHsu (<lwhsu AT FreeBSD DOT org>), EdMaste (<emaste AT FreeBSD DOT org>)

Project description

FreeBSD has replaced the GNU toolchain with the LLVM toolchain for a long time. In the LLVM toolchain, there is a bundle debugger called lldb, which is also bundled in FreeBSD. The lldb debug facilities in userland have been well implemented for a long time. However, there are some works still need to be done in the kernel space. What I want to do is to finish the parse mechanism about kernel module information for the kernel debugger

Approach to solving the problem

In kgdb’s scheme, it does the following things:

Setup the related structure offset in ./kern/linker.c and find the linkerfiles structure that is a linkedlist containing all of the loaded LKM information.
Search the LKM from the raw and absolute path to find the corresponding kernel module and get the table offset from bfd
Relocating sections in ko files found in step 2 with LKM base address found in step 1
Load symbol from symbol file and thus finish

In LLDB’s userspace implementation, it read the kinfovmentry to get the loaded module information, and thus modules can be found in the GetLoadedModuleFileSpce method. What we have to do is to combine LLDB userspace and kgdb in LLDB kernel space. Except for the LKM, I also want to support static kernel module which is also a possible module type in the FreeBSD kernel. But things are different in that the static kernel module will not appear in kldstat. In other words, it will not appear in linkerfiles, so we need a special handler on these files

Deliverables

1. LKM parse and LLDB FreeBSD Kernel Module implementation

Report

May 29th - June 12th

Busy on my school final exam. The development environment have been set up in these weeks.

June 12th - June 26th

In this week, I implement LLDB interface sketch for DynamicLoader FreeBSD Kernel (DynamicLoaderFreeBSDKernel) that contains parsing kernel loader address and designinbg kld memory Image structure that contain enough information to create lldb Module. Also, the lldb Module information for Kernel is also parsed so that can provide lldb recognize the Kernel.

June 26th - July 10th

Parse kld info and Finish all implementation about DynamicLoaderFreeBSDKernel. The kld info is in a linked list structure defined at linked_files symbol that is a TAILQ structure. The necessary member's offset in the structure is provided by the kernel to provide the compatibility in different implementation. Merge the source code from llvm-project to FreeBSD codebase

June 14th - July 2th

Found the unloaded debug caused by merging LLDB plugin from llvm codebase to freebsd codebase. Finally the plugin successfully loaded and can load plugin kernel address in the binary. However the kernel module part still needs debug more.

July 3th - July 17th

Finish all implementation and finally the module can shown and debug symbol can be loaded

Test Plan

Because performance is not that important for a debugger, I will focus the test on functionality. In the first stage of the test, I will try to write some small kernel modules with some function symbols contained for debug pur- poses. In the second stage of the test, I will try to use the large kernel module like wifi to test the functionality. In the second stage of the test, I will post my work on reviews.freebsd.org to let other module developers test it.

The Code

https://github.com/aokblast/freebsd-src/tree/lldb_dynamicloader_freebsd_kernel_review https://github.com/llvm/llvm-project/pull/67106

Final Production

As I have finished implementing DynamicLoaderFreeBSDKernel class, we have the following feature enabled on LLDB when debugging FreeBSDKernel Image:

1. Display all kernel module and it's load address

(lldb) image list
[  0] 013C9080-98CC-F2F1-237C-AFAA727809F7-8144DCF4 0xffffffff80200000 /boot/kernel/kernel
      /usr/lib/debug/boot/kernel/kernel.debug
[  1] F67FE954-8379-A3D1-848C-946C8C239092-6EAD3444 0xffffffff81f50000 /boot/kernel/zfs.ko
      /usr/lib/debug/boot/kernel/zfs.ko.debug
[  2] C580C510-DF33-FB7D-81AC-7989646C889C-26A50D5F 0xffffffff82644000 /boot/kernel/cryptodev.ko
      /usr/lib/debug/boot/kernel/cryptodev.ko.debug
[  3] 34D8ADD5-836D-7AB7-F62E-0C35987F19D0-9659DE07 0xffffffff82b18000 /boot/kernel/intpm.ko
      /usr/lib/debug/boot/kernel/intpm.ko.debug
[  4] B90304EB-A9FE-5495-9B77-E92790CF6EAF-FC35AA21 0xffffffff82b1c000 /boot/kernel/smbus.ko
      /usr/lib/debug/boot/kernel/smbus.ko.debug
[  5] 1AC7F0BF-CE67-8AD0-1A14-2E4444489F0E-7742EFF9 0xffffffff82b1f000 /boot/kernel/uhid.ko
      /usr/lib/debug/boot/kernel/uhid.ko.debug
[  6] CF312DEA-1D10-5C8B-D139-153111326704-24679FD8 0xffffffff82b23000 /boot/kernel/usbhid.ko
      /usr/lib/debug/boot/kernel/usbhid.ko.debug
[  7] B2A238B6-767A-D503-ADAC-8C828C613FF4-0D1D1D86 0xffffffff82b27000 /boot/kernel/hidbus.ko
      /usr/lib/debug/boot/kernel/hidbus.ko.debug
[  8] 9AF49EC4-8C0D-D9CE-5372-749EFF8A2F50-394D4184 0xffffffff82b2b000 /boot/kernel/wmt.ko
      /usr/lib/debug/boot/kernel/wmt.ko.debug

2. Display content of a Symbol

(lldb) p cryptodev_mod
(moduledata_t) $0 = {
  name = 0xffffffff8264718f "cryptodev"
  evhand = 0xffffffff82644000 (cryptodev.ko`cryptodev_modevent at cryptodev.c:1281)
  priv = 0x0000000000000000

Of course it's address:

(lldb) p &cryptodev_mod
(moduledata_t *) $3 = 0xffffffff826482b0

3. Some information dump from module dump

(lldb) image dump symtab cryptodev.ko
Symtab, file = /boot/kernel/cryptodev.ko, num_symbols = 120:
               Debug symbol
               |Synthetic symbol
               ||Externally Visible
               |||
Index   UserID DSX Type            File Address/Value Load Address       Size               Flags      Name
------- ------ --- --------------- ------------------ ------------------ ------------------ ---------- ----------------------------------
[    0]      1     SourceFile      0x0000000000000000                    0x0000000000000000 0x00000004 cryptodev.c
[    1]      2     Code            0x0000000000000000 0xffffffff82644000 0x000000000000006e 0x00000002 cryptodev_modevent
[    2]      3     Data            0x00000000000042c8 0xffffffff826482c8 0x00000000000000b8 0x00000001 crypto_cdevsw
[    3]      4     Data            0x0000000000004400 0xffffffff82648400 0x0000000000000008 0x00000001 crypto_dev
[    4]      5     Code            0x0000000000000070 0xffffffff82644070 0x000000000000008d 0x00000002 crypto_open
[    5]      6     Data            0x0000000000004230 0xffffffff82648230 0x0000000000000070 0x00000001 M_CRYPTODEV
[    6]      7     Code            0x0000000000002240 0xffffffff82646240 0x00000000000000de 0x00000002 fcrypt_dtor
[    7]      8     Code            0x0000000000000100 0xffffffff82644100 0x0000000000002132 0x00000002 crypto_ioctl
[    8]      9     Data            0x00000000000043f8 0xffffffff826483f8 0x0000000000000001 0x00000001 use_outputbuffers
[    9]     10     Data            0x00000000000043f9 0xffffffff826483f9 0x0000000000000001 0x00000001 use_separate_aad
[   10]     11     Code            0x00000000000023b0 0xffffffff826463b0 0x00000000000000d2 0x00000002 cod_alloc
[   11]     12     Code            0x0000000000002320 0xffffffff82646320 0x0000000000000089 0x00000002 cse_free
[   12]     13     Code            0x0000000000002490 0xffffffff82646490 0x0000000000000056 0x00000002 cryptodev_cb

SummerOfCode2023Projects/LLDBKernelModuleImprovement (last edited 2024-02-08T03:44:35+0000 by ShengYiHong)