Automated Kernel Crash Reporting System
Project description
This project aims to develop an information system for automated kernel crash reports for the FreeBSD Operating System. When a FreeBSD system has a crash and responsible is a kernel panic, a crash reporter program will collect some necessary data from the crashed system and deliver them automatically in the form of a report to a Central Collector machine. The Central Collector machine will receive these reports, check them for validity, store them in a database, recognize which crash reports refer to previously reported bugs, and provide a dynamic website for the presentation of the collected data with different features for the community and the developers of FreeBSD.
Approach to solving the problem
The approach of solving the problem is the division of the project in two parts. The first part is related with any FreeBSD system in the world that may crash (Client Side part). The second part is related with a central machine that collects the crash reports from the crashed systems (Server Side part).
Client Side part
In this part, the main goals are to locate, collect, form, and transfer the debugging information from the crashed FreeBSD system to the Central Collector machine which is responsible for storing the crash reports.
The introduction of three new variables in the /etc/rc.conf file is necessary. These variables give the freedom to the System Administrator to configure the behavior of the crash reporting system. These three variables are called crashreport_enable, crashreport_program and crashreport_email and their default values are specified in the /etc/defaults/rc.conf file.
- Variable name: crashreport_enable
- Possible values: "YES", "NO"
- Default value: "NO"
Description: Enable or disable the automated reporting of kernel crashes
- Variable name: crashreport_program
- Possible values: "/absolute/path/to/program"
- Default value: "/usr/sbin/crashreport"
- Description: The path of the crashreport program
- Variable name: crashreport_email
Possible values: "sysadmin@freebsd.org", " "
- Default value: " "
- Description: The email of the System Administrator. It will be used only for contact by the Central Collector machine.
In order to collect the debugging information from the crashed FreeBSD system and send them to the Central Collector machine, we have to create the crash reporter program. From now on, we will call it crashreport. The crashreport program is a (Bourne) Shell script. Its default location is in /usr/sbin/crashreport. It can be invoked either automatically or manually.
Automated behavior
The automated behavior refers to the automatic reporting of the kernel crash during the first multi-user boot after the crash. In every boot, the /etc/rc.d/savecore shell script executes the /sbin/savecore program that checks for a crash dump in the specified dumpdev device. If /sbin/savecore finds a crash dump, it extracts the crash dump to the dumpdir directory. If the type of the crash dump was a full memory dump or a minidump, the dumpdir directory will be filled from two files. The vmcore.X and info.X. If the type of the crash dump was a textdump, the dumpdir directory will be filled from a tarred file called textdump.tar.X and from a file called info.X. The tarred file textdump.tar.X contains a bunch of .txt files (up to 5) that provide valuable debugging information. Subsequently, /etc/rc.d/savecore checks the rc variable crashinfo_enable. If this variable is set to "YES", then /etc/rc.d/savecore executes the /usr/sbin/crashinfo shell script that analyzes the extracted core dump (so it works only with full memory dumps and minidumps) and generates a text file in the dumpdir directory. This file is called core.txt.X and contains valuable debugging information. Likewise, /etc/rc.d/savecore checks the rc variable crashreport_enable. If this variable is set to "YES", then /etc/rc.d/savecore executes the /usr/sbin/crashreport. The crashreport program will search by default the /var/crash directory for a file called core.txt.X or textdump.tar.X. If both files exist, it will select the one with the higher X number. From the selected file, it will extract the needed information and use them to form the crash report. The crash report is a XML file with tags that the Central Collector machine can recognize. Finally, it will send the report to the Central Collector machine using the SSH protocol. After the execution of the usr/sbin/crashreport the booting process continues normally.
Manual behavior
The manual behavior refers to the invocation of the /usr/sbin/crashreport explicitly from the System Administrator. This behavior is useful in the case that the System Administrator did not have enabled the variable crashreport_enable for various reasons when the system had a crash and now he/she wants to report the kernel crash. Only the superuser will be able to execute successfully the crashreport program.
Anatomy of crashreport
The /usr/sbin/crashreport is a Bourne Shell script that provides three options. The options are as follows:
-d dumpdir
Specify an alternate core dump directory that contains the debugging information. The default dump directory is /var/crash.
-f filename
Specify the name of the file that contains the debugging information. Normally, it is either core.txt.X or textdump.tar.X
-e email
Specify the email address of the System Administrator that it will be used for contact from the Central Collector machine. Normally, the email address is specified in the variable crashreport_email of the /etc/rc.conf file.
If none of the above options is specified, crashreport is capable of finding the right information on his own. That is, it can be invoked from the superuser without any options, and it will locate, collect and send the report without any further user interaction. It is important to mention that only the Superuser (root - uid 0) will be able to execute crashreport successfully. One technical reason for that is that the generated files core.txt.X and textdump.tar.X have read and write access only for root. Also, it is better only the System Administrator of the machine to be able to report a kernel crash and not every individual user that may have access to the machine.
In the first version of crashreport (known as kcrashreporter), we wanted to instruct crashreport to generate the needed debugging information on his own. This is not an issue anymore because the script /usr/sbin/crashinfo does this job pretty well. Also as we mentioned above, it is invoked from the rc.d savecore script when savecore finds and extracts successfully a new core dump. So if the crash dump type is a full memory dump or a minidump, crashinfo will generate a core.txt.X file that will contain valuable debugging information. In the case of a textdump, the usr/sbin/crashinfo script does not take place but the file textdump.tar.X contains the debugging information that we need.
An essential functionality of crashreport is the method that sends the report to the Central Collector machine. It uses the SSH protocol to transfer securely the report from the client to the server. Specifically, it uses the scp tool of OpenSSH with public key authentication. The configuration options can be divided in 2 parts, one for the client and one for the server. For the server machine we did the following:
Register a DynDNS domain (akcrs.dyndns.org) that points to the (dynamic) IP address of the server machine (currently my Desktop computer). This domain will be used from the scp tool as the hostname field and as the domain name for the website (akcrs.dyndns.org).
Create a new user (named reporter). This user will be used from the scp tool to transfer the report and store it in the directory /var/spool/crashreports that he owns.
As the user reporter, we generate with ssh-keygen a pair of keys using the RSA algorithm. This is the user pair of keys. The public key is placed in the file authorized_keys of his .ssh directory (~/.ssh). The private key is distributed with crashreport.
We check the public key of the /etc/ssh/ssh_host_ecdsa_key.pub and distribute it with crashreport in order to register the fingerprint as known hosts in the Client side.
For the client, everything about the method for sending the report is integrated inside the crashreport script. That is, the host public key, the user private key and the scp command.
For a better understanding of the anatomy of crashreport, view the source code here.
Server Side part
In this part, the main goals are to receive the crash reports, check them for validity, recognize the reports that refer to the same issue as previously logged reports, store them in a database, and provide one dynamic website with different features for the community and the developers.
The above goals define the functions of the Central Collector machine. The functions of the Central Collector machine can be divided in two parts:
- Organization of data
- Presentation of data
The Organization of data part is divided in the following phases:
- Receive
- Check
- Recognize
- Store
The Presentation of data is done via the dynamic website.
The crashreportd program is responsible to execute the organization of data, and the website the presentation of data.
Database
About the database that will store the received crash reports and any other data that will be used in the server side part, we will use a database management system and specifically the ORDBMS PostgreSQL. We choose Postgres because it is released under the PostgreSQL license, a license similar to the BSD license, it is a free and open source software, and it has high scores in various benchmarks. The schema of the database is the following:
Users(id, email, password, is_developer) Bugs(id, state, reported) Reports(id, bug_id, user_id, received_datetime, confirmation_code, confirmed, bugs_id, crashdata)
There are three tables, the table Reports, the table Bugs and the table Users. The table Reports stores the data related with the crash reports that arrive in the system, while the table Bugs exists to indicate the problems that had been reported. For example, if we have in Reports more than one report that is similar with other, that means that both reports refers to the same problem. For this reason, we assign to them the same bug_id and we store in the Bugs table only one tuple that indicates the problem. As a rule, one report refers to exactly one bug, while one bug may be referred from more than one reports. Also, the table Bugs should always have equal or less tuples than Reports. The table Users stores the email address along with a generated password and a unique ID for every submitter that send a report, either automatically or manually.
The primary key of Reports is the column named id and the columns bug_id and users_id are foreign keys that references the columns id of Bugs and id of Users respectively. The column received_datetime stores the date and time that the report was received. The column confirmation_code stores the code that is sent to the submitter in order to confirm his/her report. The column confirmed indicates whether this report has been confirmed or not. The column bugs_id is used from the recognize phase when crashreportd determine that the report refers to more than one logged bugs (more details in the Recognize phase). The column crashdata is not a single column in the actual schema, but it is here to represent the columns related with the data that has been received from the panicked system that sent the report. For example, some columns are panic which keep the panic message, backtrace which keep the backtrace, ps -axl which keep the output of the command ps with the switches a, x and l.
The primary key of Bugs is the column named id. The column state defines the current state of the bug. This column can take only one of the following values:
Open: This is the initial state of a bug. This state declares that there is at least one report that refers to it and the bug has not been reviewed from a FreeBSD developer yet.
Analyzed: This state declares that the bug is being reviewed from a FreeBSD developer but no solution exists yet. In this state, a FreeBSD developer can ask the submitter for further debugging information.
Feedback: This state declares that the bug has been solved and the submitter has been given a patch or other fix. The bug report remains in this state until the submitter acknowledges that the solution works.
Closed: This state declares that the bug has been solved and the submitter has confirmed the solution.
Suspended: This state declares that the bug has been analyzed from a developer and marked as impossible to be fixed any time soon for various reasons.
The bug states that described above have many similarities with the states of Problem Reports in the GNATS bug tracking system that is currently used from the FreeBSD project.
The column reported is an integer that shows the importance of a bug based on the number of reports that arrive to the system and point to a particular bug. For example, if we have ten different reports that refer to the same bug, then the reported column of this bug will have the value 10. The presence of the column reported is for performance reasons, for easier manipulation of the data and logical design. Instead of counting the entries of a specific bug_id in the Reports table every time that we want to query the database for the most frequent bug report or something similar, we simply check the reported column.
Organization of Data
Receive phase
In this phase we define the way that crashreportd program will understand that a new crash report has arrived in the system. The crashreportd program is a daemon that checks constantly the directory where the crash reports arrive (/var/spool/crashreports). If the directory is not empty, it makes a list of the detected reports and start examining them one by one.
Check phase
This second phase exists primarily for security reasons. Our goal is to identify if a report is really a valid crash report and not junk that wants to pollute our system. In order to achieve this, we perform a series of checks on the examined report. A valid crash report is a tar.gz file with filename a strict naming pattern. The gunzip file contains only one file, which has a filename of the same pattern but ends with .xml. Also, the XML file should comply against a DTD that checks if the XML file is a well formed and valid crash report. If these checks are successful, then crashreportd sends a confirmation email to the submitter. This is the last check that will determine that the crash report is valid.
Recognize phase
The third phase is the recognize phase. Now that we know that the report is syntactically valid (we perform this phase without knowing that the report has been confirmed thought), crashreportd will try to determine if the report refers to a problem that previously logged reports had been referred too. If the report is similar with a previously logged bug, then we will assign the bug_id to the new report. If the report is not similar with any previously logged bug, then we will create a new bug in the Bugs table and we will assign the id of this bug to the report. If the report is similar with more than one logged bugs, then we will store the IDs of these bugs in the column bugs_id. This is an indication that our algorithm is not very accurate. The column bugs_id could be used from a developer in order to understand which bugs the algorithm thought that the report may refers to, and manually set the correct bug or create a new one.
The process of how we will actually decide if the new report is the same as a previously logged report is a crucial part of the system. Firstly, the whole process is based on the following assumptions:
- Every backtrace contains stack frames that identify the source of the problem, and stack frames that are in common with every stack backtrace.
Stack frames that have as a function name a name that contains the words syscall, panic, trap, lock, sleep or ?? are insignificant.
- Stack frames that have as a function name a name that does not contain the above words and lie between the last insignificant stack frame and the first insignificant stack frame after the significant stack frames (while searching from bottom to top).
The above assumptions have been derived from examining the backtraces of different kernel panics reported to the GNATS database, and from the problem reports found from the kernel stress test suite Stress 2 project.
For example, consider the following backtrace:
#0 kdb_backtrace #1 panic #2 rw_destroy #3 padlock_detach #4 device_detach #5 devclass_driver_deleted #6 devclass_delete_driver #7 driver_module_handler #8 module_unload #9 linker_file_unload #10 kern_kldunload #11 amd64_syscall #12 Xfast_syscall
From this backtrace, the significant frames are from #2 to #10. The most significant frame is the frame #2. The frames #0, #1 and #11, #12 are not vital in identifying the bug. How do we found that? We start searching from the bottom stack frame (i.e #12) up to the top. The #12 stack frame has a function name that contains the word syscall, so we mark it as insignificant. We continue with the #11 stack frame that also has the syscall word. We move to the #10 stack frame that does not contain any insignificant word. So it is the first significant stack frame that we found. We continue with the same manner up to the stack frame #1 that is the first time that we encounter an insignificant frame after our significant frames. This marks the end of our algorithm.
Also, consider the following backtrace:
#0 doadump #1 boot #2 panic #3 trap_fatal #4 trap #5 calltrap #6 rum_setup_tx_desc #7 rum_start #8 if_transmit #9 ieee80211_start #10 if_transmit #11 ether_output_frame #12 ether_output #13 bpfwrite #14 devfs_write_f #15 dofilewrite #16 kern_writev #17 write #18 syscall #19 Xfast_syscall #20 ??
From this backtrace, the significant frames are from #6 to #17. The most significant frame is the frame #6. The frames #0, #1, #2, #3, #4, #5 and #18, #19, #20 are not vital in the identification.
After examining the reports, we decided that the factors that will influence the matching of two reports are:
- the panic message
- the significant frames of the backtrace
From the significant stack frames, we say that the stack frame in the bottom is the less significant frame, while the frame in the top is the most significant frame.
In order to deal with different function names from release to release and from architecture to architecture, we have to perform a more flexible approach than comparing two strings character by character and return true if they are exactly the same or false if they are not.
For this reason, we add the notion of percentage in every factor. If two reports are exactly the same, then the matching returns a 100% similarity. For the two factors, we choose that the panic message costs 25% and the backtrace costs 75%. For the backtrace, the most significant frame costs 25% and all the others a 50%. For the remaining significant stack frames, we allocate incrementally and symmetrically this 50% percentage based on their place in the stack.
For every match, we calculate the similarity and if it is above 50%, then these reports refer to the same bug.
For a better understanding of the recognize phase, it is better to check the actual source code that is written in Python.
Store phase
The last phase is the store phase where crashreportd stores the report in the database as unconfirmed. We create a new user in Users if the email address of the report does not exist in our system and we create a new bug in Bugs if in the recognize phase we determined that this report refers to a new bug. Otherwise we assign the appropriate user_id and bug_id. This unconfirmed report will not be treated as a report until the user confirms it. That is, it will not presented in the website or compared against another report in the recognize phase.
Presentation of Data
Website
The website is a dynamic website responsible to present the data that we have collected to the public. Everyone is able to view the received crash reports along with their crash information and view the identified bugs. Only the submitters and the FreeBSD developers will be able to log in into the system. The developers can perform further actions like evaluating a crash report, changing the state of a bug and have access to the email address of submitters in order to contact them. In the future, more features can be implemented.
Central Collector Machine
In the duration of the development of the project and until the FreeBSD project accept the system, the Central Collector machine will be my desktop computer (probably this will change in the near future) running FreeBSD 9.0-RELEASE with a stable ADSL connection. The role of the DBMS will be played from the PostgreSQL Server 9.0.8, the Apache HTTP Server 2.22.22 will serve the website, and the Python 2.7.3 will execute the crashreportd program and the Pyramid powered website.
Toolbox
Our toolbox contains (Bourne) Shell programming for the construction of the crashreport program, the FreeBSD Operating System (a RELEASE version) for the Operating System of the Central Collector machine, the Apache HTTP Server as the Web Server that will serve the two web pages, the use of the ORDBMS PostgreSQL for the management of our database, the use of the Python programming language along with a set of libraries for the construction of the crashreportd program and the web framework Pyramid for the dynamic website.
Deliverables
An information system for automated and manual kernel crash reports for the FreeBSD project.
Milestones
May 21 - June 03 (Week 01, 02)
- Implementation of the crash reporter program (client-side)
June 04 - June 17 (Week 03, 04)
- Implementation of the rc.d script (server-side)
June 18 - July 01 (Week 05, 06)
- Connecting the client-side with the server-side
- XML crash reports
July 02 - July 15 (Week 07, 08)
- Implementation of the receive phase
July 16 - July 22 (Week 09)
- Implementation of the check phase
July 23 - July 29 (Week 10)
- Implementation of the store phase
July 30 - August 05 (Week 11)
- Implementation of the recognize phase
- Reorganization of server-side code
August 06 - August 12 (Week 12)
- Construction of the dynamic website that will present the data
August 13 - August 19 (Week 13)
- Updates for the website
- Documentation
- Testing the whole system
Test Plan
How to test the Client Side part
The recommended way to test the Client Side part is to create a virtual machine (ex. using VirtualBox) and install a FreeBSD version of your choice. Then, you can download the source in /usr/src from the socsvn repository and build the system. In the end, you will have a virtual machine running FreeBSD 10.0-CURRENT (at the moment a 2 months old revision) with this project's added code. The following is a guide that describes the steps after creating your FreeBSD virtual machine.
# Once installation is complete, obtain the Ports Collection portsnap fetch portsnap extract portsnap update # Install Subversion from the Ports Collection cd /usr/ports/devel/subversion make -DBATCH make install make clean # Check out the project's client side code into /usr/src (your Working Copy) svn checkout https://socsvn.freebsd.org/socsvn/soc2012/tzabal/client-side/akcrs-head/ /usr/src # The following is from: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html cd /usr/src make buildworld make buildkernel make installkernel shutdown -r now # Boot in single-user mode mount -u / mount -a -t ufs adjkerntz -i mergemaster -p cd /usr/src make installworld mergemaster reboot # Periodically update your working copy cd /usr/src svn update # In the case of minor changes, install manually # Example of having only the usr.sbin/crashreport updated cd usr.sbin/crashreport make install
The Code
http://svnweb.freebsd.org/socsvn/soc2012/tzabal/