Regression Test Suite for Audit Framework
Student: Aniket Pandey (<aniketp AT freebsd DOT org>)
Mentors:
Alan Somers (<asomers AT freebsd DOT org>)
Robert N. M. Watson (<rwatson AT freebsd DOT org>)
George V. Neville Neil (<gnn AT freebsd DOT org>)
Contents
- Regression Test Suite for Audit Framework
- Project Overview
- Project description
- Intricacies of Event Auditing
The Code
The Audit Test-Suite can be found here: AuditTestSuite
The audit(4) sub-directory within FreeBSD source tree: tests/sys/audit
The auditpipe(4) sub-directory within FreeBSD source tree: tests/sys/auditpipe
The praudit(1) sub-directory within FreeBSD source tree: usr.sbin/praudit/tests
Differential revisions Submitted
Week |
Differential |
Title |
Status |
1 |
Add initial set of tests for audit(4) subsystem |
||
2 |
#Include "bsm/audit.h" in "security/audit/audit_ioctl.h" for au_mask_t |
||
Correct the success return value for au_read_rec() in au_io.3 |
|||
3 |
Fix the incorrect argument types of auditpipe(4) in man-page |
||
audit(4): Add tests for readlink(2) and readlinkat(2) system calls |
|||
audit(4): Add tests for truncate(2) and ftruncate(2) system calls |
|||
4 |
Add tests for 5 syscalls in "file-delete" audit class |
||
Add tests for 4 syscalls in "file-close" audit class |
|||
Add tests for all 24 audit events of open(2) and openat(2) |
|||
Add tests for execve(2) and fexecve(2) in "ex" audit class |
Open |
||
Correct the value of ARG_TERMID_ADDR in security/audit/audit-private.h |
|||
5 |
Add tests for stat(2), fstat(2), lstat(2), fstatat(2) of "fa" audit class |
||
Syscalls dealing with {mounted/unmounted} file system statistics |
|||
Add a single test-program for praudit(1) utility |
|||
Fix the file-descriptor resource leakage as reported by Coverty |
|||
Initialize file-attribute-modify audit class test-program |
|||
Correct formatting of open(2) test-case generator macro |
|||
Syscalls dealing with file access via file-handles |
|||
Introduce test-program for network socket system calls |
|||
audit(4): Syscalls dealing with change of a file-mode |
|||
audit(4): Syscalls to check the accessibility of a file 'fa' |
|||
Add tests for syscalls used for modifying the ownership of file |
|||
Syscalls concerned with retrieving configurable pathname variables |
|||
audit(4): Add tests for bind(2), bindat(2) and listen(2) |
|||
6 |
Introduce test-program for System V standardized IPC syscalls |
||
audit(4): Add tests for connect(2), connectat(2) and accept(2) |
|||
audit(4): Syscalls concerned with modifying file 'flags' |
|||
Syscalls concerned with retrieving/listing the extended attributes of a file |
|||
SysV IPC syscalls for manipulating shared memory segments |
|||
Introduce test-program for administrative system calls |
|||
Syscalls concerned with modifying/deleting extended attributes of a file |
|||
Syscalls concerned with sending/receiving message via sockets |
|||
Add tests for {get/set}auid, {get/set}audit, {get/set}audit_addr |
|||
Introduce test-program for ioctl(2) system call |
|||
utimes(2), futimes(2), lutimes(2), futimesat(2), mprotect(2) and undelete(2) |
|||
Add tests for sendmsg(2), recvmsg(2), shutdown(2), sendfile(2) |
|||
SysV standardized IPC syscalls concerned with manipulating semaphore sets |
|||
Critical and miscellaneous administrative system calls |
|||
Add tests for pipe(2) and POSIX standardized IPC syscalls |
|||
7 |
Introduce Test Program for process-control group of syscalls |
||
Fix wrong sizeof() argument and descriptor leak reported by Coverity |
|||
audit(4): Add tests for setsid(2), wait4(2), wait6(2) and kill(2) |
|||
8 |
Process-Control: Add tests for procctl(2) system call |
||
Process-Control: Introduce tests for Capability mode System calls and _exit(2) |
|||
Introduce tests for sysctl(3), and sysarch(2) for AMD64, i386, ARM & MIPS |
|||
9 |
audit_syscall_exit: Don't acquire evclass_lock with a spinlock held |
||
auditon(2): Fix the misplacement of negation symbol in A_SETPOLICY command |
|||
Separate the tests for auditon(2) based on individual commands |
|||
10 |
Add initial expected failures cases for chflagsat(2), lgetfh(2), setfib(2) |
||
Default case and ENOSYS expected cases for auditon(2) system call |
|||
11 |
Fix audit of chflagsat(2), lgetfh(2), setfib(2) (reviewed) |
||
Introduce test-program for auditpipe(4) initial ioctls |
|||
Fix incorrect operator in the AUDITPIPE_SET_QLIMIT bounds check |
|||
12 |
Add various other ioctls for auditpipe(4) and bound-check condition |
Open |
|
Introduce test-program for auditon(2) syscall's functionalities |
Open |
Project Overview
FreeBSD is a rapidly developing operating system with an extreme focus on advanced security & networking features. For an OS with a widespread usage and development, testing and monitoring of security regressions becomes a critical measure. FreeBSD has an audit subsystem which is responsible for monitoring a variety of security-relevant system events, such as user-logins, configuration changes, file system & network access. Although the audit framework is indispensable for security conscious organisations running FreeBSD servers, currently there is no tool to test its reliability and the intended behaviour.
Project description
The project aims to develop a regression test-suite, which will evaluate the audit framework for proper logging of most auditable system calls classified in TCP/IP & UDP sockets, File I/O, process control and device management, along with the semantics of audit trail's BSM/XML/text output.
BSM tokens can be obtained via synchronous I/O multiplexing on a special clonable device /dev/auditpipe, by configuring various preselection parameters for local mode auditing with the provided IOCTLs. Several libbsm(3) APIs and functions within the FreeBSD kernel can be used to analyse syscall tokens in the audit record. Finally, kyua(7)'s run-time engine will be used to automate regression testing of entire operating system at once, audit(4) included.
Explicit System call testing
This application would consist of a set of test programs written in atf-c(3), containing independent test cases for triggering similar system calls, e.g open(2) & openat(2). Each program would test a wide variety of functionalities of corresponding system calls. Testing scenarios would ensure that the audit record contains all expected parameters, e.g the arguments, valid argument types, return values etc. The testing will be done for various success and failure modes, with cross-checking for appropriate errno codes in case of failure mode.
Intricacies of Event Auditing
How is event auditing implemented
FreeBSD uses OpenBSM, an implementation of Sun's BSM event auditing file format and API. OpenBSM is made up of a plethora of tools including audit-viewer applications, e.g praudit(1) & auditreduce(1) as well as the libbsm(3) library to provide an interface to BSM audit record stream. The auditd(8) daemon is responsible for managing the kernel’s audit(4) mechanism and also rotates the log files whenever required.
How to enable event auditing
FreeBSD already has user-space support for audit system. The audit daemon, auditd(8) can be enabled by adding the following line to /etc/rc.conf:
echo ‘auditd_enable=“YES”’ >> /etc/rc.conf
A safer way is to use sysrc(8)
sysrc auditd_enable=“YES”
Then start (and stop) the audit daemon, with a new audit trail:
service auditd {one}start && audit -n;
service auditd {one}stop
Various configuration options in audit framework
dir=‘/etc/security’
Audit configuration is defined in ${dir}/audit_control. Various event selection expressions are defined here, also found in ${dir}/audit_class, which lets a user configure the events to be audited. For example,
lo for login-logout
nt for network communication
pc for process control
ad for administration
${dir}/audit_user specifies events to be audited for each system user, like login attempts. ${dir}/audit_event contains a list of all auditable system events.
Approach to solving the problem
The test program here is an initial attempt at creating independent tests for similar system calls which can then be automated by Kyua.
The independence is achieved by following these steps:-
In a situation where the system does not have the audit system enabled, the tests would usually fail or skip. However, the audit daemon can be started (and stopped) by “service auditd one{start, stop}” without modifying /etc/rc.conf. This eliminates the need for a setup script as described in the deprecated test plan.
Create separate functions for the startup and termination of the audit daemon. This step also takes into account the possibility of auditd(8) running beforehand. If so, the process termination is avoided to maintain the current state of the machine. The running status can be checked by service auditd onestatus.
Audit system also allows live auditing via auditpipe(4) for IDS and testing purpose. This enables us to open /dev/auditpipe and verify the proper audit of the system call in question. Usually in cases like these, poll(2) helps in confirming if the opened file descriptor is ready to perform I/O operations.
The system may not have the proper settings in /etc/security. This would fail the events not covered by “audit_control” configuration file. However, auditpipe(4) describes quite a few ioctl(2) requests in audit_ioctl.h for such scenarios. getauclassnam(3) can be used to obtain required information from the audit_class(5) database.
Deliverables
Deliverables for the first evaluation
- Set of tests for a small subset of the system calls supported by OpenBSM.
- Initial work done on writing Kyua compatible test programs, presumably using atf-c-api(3).
Deliverables for the second evaluation
- A comprehensive and automated test-suite covering “most” system calls.
- Stand-alone tests for all system calls tested till now, to facilitate regression testing of the entire operating system at once.
- Set of test cases for checking file-format of audit-trails (text/XML/BSM).
Deliverables for the third evaluation
- Conclusion of independent and stand-alone tests for audit system, arranged in the appropriate directory structure.
atf-sh(3) & atf-c(3) tests for all audit tracepoints integrated into Kyua framework.
- An attempt will be made to test the integrity of audit_control(5) configuration file. (Stretch Goal)
Weekly Reports
Week |
Report |
1 |
|
2 |
|
3 |
|
4 |
|
5 |
|
6 |
|
7 |
|
8 |
|
9-12 |
Test Suite: Audit Class Summary
Audit Class |
No of Syscalls |
No of Test cases |
administrative |
16 |
25 |
file-close |
4 |
7 |
program execution |
2 |
4 |
file-attribute-access |
24 |
82 |
file-create |
14 |
48 |
file-delete |
7 |
34 |
file-attribute-modify |
29 |
89 |
file-read |
4 |
36 |
file-write |
4 |
36 |
ioctl control |
1 |
2 |
inter process comm. |
15 |
59 |
network socket |
18 |
36 |
process control |
29 |
47 |
Total |
167 |
505 |
List of System Calls tested
System Calls |
Success Mode |
Failure Mode |
Audit Class |
* |
System Calls |
Success Mode |
Failure Mode |
Audit Class |
||||||||||
socket(2) |
Passed |
Passed |
network |
* |
open(2) |
Passed |
Passed |
Multiple |
||||||||||
socketpair(2) |
Passed |
Passed |
network |
* |
openat(2) |
Passed |
Passed |
Multiple |
||||||||||
bind(2) |
Passed |
Passed |
network |
* |
mkdir(2) |
Passed |
Passed |
file-create |
||||||||||
bindat(2) |
Passed |
Passed |
network |
* |
mkdirat(2) |
Passed |
Passed |
file-create |
||||||||||
setsockopt(2) |
Passed |
Passed |
network |
* |
mkfifo(2) |
Passed |
Passed |
file-create |
||||||||||
listen(2) |
Passed |
Passed |
network |
* |
mkfifoat(2) |
Passed |
Passed |
file-create |
||||||||||
accept(2) |
Passed |
Passed |
network |
* |
mknod(2) |
Passed |
Passed |
file-create |
||||||||||
sendto(2) |
Passed |
Passed |
network |
* |
mknodat(2) |
Passed |
Passed |
file-create |
||||||||||
send(2) |
Passed |
Passed |
network |
* |
rename(2) |
Passed |
Passed |
file-create |
||||||||||
recvfrom(2) |
Passed |
Passed |
network |
* |
renameat(2) |
Passed |
Passed |
file-create |
||||||||||
recv(2) |
Passed |
Passed |
network |
* |
link(2) |
Passed |
Passed |
file-create |
||||||||||
connect(2) |
Passed |
Passed |
network |
* |
linkat(2) |
Passed |
Passed |
file-create |
||||||||||
connectat(2) |
Passed |
Passed |
network |
* |
symlink(2) |
Passed |
Passed |
file-create |
||||||||||
sendmsg(2) |
Passed |
Passed |
network |
* |
symlinkat(2) |
Passed |
Passed |
file-create |
||||||||||
recvmsg(2) |
Passed |
Passed |
network |
* |
readlink(2) |
Passed |
Passed |
file-read |
||||||||||
shutdown(2) |
Passed |
Passed |
network |
* |
readlinkat(2) |
Passed |
Passed |
file-read |
||||||||||
sendfile(2) |
Passed |
Passed |
network |
* |
truncate(2) |
Passed |
Passed |
file-write |
||||||||||
setfib(2) |
Passed |
Passed |
network |
* |
ftruncate(2) |
Passed |
Passed |
file-write |
||||||||||
msgget(2) |
Passed |
Passed |
inter-process |
* |
rmdir(2) |
Passed |
Passed |
file-delete |
||||||||||
msgsnd(2) |
Passed |
Passed |
inter-process |
* |
rename(2) |
Passed |
Passed |
file-delete |
||||||||||
msgrcv(2) |
Passed |
Passed |
inter-process |
* |
renameat(2) |
Passed |
Passed |
file-delete |
||||||||||
msgctl(2) |
Passed |
Passed |
inter-process |
* |
unlink(2) |
Passed |
Passed |
file-delete |
||||||||||
shmget(2) |
Passed |
Passed |
inter-process |
* |
unlinkat(2) |
Passed |
Passed |
file-delete |
||||||||||
shmat(2) |
Passed |
Passed |
inter-process |
* |
munmap(2) |
Passed |
Passed |
file-close |
||||||||||
shmdt(2) |
Passed |
Passed |
inter-process |
* |
close(2) |
Passed |
Passed |
file-close |
||||||||||
shmctl(2) |
Passed |
Passed |
inter-process |
* |
closefrom(2) |
Passed |
No Failure |
file-close |
||||||||||
shm_open(2) |
Passed |
Passed |
inter-process |
* |
revoke(2) |
Passed |
Passed |
file-close |
||||||||||
shm_unlink(2) |
Passed |
Passed |
inter-process |
* |
stat(2) |
Passed |
Passed |
file-attribute access |
||||||||||
semget(2) |
Passed |
Passed |
inter-process |
* |
lstat(2) |
Passed |
Passed |
file-attribute access |
||||||||||
semop(2) |
Passed |
Passed |
inter-process |
* |
fstat(2) |
Passed |
Passed |
file-attribute access |
||||||||||
semctl(2) |
Passed |
Passed |
inter-process |
* |
fstatat(2) |
Passed |
Passed |
file-attribute access |
||||||||||
pipe(2) |
Passed |
Passed |
inter-process |
* |
statfs(2) |
Passed |
Passed |
file-attribute access |
||||||||||
posix_openpt(2) |
Passed |
Passed |
inter-process |
* |
fstatfs(2) |
Passed |
Passed |
file-attribute access |
||||||||||
flock(2) |
Passed |
Passed |
file-attribute modify |
* |
getfsstat(2) |
Passed |
Passed |
file-attribute access |
||||||||||
fcntl(2) |
Passed |
Passed |
file-attribute modify |
* |
lgetfh(2) |
Passed |
Passed |
file-attribute access |
||||||||||
fsync(2) |
Passed |
Passed |
file-attribute modify |
* |
fhopen(2) |
Passed |
Passed |
file-attribute access |
||||||||||
chmod(2) |
Passed |
Passed |
file-attribute modify |
* |
fhstat(2) |
Passed |
Passed |
file-attribute access |
||||||||||
fchmod(2) |
Passed |
Passed |
file-attribute modify |
* |
fhstatfs(2) |
Passed |
Passed |
file-attribute access |
||||||||||
lchmod(2) |
Passed |
Passed |
file-attribute modify |
* |
access(2) |
Passed |
Passed |
file-attribute access |
||||||||||
fchmodat(2) |
Passed |
Passed |
file-attribute modify |
* |
eaccess(2) |
Passed |
Passed |
file-attribute access |
||||||||||
chown(2) |
Passed |
Passed |
file-attribute modify |
* |
faccessat(2) |
Passed |
Passed |
file-attribute access |
||||||||||
fchown(2) |
Passed |
Passed |
file-attribute modify |
* |
pathconf(2) |
Passed |
Passed |
file-attribute access |
||||||||||
lchown(2) |
Passed |
Passed |
file-attribute modify |
* |
lpathconf(2) |
Passed |
Passed |
file-attribute access |
||||||||||
fchownat(2) |
Passed |
Passed |
file-attribute modify |
* |
fpathconf(2) |
Passed |
Passed |
file-attribute access |
||||||||||
extattr_set_file(2) |
Passed |
Passed |
file-attribute modify |
* |
extattr_get_file(2) |
Passed |
Passed |
file-attribute access |
||||||||||
extattr_set_fd(2) |
Passed |
Passed |
file-attribute modify |
* |
extattr_get_fd(2) |
Passed |
Passed |
file-attribute access |
||||||||||
extattr_set_link(2) |
Passed |
Passed |
file-attribute modify |
* |
extattr_get_link(2) |
Passed |
Passed |
file-attribute access |
||||||||||
extattr_delete_file(2) |
Passed |
Passed |
file-attribute modify |
* |
extattr_list_file(2) |
Passed |
Passed |
file-attribute access |
||||||||||
extattr_delete_fd(2) |
Passed |
Passed |
file-attribute modify |
* |
extattr_list_fd(2) |
Passed |
Passed |
file-attribute access |
||||||||||
extattr_delete_link(2) |
Passed |
Passed |
file-attribute modify |
* |
extattr_list_link(2) |
Passed |
Passed |
file-attribute access |
||||||||||
chflags(2) |
Passed |
Passed |
file-attribute modify |
* |
execve(2) |
Passed |
Passed |
exec, process-control |
||||||||||
lchflags(2) |
Passed |
Passed |
file-attribute modify |
* |
fexecve(2) |
Passed |
Passed |
exec, process-control |
||||||||||
fchflags(2) |
Passed |
Passed |
file-attribute modify |
* |
ioctl(2) |
Passed |
Passed |
ioctl |
||||||||||
chflagsat(2) |
Passed |
Passed |
file-attribute modify |
* |
fork(2) |
Passed |
Not Possible |
process control |
||||||||||
utimes(2) |
Passed |
Passed |
file-attribute modify |
* |
rfork(2) |
Passed |
Passed |
process control |
||||||||||
futimes(2) |
Passed |
Passed |
file-attribute modify |
* |
chdir(2) |
Passed |
Passed |
process control |
||||||||||
lutimes(2) |
Passed |
Passed |
file-attribute modify |
* |
fchdir(2) |
Passed |
Passed |
process control |
||||||||||
futimesat(2) |
Passed |
Passed |
file-attribute modify |
* |
chroot(2) |
Passed |
Passed |
process control |
||||||||||
mprotect(2) |
Passed |
Passed |
file-attribute modify |
* |
umask(2) |
Passed |
No Failure |
process control |
||||||||||
undelete(2) |
Not Possible |
Passed |
file-attribute modify |
* |
setuid(2) |
Passed |
No Failure |
process control |
||||||||||
settimeofday(2) |
Passed |
Passed |
administrative |
* |
seteuid(2) |
Passed |
No Failure |
process control |
||||||||||
adjtime(2) |
Passed |
Passed |
administrative |
* |
setgid(2) |
Passed |
No Failure |
process control |
||||||||||
nfs_getfh(2) |
Passed |
Passed |
administrative |
* |
setegid(2) |
Passed |
No Failure |
process control |
||||||||||
getauid(2) |
Passed |
Passed |
administrative |
* |
setreuid(2) |
Passed |
No Failure |
process control |
||||||||||
setauid(2) |
Passed |
Passed |
administrative |
* |
setregid(2) |
Passed |
No Failure |
process control |
||||||||||
getaudit_addr(2) |
Passed |
Passed |
administrative |
* |
setresuid(2) |
Passed |
Passed |
process control |
||||||||||
setaudit_addr(2) |
Passed |
Passed |
administrative |
* |
setresgid(2) |
Passed |
Passed |
process control |
||||||||||
reboot(2) |
Not Possible |
Passed |
administrative |
* |
getresuid(2) |
Passed |
Passed |
process control |
||||||||||
acct(2) |
Passed |
Passed |
administrative |
* |
getresgid(2) |
Passed |
Passed |
process control |
||||||||||
quotactl(2) |
Not Possible |
Passed |
administrative |
* |
setpriority(2) |
Passed |
Passed |
process control |
||||||||||
mount(2) |
Not Possible |
Passed |
administrative |
* |
setgroups(2) |
Passed |
Passed |
process control |
||||||||||
nmount(2) |
Not Possible |
Passed |
administrative |
* |
setpgrp(2) |
Passed |
Passed |
process control |
||||||||||
ntp_adjtime(2) |
Passed |
Passed |
administrative |
* |
setrlimit(2) |
Passed |
Passed |
process control |
||||||||||
auditctl(2) |
Passed |
Passed |
administrative |
* |
execve(2) |
Passed |
Passed |
process control |
||||||||||
swapon(2) |
Passed |
Passed |
administrative |
* |
fexecve(2) |
Passed |
Passed |
process control |
||||||||||
swapoff(2) |
Passed |
Passed |
administrative |
* |
mlock(2) |
Passed |
Passed |
process control |
||||||||||
clock_settime(2) |
Passed |
Passed |
administrative |
* |
munlock(2) |
Passed |
Passed |
process control |
||||||||||
auditon(2) |
Passed |
Passed |
administrative |
* |
minherit(2) |
Passed |
Passed |
process control |
||||||||||
setsid(2) |
Passed |
Passed |
process control |
* |
setlogin(2) |
Passed |
Passed |
process control |
||||||||||
wait4(2) |
Passed |
Passed |
process control |
* |
rtprio(2) |
Passed |
Passed |
process control |
||||||||||
wait6(2) |
Passed |
Passed |
process control |
* |
profil(2) |
Passed |
Passed |
process control |
||||||||||
kill(2) |
Passed |
Passed |
process control |
* |
ptrace(2) |
Passed |
Passed |
process control |
||||||||||
killpg(2) |
Passed |
Passed |
process control |
* |
ktrace(2) |
Passed |
Passed |
process control |
||||||||||
procctl(2) |
Passed |
Passed |
process control |
* |
cap_enter(2) |
Passed |
No failure |
process control |
||||||||||
jail(2) |
Not done |
Passed |
process control |
* |
cap_getmode(2) |
Passed |
Passed |
process control |
||||||||||
sysctl(3) |
Passed |
Passed |
miscellaneous |
* |
_exit(2) |
Passed |
No failure |
process control |
||||||||||
audit(2) |
Passed |
Passed |
miscellaneous |
* |
sysarch(2) |
Passed |
Passed |
miscellaneous |
||||||||||
Note:
Evaluating the proper audit of these system calls in either success or failure mode (or both) was not possible due to administrative and security issues. 
BSD libc converts open(2) to openat(2) to optimize resource usage when certain arguments are passed. The testing was done using syscall(2). This Wiki's general construct was heavily borrowed from SmokeTestingOfBaseUtilities and TCP-IP-RegressionTestSuite, authored by ShivanshRai.
Acknowledgement
I can't thank my mentors enough for all the help they provided during the summers. Also, thanks to ShivanshRai, MateuszPiotrowski & DomagojStolfa for helping me at various stages of my project when I was stuck. It has been a valuable learning experience and definitely something way more than I could have expected for a summer! I look forward to continue the work on improving the audit(4) subsystem and also exploring various other areas of FreeBSD.
And thanks to The FreeBSD Project for accepting this project and welcoming me into the community!