• SummerOfCode2014
• IntelSMAPandKernelPatching

Implement Intel SMAP and kernel patching framework

• Student: OliverPinter

• Mentor: GavinAtkinson

Project description

In first phase, I want to implement the Intel SMAP (Supervisor Mode Access Prevention) technology for x86-64 architecture. In second phase, I plan to implement boot/load time kernel and kernel module patching framework.

Approach to solving the problem

Intel SMAP is a hardware extension to support advanced kernel self-protection. The SMAP technology will prevent unintended data access from kernel to userland memory. The technology will appear in Intel Broadwell architecture in 2014Q2/Q3. Currently there is an emulator - namely Qemu with TCG - which supports this technology.

Runtime kernel/kernel module patching is required, otherwise the processor will fail when processing unknown instruction. Newer processors introducing newer instructions which didn't exist on older one. To solve this situation this framework makes the kernel and kernel modules self-modifiable in common way.

Deliverables

phase #1:

• - Improved security of FreeBSD kernel in future x86-64 processors

phase #2:

• - generic framework for boot-time/runtime kernel image and kernel modules patching

  - elliminate hackish "manual" instruction patching: http://svnweb.freebsd.org/base/head/sys/amd64/amd64/cpu_switch.S?r1=238450&r2=238449&pathrev=238450

Milestones

phase #1:

• 1.1	May 19 - May 25	update Intel SMAP knowledege	DONE
  1.2	May 26 - June 8	update relevant FreeBSD kernel knowledge	DONE
  1.3	June 9 - June 15	implement/refine trap handler and add/refine required code to relevant parts of kernel	DONE
  1.4	June 16 - June 22	test and fix	DONE

phase #2:

• 2.1	June 23 - June 29	identify the required places to modify in booting process and kernel module loading process	DONE
  2.2	June 30 - July 6	design the kernel patching framework	DONE
  2.3	July 7 - July 20	implement the kernel patching framework	DONE
  2.4	July 21 - July 27	adapt XSAVE and SMAP instructions to new framework	DONE
  2.5	July 27 - August 3	optimize implementation	DONE
  2.6	August 3 - EoC August 14	test, test, fix, test	DONE

Test Plan

phase #1 - SMAP:

• create a VM image	http://svnweb.freebsd.org/socsvn/soc2014/op/tools/create_smap_test_vm.csh?view=log	DONE
  write vulnerable kernel module and PoC, and test	http://svnweb.freebsd.org/socsvn/soc2014/op/tests/smap-tester/	DONE
  test in qemu with SMAP emulation	http://svnweb.freebsd.org/socsvn/soc2014/op/tools/run_smap_test_vm.csh?view=log	DONE

phase #2 - kernel patching:

• create a VM image		DONE
  boot test in qemu		DONE
  kernel module test in qemu		DONE
  test in qemu with enabled SMAP		DONE
  test on real hardware with XSAVE/XSAVEOPT		DONE
  stress test		DONE

The Code

https://svnweb.freebsd.org/socsvn/soc2014/op/

https://github.com/opntr

https://github.com/opntr/opBSD/tree/op/gsoc2014/master

https://github.com/opntr/opBSD/tree/op/gsoc2014/smap

https://github.com/opntr/opBSD/tree/op/gsoc2014/kpatch

http://www.crysys.hu/~op/freebsd/vm-image/

Useful links

http://forums.grsecurity.net/viewtopic.php?f=7&t=3046

https://lwn.net/Articles/517475/

http://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/subversion-primer.html

http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-manual-325462.pdf

https://software.intel.com/sites/default/files/managed/68/8b/319433-019.pdf

http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2012/10/24592_APM_v11.pdf

http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2012/10/24593_APM_v21.pdf

http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2008/10/24594_APM_v3.pdf

http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2012/10/26568_APM_v41.pdf

http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2012/10/26569_APM_v51.pdf

http://tigcc.ticalc.org/doc/gnuasm.html#SEC109

http://llvm.org/docs/doxygen/html/AsmParser_8cpp_source.html

http://clang.llvm.org/compatibility.html

http://www.ibm.com/developerworks/aix/library/au-gdb.html

https://sourceware.org/binutils/docs/ld/Scripts.html#Scripts

1.1

doc

• 319433-019.pdf : 10.3; table 2-5; page 887; page 894

CPUID

• CPUID.(EAX=07H,ECX=0H):EBX.SMAP[bit 20] = 1

instructions

• stac:

  • STAC sets the AC flag bit in EFLAGS/RFLAGS without affecting other bits. Attempt to execute STAC when CPL > 0 will cause #UD. This instruction's operation is the same in non-64-bit modes and 64-bit mode.

  clac:

  • CLAC clears the AC flag bit in EFLAGS/RFLAGS without affecting other bits. Attempt to execute CLAC when CPL > 0 will cause #UD. This instruction's operation is the same in on-64-bit modes and 64-bit mode.

status bit

• EFLAGS.AC

enabling

• "System software enables SMAP by setting the SMAP flag in control register CR4 (bit 21)."

1.2

• clang SMAP support	YES	https://github.com/freebsd/freebsd/commit/aa45f148926e3461a1fd8b10c990f0a51a908cc9	dim@
  gcc SMAP support	YES	https://github.com/freebsd/freebsd/commit/72b58728c7463667c6eb881e2f8a0c6d98e8c661	pfg@
  ddb SMAP support	YES	https://github.com/freebsd/freebsd/commit/0fa3283d5a6ef0a3095fd7dd71bf4cea8ecd83ee	jhb@

trap handler memo

• "If CPL < 3, SMAP protections are disabled if EFLAGS.AC = 1. If CPL = 3, SMAP applies to all supervisor-mode data accesses (these are implicit supervisor accesses) regardless of the value of EFLAGS.AC." - Intel Ref. # 319433-014 9.3.2

•  BOOL
   check_smap_access_violation(FRAME, USERMODE)
   {
    if SMAP_DISABLED :
     RETURN FALSE;
   
    if  CPL == 3 OR RFLAGS@FRAME BITAND EFLAGS.AC :
     RETURN FALSE;
   
    RETURN TRUE;
   }

1.3

kernel parts

• exception handlers		sys/amd64/amd64/exception.S
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269004
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269065
  	git	https://github.com/opntr/opBSD/commit/95042600c3fdaa1be4ed89e0a07cdd087ad8973b
  	git	https://github.com/opntr/opBSD/commit/4916593d9b06dded2d9b91ad24fee80db3b01b83
  		sys/i386/i386/exception.s
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269490
  	git	https://github.com/opntr/opBSD/commit/ff125771d93fd57776bf2482726111cb7fe342ec
  cpu identification		sys/amd64/amd64/identcpu.c
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=268999
  	git	https://github.com/opntr/opBSD/commit/33b3063c833f2b5d1d04298b38a30ec59a466d01
  		sys/i386/i386/identcpu.c
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269491
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269656
  	git	https://github.com/opntr/opBSD/commit/a87bb07998a4f5fac9b9dfa1f6f31c11f46e4b62
  	git	https://github.com/opntr/opBSD/commit/3a5c691cf6427f5225fe0eb121076d8e4c404239
  cpu initialization		sys/amd64/amd64/initcpu.c
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269000
  	git	https://github.com/opntr/opBSD/commit/de1859f4cbb12f9ca9517b74e0fa6705094ab34e
  		sys/i386/i386/initcpu.c
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269505
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269655
  	git	https://github.com/opntr/opBSD/commit/c398ca5d1009d5e77be877afcdf9c4c6e0b5ad34
  	git	https://github.com/opntr/opBSD/commit/4b488dfbfcd07aefcc126a6c433283047484ed0e
  low-level memory management		sys/amd64/amd64/pmap.c
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269002
  	git	https://github.com/opntr/opBSD/commit/de1859f4cbb12f9ca9517b74e0fa6705094ab34e
  		sys/i386/i386/pmap.c
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269504
  	git	https://github.com/opntr/opBSD/commit/75dde2528c3ebb841f4227a3de5236905bd32965
  low-level functions		sys/amd64/amd64/support.S
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269063
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269064
  	git	https://github.com/opntr/opBSD/commit/803e9dc13fb7cf182749e3680cd6b4f74e13fcaa
  	git	https://github.com/opntr/opBSD/commit/555ad43c95b506b58eaff989a5e3f5b8ca854897
  		sys/i386/i386/support.s
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269495
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269496
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269497
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269659
  	git	https://github.com/opntr/opBSD/commit/3411fb2ebd8cac1fb759f619e62b7ca3d35ef866
  	git	https://github.com/opntr/opBSD/commit/013136cb04966afd3fc4f04abf23ba3f77d701f5
  	git	https://github.com/opntr/opBSD/commit/c6b2b9427cac8c43db66b864ebe4eb06c16c3726
  	git	https://github.com/opntr/opBSD/commit/f9543ea8d683841ea33b8069617889ff1b195913
  trap handler		sys/amd64/amd64/trap.c
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269001
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269489
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269492
  	git	https://github.com/opntr/opBSD/commit/2388bdd2fae0cb28964316a1144b8fb6be4d2bf7
  	git	https://github.com/opntr/opBSD/commit/907e4c655b4481e6cdb750c0e1b397377cc4fd2f
  	git	https://github.com/opntr/opBSD/commit/6dba3deadc1a89fbe733ce6a4bfed242ea8728ec
  		sys/i386/i386/trap.c
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269487
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269488
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269657
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269658
  	git	https://github.com/opntr/opBSD/commit/1776453a82f80c255f01176821b404f4f3d19aac
  	git	https://github.com/opntr/opBSD/commit/37c9163a2ab9259718d74a43a9f6e45bafb718fb
  	git	https://github.com/opntr/opBSD/commit/9163a4943f56412f33f80738f5e6bc98a184a499
  	git	https://github.com/opntr/opBSD/commit/5a79321d287c35c339a2ef1292140c5af70454a8
  asm macros		sys/amd64/include/asmacros.h
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269003
  	git	https://github.com/opntr/opBSD/commit/7618c2fb397920fc3908a96784a7b86be344b37e
  		sys/i386/include/asmacros.h
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269502
  	git	https://github.com/opntr/opBSD/commit/913d70df386e0f509914b927f6ac5fc3820b126a
  low-level functions		sys/amd64/include/cpufunc.h
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=268658
  	git	https://github.com/opntr/opBSD/commit/ceba73a18c80975382d9ec621efcec36376c7622
  		sys/i386/include/cpufunc.h
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269503
  	git	https://github.com/opntr/opBSD/commit/d2a7cc7a91a89fba06015840a2b8078eabdb8915
  configuration NOTES		sys/conf/NOTES
  	svn
  	git
  configurations		sys/conf/options.amd64
  	svn
  	git
  		sys/conf/options.i386
  	svn
  	git
  eflags flags		sys/x86/include/psl.h
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=268998
  	git	https://github.com/opntr/opBSD/commit/4482d7f9a8a9296b3519f46e0f48eea3eee596b6
  cpu flags		sys/x86/include/specialreg.h
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=268997
  	git	https://github.com/opntr/opBSD/commit/3c7e40e5aad8ddc467949aef6ed8bae2d0729d14
  compat exceptions		sys/amd64/ia32/ia32_exception.S
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269062
  	git	https://github.com/opntr/opBSD/commit/5c65f66371c725a8fa6155de9dfc352adb88b4e7
  disassembler		sys/i386/i386/db_disasm.c
  	svn	http://svnweb.freebsd.org/socsvn?view=revision&revision=269486
  	git	https://github.com/opntr/opBSD/commit/10ece4f886860feffeb816426fafdf5d69e22601

notes

• i386 part originally not part of this project
• i386 part currently not tested
• do NOT compile the kernel with CPUTYPE=core-avx2

2.1

• linker script	sys/conf/ldscript.amd64
  elf linker	sys/kern/kern_linker.c
  	sys/kern/link_elf.c
  	sys/sys/linker.h
  nop tables	XXX
  kernel image patching	sys/amd64/amd64/machdep.c
  files, which containes stac, clac or xsave	sys/amd64/amd64/support.S
  	sys/amd64/include/asmacros.h
  	sys/amd64/include/cpufunc.h
  	sys/amd64/ia32/ia32_exception.S
  	amd64/amd64/cpu_switch.S

• potential useable kernel API's:
  • linker_file_lookup_set(...)
  • pmap_enter(...)
  • pmap_kenter(...)
  • sys/linker_set.h
  • CPU rendezvous

2.2

• basic structures
  • lf_selfpatch_t

    • typedef struct lf_selfpatch {
              void    *patchable;           /* address of patchable instractions */
              void    *patch;               /* address of patch */
              short   feature_selector;
              int     feature;
              char    *comment;
      } lf_selfpatch_t;

• elf sections
  • selfpatch_metainfo
  • selfpatch_patch

• assembly instruction size calculation like in https://github.com/freebsd/freebsd/blob/master/sys/amd64/include/asmacros.h#L217

• patching scenarios
  • kernel

    • -> hammer_time

      • -> lf_selfpatch(linker_kernel_file)

  • module

    • -> modload

      • -> lf_selfpatch(linker_module_file)

  • module -> kernel (patching the kernel image, from newly loaded kernel modules, probably one way to apply security patches run-time)

    • -> onload

      • -> lf_patch(linker_module_file, linker_destination_file)

        • -> smp_rendezvous_cpu(NULL, lf_selfpatch(foo), NULL, arg)

2.3

• implemented kernel image patching on boot
• implemented module patching at preload and on load
• tested kernel image patching on boot
• updated build tools and maintaining tools

• see commits in https://github.com/opntr/opBSD/tree/op/gsoc2014/kpatch

tools

• svn-cherry-pick-from-git.csh	http://svnweb.freebsd.org/socsvn/soc2014/op/tools/svn-cherry-pick-from-git.csh?view=log
  build_kernel_32bit.csh	http://svnweb.freebsd.org/socsvn/soc2014/op/tools/build_kernel_32bit.csh?view=log
  build_kernel_64bit.csh	http://svnweb.freebsd.org/socsvn/soc2014/op/tools/build_kernel_64bit.csh?view=log
  create_smap_test_vm.csh	http://svnweb.freebsd.org/socsvn/soc2014/op/tools/create_smap_test_vm.csh?view=log
  run_smap_test_vm.csh	http://svnweb.freebsd.org/socsvn/soc2014/op/tools/run_smap_test_vm.csh?view=log

weekly status

• 20140529	http://lists.freebsd.org/pipermail/soc-status/2014-May/000777.html
  20140606	http://lists.freebsd.org/pipermail/soc-status/2014-June/000787.html
  20140613	http://lists.freebsd.org/pipermail/soc-status/2014-June/000797.html
  20140620	http://lists.freebsd.org/pipermail/soc-status/2014-June/000810.html
  20140628	http://lists.freebsd.org/pipermail/soc-status/2014-June/000819.html
  20140704	http://lists.freebsd.org/pipermail/soc-status/2014-July/000829.html
  20140711	http://lists.freebsd.org/pipermail/soc-status/2014-July/000829.html
  20140718	http://lists.freebsd.org/pipermail/soc-status/2014-July/000838.html
  20140725	http://lists.freebsd.org/pipermail/soc-status/2014-July/000845.html
  20140801	http://lists.freebsd.org/pipermail/soc-status/2014-August/000863.html
  20140808	http://lists.freebsd.org/pipermail/soc-status/2014-August/000863.html
  20140815	http://lists.freebsd.org/pipermail/soc-status/2014-August/000874.html

test results and memos

1

1.3

• SummerOfCode2014/IntelSMAPandKernelPatching/Notes

commits