Machine readable output from userland utilities
Student: ZarkoKorchev (zkorchev@)
Mentor: EitanAdler (lists@eitanadler.com)
Project description
The output of many commands is intended to be read by humans. An unfortunate consequence is that machine parsing of these utilities is difficult and time consuming. The unfortunate effect is that retrieving most infomration requires either complex scripting and regular expressions, or it requires coding C directly to the system calls.
This proposal aims to fix that by providing an interface for userland utilities to emit machine readable output which can be easily parsed by other software components.
Commercial vendors have been modifying FreeBSD for many years to emit machine readable (XML/JSON/YAML) output for common commands to aid scripting against their implementations. A noteable example is JUNOS from Juniper Networking.
What we would do is leverage an existing library for emiting the most commmon scalar types (string, integer, float, array) and modify base FreeBSD utilities to call a library emit XML, JSON or YAML as opposed to directly calling printf.
Utilities we will tackle include:
- sysctl
- ifconfig
- netstat
- iostat
- vmstat
Approach to solving the problem
I am going to use the existing libraries (libnv and libucl). In the process I will make some changes to these libraries (like adding XML support to libucl). I will convert command line utilities like ls, sysctl, ifconfig, netstat, iostat, vmstat, procstat, fstat, etc. to use these libraries as an output layer.
Deliverables
- Support for JSON, XML and YAML output for ls, sysctl, ifconfig, netstat, iostat, vmstat, procstat, fstat.
- A library with plugable backends for formatted data output.
- Documentation and blog posts during the writing process.
After this project is ready, it will be easy to add support for machine-readable output to any command line tool. The converted tools will serve as examples for future contributors.
Milestones
- Until 18 May: Get familiar with the command-line tools and the libraries. Analyze the output format of the command-line tools.
- 19 May - 25 May: Design a suitable output format for the command-line utilities.
- 26 May - 1 June: Design output abstractions that will allow to easily switch between output formats. Modify the output libraries if necessary.
- 2 June - 29 June: Convert various command-line utilities to use the output libraries.
- 30 June - 13 July: Test and fix bugs.
- 14 July - 27 July: Add XML support for libucl. Add any additional features if required.
- 28 July - 3 August: Do anything for which there was not enough time before. Sanitize the code.
- 4 August - 10 August: Test whether everything works and fix the bugs.
- 11 August - 17 August: Write documentation.