SysconsUnicodeProject
For any reader not familiar with Unicode, reading SysconsUnicodeProject/UnicodeBasics is suggested.
This project aims to enhance Unicode support on FreeBSD operating system by implementing support for UTF-8 in the Syscons console driver. This is not about full support for Unicode-based applications (because has no impact on correctness of sorting text strings, for example). Currently system console driver deals with 8-bit character sets only.
News
23/08/2008: Project initiated by Alexander Churanov. See this message to the lists.
- 01/01/2009: Syscons terminal emulator replaced with new implementation by Ed Schouten, which has optional xterm and UTF-8 support.
- 29/01/2009: The implementation of unicode maps, that allow using existing fonts in UTF-8 mode was created by Alexander Churanov.
- 04/08/2009: Unicode support for the keyboard code is present in Perforce and will be committed to HEAD after it has been unfrozen.
TODO
Description |
Assigned to |
Status |
Make terminal emulator UTF-8 aware |
Ed |
Committed (01/01/2009) |
Implement unicode-to-8-bit mapper for existing fonts |
Alexander |
Implemented (29/01/2009) |
UTF-8 aware input layer, keymaps. |
Ed |
Semi-done |
UTF-8 aware font renderer |
Ed |
TBD |
Unicode fonts, not limited to 256 chars |
Alexander |
TBD, not scheduled |
Terminal emulator
The old terminal renderer used by Syscons is not suitable to use with UTF-8. Right now we use a terminal type called cons25. The disadvantages of cons25:
At least on FreeBSD, we use non-ASCII characters by default. The box drawing is done using characters above 0x7f. This means it is impossible to use UTF-8, because if someone runs a non-UTF-8 application or sets LC_CTYPE to, say, C, applications like dialog(1) would generate illegal output.
cons25 and UTF-8 is not something that exists right now and is proven to work. xterm and UTF-8 is known to Just Work (tm).
cons25 also has some other problems, not related to UTF-8. It is not supported by many operating systems and devices and lacks a lot of optimizations (scrolling regions).
This is why we should migrate to xterm at the same time. This adds a little more complexity to the terminal emulator (old VT100 character sets, keypad application mode, etc), but has its advantages.
As of 01/01/2009, the old Syscons emulator has been replaced by libteken. libteken is still configured to be a cons25 emulator, but already supports a lot more escape sequences. This means you should already be able to SSH to your Cisco/HP switch without problems.
If you want to try the new xterm + UTF-8 support, recompile your kernel with TEKEN_XTERM and TEKEN_UTF8 in your kernel configuration file. All non-ASCII characters will still be displayed as a yellow/red question mark. Be sure to replace cons25 with xterm in /etc/ttys!
Input layer
Right now the input layer only has 8-bit keymaps. This should be changed to Unicode. XXX: Extend.
Font rendering
Because Syscons (and the hardware!) is restricted to 8-bit fontmaps, we should add a Unicode-to-8-bit fontmap conversion. An advantage is that we should be able to display exactly 256 different characters now, instead of 240-250 (not sure). We could even map multiple Unicode characters to the same graphical character (German Eszett and Beta?).
We could consider writing only a very small amount of fonts and some kind of file format, describing which characters should be included in the 8-bit map. This means we can generate a lot of fontmaps. I guess it's a little more tricky to add support for full width characters, because these should consume two graphical characters. Maybe we can save some space by just rendering full-width ASCII characters as regular characters, followed by a space?
This is still very premature. WIP. TBD.
Integration
In order to make integration into the operating system as easy as possible, we should at least make UTF-8 support a compile-time switch. Users/developers may still want to migrate their key/fontmaps to the new format; POLA.
Testing
To verify that the project has reached its goals and met all requirements performing the following is necessary:
- performing all manual tests in /usr/src/tools/test/syscons
- verifying font correctness (for each added/modified font)
- verifying keyboard input map correctness (for each added/modified map)
- verifying font catalog completness
- verifying keyboard input map catalog completness