• TTYRedesign

Introduction

Like almost any other UNIX-like operating system, FreeBSD has a TTY layer. A TTY layer allows you to do a lot of things, but in most situations it's used by users to run commands in a shell. FreeBSD's TTY layer works like it should, but it has a lot of room for improvements:

• The TTY layer still uses the Giant Lock (see SMPTODO)

• The buffering mechanisms used inside the TTY objects could be a lot smarter (less memory overhead, less copying of data)
• The layer lacks any form of abstraction, which makes it really hard to add any new features to this layer
• Hotplugging isn't handled properly

I (EdSchouten) am going to work on the TTY layer for my internship the first half year of 2008. I am willing to work on it even further after I've graduated. My work will be stored in the //depot/user/ed/mpsafetty branch in Perforce.

Design notes

This chapter will describe the differences between the old and the new TTY layer. It's not a formal guide (yet), but it should make it easier to port existing drivers to the new code.

First off, I'll summarize the major differences between the old and the new TTY layer, so we get a better idea why the API is like it is:

• TTY drivers no longer interact with devfs and character device routines directly. The TTY layer now resides between devfs and the device driver. This means you must no longer create device nodes with routes patches to the TTY layer directly.
• By default, each TTY has its own mutex to protect its internal datastructures. This TTY can also be used by the device driver, though it may be smarted to split it up and use a mutex to protect the driver's data. To ease migration and implementation of various drivers, a driver can force the TTY layer to use a specific mutex, instead of a per-TTY one.

• Instead of patching up the routines inside the tty structure, the new TTY layer has a per-class structure, similar to cdevsw. This means you cannot patch routines inside the TTY layer after allocating it.

The basics: allocating TTY's

The first thing most device drivers can do, is allocate the actual TTY. The new TTY code uses a two-step procedure to allocate TTY's:

struct tty *tty_alloc(struct ttydevsw *, void *, struct mtx *);
void    tty_makedev(struct tty *, struct ucred *, const char *, ...) __printflike(3, 4);

They can be used like this:

struct tty *tp;

tp = tty_alloc(&my_device_class, pointer_i_want_to_store_inside_the_tty, NULL);
tty_makedev(tp, credentials, "z%d", 0);

This will create a device node called ttyz0 and maybe some nodes called cuaz0, ttyz0.lock, etc. There are a couple of things I haven't really described yet, so I will do this here:

The &my_device_class is a reference to a ttydevsw structure, which describes the class of devices we're dealing with. my_device_class could have looked like this:

static struct ttydevsw my_device_class = {
        .tsw_flags          = // binary flags such as TF_NOPREFIX (no "tty" prefix for device node names), TF_INITLOCK (create ".init" and ".lock" nodes) and TF_CALLOUT (create "cua" nodes).
        .tsw_open           = // function to notify the driver that the device is being opened
        .tsw_close          = // function to notify the driver that the device is being closed
        .tsw_outwakeup      = // function to notify the driver that data is available in the output buffers of the TTY
        .tsw_inwakeup       = // function to notify the driver that after the driver was unable to deliver (rint) data to the TTY, the TTY has now successfully made space so that the driver can rint() again (yes, i know, hard to describe, but it's somewhat the converse of outwakeup).
        .tsw_ioctl          = // may be used by the driver to add additional ioctl()'s to the device nodes (return ENOIOCTL on unimplemented ones)
        .tsw_param          = // device parameters changes (baud rate, flow control, etc)
        .tsw_modem          = // modem flags changed
        .tsw_mmap           = // XXX: only used by syscons: syscons allows you to mmap() a TTY to get a mapping to a screen buffer
        .tsw_free           = // we'll discuss this later: a callback to notify the driver that the TTY has been successfully remove from the system
};

(Please take a look at sys/ttydevsw.h for details)

A nice thing is that none of the functions need to be implemented. The first time you allocate a TTY using this device class, it patches all the NULL pointers in the structure to a default implementation, which does mostly nothing. open(), close(), inwakeup() and ioctl() do nothing, param() just enforces some default settings (hardcoding the baud rate at 38400, to enforce a sane buffer size), but some may cause a panic when called. It makes no sense not to implement a free() routine when your driver implements hotplugging, because the driver will need to know when the device unit number may be reclaimed.

Now the second argument of tty_alloc(): just like si_drv1 and si_drv2, the TTY structure has its own pointer space that can be used by the device driver. Because we don't allow access to the devfs device nodes, we will not toy around with si_drvX anymore. You can use tty_softc() to obtain the value from the TTY pointer.

The third argument: the mutex. In most cases you just want to pass NULL here. There are some reasons why you don't want to pass NULL:

• If you pass &Giant, the TTY will be locked down using the Giant lock. This means the following:

  • All the ttydevsw routines will be called by the TTY layer holding Giant, just like the old TTY layer which used D_NEEDGIANT.

  • You can just call into the TTY layer without calling TTY layer locking routines first. If you hold Giant, you can just rint(), etc.
• Drivers like nmdm are a little complex to lock down: two TTY's talking to each other. To keep it simple right now, we just allocate one mutex to lock both of them.

But keep in mind: we could decide to remove the mutex-argument from tty_alloc() somewhere in the very far future. It is just a tool to make things a little more simple right now.

Now that we've discussed tty_alloc(), let's take a look at the tty_makedev(). This routine should be called once the driver is ready and it is safe to expose our TTY to the world. It may create one, but can create up to six device nodes in /dev, depending on the tsw_flags. The cred argument you're seeing: by default tty_makedev() creates TTY's with system-like privileges (root:wheel), but when credentials are passed, it does the following:

• Devices are created with ownership of the real user ID of the process.

• The group is set to tty.

• Permissions are set to 0640.
• Device nodes are protected from being opened from another jail.

This should mainly be used by device drivers where we can say: they are owned by the user who wants them. It should be used by pts(4) and nmdm(4).

Interacting with TTY's

Now that you've got yourself a TTY and know about the hooks it offers - how do I extract data from it? This is the basic list of functions you can use:

// indicate a change in modem events (carrier drop)
void    ttydisc_modem(struct tty *, int);
// is the TTY in an optimized state, allowing me to pass the entire buffer at once? similar to TF_CAN_BYPASS_L_RINT
#define ttydisc_can_bypass(tp) ((tp)->t_flags & TF_BYPASS)
// deliver a single byte of input to the TTY layer. the 3rd argument is a bitmask of the TRE_ definitions in sys/ttydisc.h
int     ttydisc_rint(struct tty *, char, int);
// deliver an entire buffer. will panic if ttydisc_can_bypass(tp) == 0. then you should deliver it to ttydisc_rint() with error data
size_t  ttydisc_rint_bypass(struct tty *, char *, size_t);
// this should be called after rint()'ing data! unlike the old implementation, we don't generate wakeups after each byte has been delivered. you have to do it manually!
void    ttydisc_rint_done(struct tty *);
// obtain output data from the TTY
size_t  ttydisc_getc(struct tty *, void *buf, size_t);
// do not use: `fast path' way to copy data directly to userspace. i guess only pts(4) will use this.
int     ttydisc_getc_uio(struct tty *, struct uio *);

(Please take a look at sys/ttydisc.h for details)

Locking

Locking is quite easy if you remember the following rules:

• Always hold the TTY lock when entering the ttydisc_*() routines.

• The lock is always held inside the ttydevsw routines.

Abandoning a TTY

The TTY layer uses two steps to remove a TTY:

• The driver calls tty_rel_gone() while holding the TTY lock. The lock is unlocked by the TTY layer. Do not touch the TTY after you've called this. (ACHTUNG!)

• When the TTY layer is finished evacuating all the threads and removing the device nodes from /dev, it calls tsw_free, passing only the pointer to the softc, which means you can now clean up your data and also reclaim the device unit number. You MUST NOT reuse the number before tsw_free is called, because this means we'll have two devices in /dev sharing the same name. If a private mutex was passed in with tty_alloc then it must remain valid until tsw_free is called.

Flow control

There is a big difference in the way the new TTY code handles flow control. Unlike the old layer where the driver just checks TS_* flags before poking around in the inq, the new TTY code uses a very simple mechanism:

• Just try to rint() as much data as possible.

• If this fails (check the return value), this means you've reached the high watermark. Your data has not been stored.

• This will trigger a flag inside the TTY layer, which means you cannot rint() until we're going below the watermark again.

• The TTY layer will call tsw_inwakeup() to notify you that you can now rint() again.

This may make older code a little harder to port, but it should make sense. An advantage of this model is that it is both suited for physical and virtual drivers. The pts(4) driver may need to actually implement blocking when writing too much data to a PTY master device. The old TTY layer didn't really offer a mechanism for this without patching the cdev routines.

Examples

Be sure to take a look at the really primitive drivers. I'll discuss the IA64 /sys/ia64/ia64/ssc.c driver here, which is only about 180 lines. Please refer to the original file. The following pieces of code are just the interesting snippets (which are (C) Doug Rabson).

The TTY device class

As you can see, the interesting part of the driver starts here:

static tsw_open_t       ssc_open;
static tsw_outwakeup_t  ssc_outwakeup;
static tsw_close_t      ssc_close;

static struct ttydevsw ssc_class = {
        .tsw_flags      = TF_NOPREFIX,
        .tsw_open       = ssc_open,
        .tsw_outwakeup  = ssc_outwakeup,
        .tsw_close      = ssc_close,
};

The SSC driver only implements three functions, namely the open() and close() routines to start and stop a timer, which polls the interface, because it does not seem to be interrupt driven. It also implements outwakeup(), which is used to trigger the driver to start transmitting data.

The actual TTY is allocated inside ssc_cnattach(), which is called by the low-level SYSINIT macro:

static void
ssc_cnattach(void *arg)
{
        struct tty *tp;

        tp = tty_alloc(&ssc_class, NULL, NULL);
        tty_makedev(tp, NULL, "ssccons");
}

SYSINIT(ssc_cnattach, SI_SUB_DRIVERS, SI_ORDER_ANY, ssc_cnattach, 0);

The three routines referred by the ttydevsw look like this:

static int
ssc_open(struct tty *tp)
{

        polltime = hz / SSC_POLL_HZ;
        if (polltime < 1)
                polltime = 1;
        ssc_timeouthandle = timeout(ssc_timeout, tp, polltime);

        return (0);
}

static void
ssc_close(struct tty *tp)
{

        untimeout(ssc_timeout, tp, ssc_timeouthandle);
}

static void
ssc_outwakeup(struct tty *tp)
{
        char buf[128];
        size_t len, c;

        for (;;) {
                len = ttydisc_getc(tp, buf, sizeof buf);
                if (len == 0)
                        break;

                c = 0;
                while (len-- > 0)
                        ssc_cnputc(NULL, buf[c++]);
        }
}

The open() and close() routines are simple to understand: depending on hz, we calculate the interval for a timeout. The outwakeup() routine is where the interesting bits are: the driver just obtains the data from the TTY and writes it byte by byte to to the physical hardware.

The ssc_timeout() routine is called to poll the hardware and store the input inside the TTY:

static void
ssc_timeout(void *v)
{
        struct tty *tp = v;
        int c;

        tty_lock(tp);
        while ((c = ssc_cngetc(NULL)) != -1)
                ttydisc_rint(tp, c, 0);
        ttydisc_rint_done(tp);
        tty_unlock(tp);

        ssc_timeouthandle = timeout(ssc_timeout, tp, polltime);
}

Because ssc_cngetc() just returns the characters one by one, we need to rint() them. When we're done, we must not forget to call rint_done() to wake up any blocked threads. Please note that it's not bad to call ttydisc_rint_done() even if you weren't able to rint() any data, because the TTY layer automatically filters spurious wakeups.

Questions?

Send me an email or poke me on IRC!

Designing the new TTY Hooks layer

We need some kind of mechanism to make in-kernel TTY consumers work again. Right now we've got 5 broken drivers (snp, ppp, sl, ng_h4, ht_tty), which we should somehow fix. Right now the snp(4) driver works like this. The function names used inside the TTY layer refer to the new TTY layer.

                                             | old snp(4) input
        +------------------------------------v+
<-------|             INPUT PROCESSING        |<------------------
 read() | ttydisc_read()       ttydisc_rint() | driver_intr()
        |                                     |
        |               TTY LAYER             |
        |                                     |
------->|            OUTPUT PROCESSING        |------------------>
write() | ttydisc_write()      ttydisc_getc() | driver_outwakeup()
        +|------------------------------------+
         v old snp(4) output

I don't entirely agree with this design, because things probably go bad when you turn on the various output processing options. You'd be better off switching your terminal to some kind of raw mode and let snp(4) copy all the data that gets getc()'d by the driver. When you want to inject data, you want to do it at the same spot. That's why I propose the following design:

                                            |^ Capture/inject hooks on the input path
        +-----------------------------------v|+
<-------|             INPUT PROCESSING        |<------------------
 read() | ttydisc_read()       ttydisc_rint() | driver_intr()
        |                                     |
        |               TTY LAYER             |
        |                                     |
------->|            OUTPUT PROCESSING        |------------------>
write() | ttydisc_write()      ttydisc_getc() | driver_outwakeup()
        +----------------------------------^|-+
                                           |v Capture/inject hooks on the input path

So basically we've got two different categories of hooks:

• Drivers that just want to sniff (snoop). It just wants to receive a copy of the data that is being handed to the device driver.
• Drivers that want complete control of the device I/O (PPP, Netgraph, etc).

XXX: propose API?

CategoryHistorical