How the sectored antenna support works
The AR5212 era NICs include something called "sectored antenna." Here, four antenna control bits (ANTA->ANTD) from the MAC control an external antenna switch. These four antenna control bits can be directly controlled for transmitting each packet.
Before sectored antennas
The non-sectored support includes only two antenna configurations. There's multiple external control bits for these two antenna configurations, but various parts of the MAC (eg the "last received antenna" field in the keycache) only allows for two antennas. These features are designed for "automatic" behaviour - ie, where the MAC implements antenna control itself, separate from the software.
What is it?
The sectored antenna support allows for up to 16 external antennas. The normal use would be a single omni-directional antenna for receive, and then multiple antennas - either all omni-directional, or all narrow field antennas, or a combination of the two. The idea is to try and find an antenna that gives "better" behaviour when transmitting to a given client.
The antenna control interface and antenna arrangement is definitely up to the individual NIC and product vendor.
Using sectored antennas
If configured, the hardware will instead leave antenna selection up to the software.
- The DEFANT register controls the default receive antenna configuration
The 4 bit AntModeXmit field maps directly to the values of ANTA->ANTD on the MAC
- The "0000" value antenna should be an omni-directional antenna, to listen for received frames (which DEFANT can affect, as mentioned above)
If an RTS is sent, the hardware can optionally send it on the antenna configuration and listen for the CTS response on the same antenna configuration, or it can listen to it on the default (omni-directional) configuration.
Here the hardware doesn't try to track the last recieved antenna at all. It relies on the software stack to determine the best transmit antenna configuration. It's likely the rate control code would be involved in tracking the transmission statistics and choosing the best configuration for a given node and rate.