TCP Anomaly detector

Mentor: Andre Oppermann

Blog URL: http://planet.freebsd.org/rpaulo/

P4 depot: //depot/projects/soc2008/rpaulo-tcpad/

Snapshot: http://people.freebsd.org/~rpaulo/tcpad.tgz

Abstract

The Transmission Control Protocol is probably the most used protocol nowadays. Since it's inception, the protocol has been widely deployed and extended. Although there are tools like Wireshark that are able to find problems in a given TCP connection, researchers and operating system developers need a tool that will help them have a better view of the Internet. Such tool must monitor traffic in a 24/7 basis and not only snippets of it. By designing an anomaly detector are one step ahead of finding internetworking problems and operating system bugs.

How tcpad works

tcpad listens for TCP packets on the wire and builds a virtual TCP stack for each TCP endpoint. This means that, for example, if you run tcpad on a gateway, tcpad will monitor every connection between the hosts behind the gateway, the hosts reachable by the gateway (usually the Internet) and the connections to/from the gateway itself. After the initial packets, tcpad has built a virtual TCP stack for each endpoint. If you want to see how it looks like, check the error reporting section below. Along with this virtual TCP stack, tcpad monitors for abnormalities within the transmitted packets. For further inspection, tcpad keeps every TCP packet in memory and then dumps it into a pcap file. If you suspect a bug in a TCP stack or tcpad itself, you can boot tcpdump(1) or wireshark(1) and see the packet stream for yourself.

Building

Right now, you can build tcpad only with the help of FreeBSD make(1). This means that you can compile tcpad on FreeBSD itself or Mac OS X with the bsdmake(1) utility. The building process is trivial, just type "make" on the tcpad directory. If you find any errors, please contact me.

Error reporting

Whenever tcpad detects a strange condition, a dumpfile is created with the status of each virtual TCP stack. For example:

tcpad (TCP Anomaly Detector) version 0.1

Related pcap file:      dumpfiles/dump000.cap
Error message:          ACK with no known state

                |   Local endpoint   |   Remote endpoint  |
IP:             |           10.0.1.2 |      216.34.181.48 |
TCP Port:       |              49769 |                 80 |

FSM state:      |                  5 |                  9 |
ISS:            |         1862966176 |         2244299664 |
SND.UNA:        |         1862967000 |         2244393023 |
SND.MAX:        |                  0 |                  0 |
SND.NXT:        |         1862967000 |         2244393023 |
SND.UP:         |                  0 |                  0 |
SND.WL1:        |         2244391615 |         2244299664 |
SND.WL2:        |         1862967000 |         1862966177 |
SND.WND:        |               1264 |               4260 |
IRS:            |         2244299664 |         1862966176 |
RCV.NXT:        |         2244299665 |         1862966177 |
RCV.WND:        |                133 |              65535 |
RCV.UP:         |                  0 |                  0 |

RFC 1323 variables:
SND.SCALE:      |                  3 |                  5 |
RCV.SCALE:      |                  5 |                  3 |

Along with the related pcap file:

% tcpdump -nr dump000.cap
reading from file dump000.cap, link-type EN10MB (Ethernet)
15:59:13.509078 IP 10.0.1.2.49769 > 216.34.181.48.80: SWE 1862966176:1862966176(0) win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 561918935 0,sackOK,eol>
15:59:13.860143 IP 216.34.181.48.80 > 10.0.1.2.49769: S 2244299664:2244299664(0) ack 1862966177 win 4260 <mss 1420,nop,wscale 5,nop,nop,timestamp 1089747507 561918935,sackOK,eol>
15:59:13.860806 IP 10.0.1.2.49769 > 216.34.181.48.80: . ack 1 win 65535 <nop,nop,timestamp 561918939 1089747507>
...
(continued)

Further work

Reduce the number of false positives
TCP Timestamp option
TCP SACK validation
TCP ECN validation