2015-06-16 Bufferbloat solved with Vyos

I have been interested in the bufferbloat problem since it first came to my attention via a Slashdot post that publicized Jim Gettys' blog. I eventually found Mr. Brouer's masters thesis. Communications of the ACM also has a very good article by Jim Gettys.

I wish to thank nuclearcat for http://www.nuclearcat.com/mediawiki/index.php/U32_tips_tricks, since that page contained (and linked to) http://ace-host.stuart.id.au/russell/files/tc/doc/cls_u32.txt which held many of the keys to making this work. In particular the discussion of hash tables, linking, and the header offsets was important.

Now that Vyos "helium" is available with a Linux 3.13 kernel, the fq_codel queueing discipline can be used to solve many bufferbloat issues. The nightly "lithium" builds contain my patches that allow fq_codel to be used via the native Vyos configuration system.

In the following example, the dsl link is claimed by the vendor to have a 768Kbps upstream. However, the dsl "modem" (Westel c90-610015) has excessive internal buffers, and if we feed it data at 768Kbps, the buffers in that device slowly fill up, and our latency grows. If we cap the bandwidth in the Vyos at 750Kbps, the Vyos is the bottleneck, and the buffers in the Westel device are always empty.

The "match tiny ip small" and "match tiny6 ipv6 small" configuration nodes generate Linux traffic control filters to match syn/ack and small packets.

interfaces {
    ethernet eth1 {
        address dhcp
        description "dsl link"
         traffic-policy {
            out myshaper
        }
    }
}
traffic-policy {
    shaper myshaper {
        bandwidth 750kbit
        class 2 {
            bandwidth 30%
            burst 2kb
            ceiling 100%
            description "syn ack bufferbloat"
            match tiny4 {
                ip {
                    small
                }
            }
            match tiny6 {
                ipv6 {
                    small
                }
            }
            queue-type fq-codel
        }
        class 10 {
            bandwidth 15%
            burst 2kb
            ceiling 100%
            description "voip rtp traffic"
            match voip-rtp {
                ip {
                    dscp 46
                }
            }
            queue-type fq-codel
        }
        class 20 {
            bandwidth 5%
            burst 2kb
            ceiling 100%
            description "voip sip traffic"
            match voip-sip {
                ip {
                    dscp 26
                }
            }
            queue-type fq-codel
        }
        default {
            bandwidth 50%
            burst 2kb
            ceiling 100%
            queue-type fq-codel
        }
        description "output policy"
    }
}

The resulting traffic control filters are:

tc filter show dev eth1
filter parent 1: protocol ip pref 1 u32
filter parent 1: protocol ip pref 1 u32 fh 2: ht divisor 1
filter parent 1: protocol ip pref 1 u32 fh 2::800 order 2048 key ht 2 bkt 0 flowid 1:2
  match 00020000/00020000 at 12
filter parent 1: protocol ip pref 1 u32 fh 2::801 order 2049 key ht 2 bkt 0 flowid 1:2
  match 00100000/00100000 at 12
filter parent 1: protocol ip pref 1 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:2
  match 00010000/00ff0000 at 8
filter parent 1: protocol ip pref 1 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid 1:2
  match 00060000/00ff0000 at 8
  match 00100000/00ffff00 at 0
filter parent 1: protocol ip pref 1 u32 fh 800::802 order 2050 key ht 800 bkt 0 link 2:
  match 00060000/00ff0000 at 8
  match 00000000/0000ff80 at 0
  match 00000000/00003fff at 4
    offset 0f00>>6 at 0  eat
filter parent 1: protocol ipv6 pref 2 u32
filter parent 1: protocol ipv6 pref 2 u32 fh 3: ht divisor 1
filter parent 1: protocol ipv6 pref 2 u32 fh 3::800 order 2048 key ht 3 bkt 0 flowid 1:2
  match 00020000/00020000 at 12
filter parent 1: protocol ipv6 pref 2 u32 fh 3::801 order 2049 key ht 3 bkt 0 flowid 1:2
  match 00100000/00100000 at 12
filter parent 1: protocol ipv6 pref 2 u32 fh 801: ht divisor 1
filter parent 1: protocol ipv6 pref 2 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1:2
  match 00003a00/0000ff00 at 4
filter parent 1: protocol ipv6 pref 2 u32 fh 801::801 order 2049 key ht 801 bkt 0 flowid 1:2
  match 00000600/ff80ff00 at 4
  match 01000000/0ff00000 at 0
filter parent 1: protocol ipv6 pref 2 u32 fh 801::802 order 2050 key ht 801 bkt 0 link 3:
  match 00000600/ffc0ff00 at 4
    offset plus 40  eat
filter parent 1: protocol all pref 3 u32
filter parent 1: protocol all pref 3 u32 fh 802: ht divisor 1
filter parent 1: protocol all pref 3 u32 fh 802::800 order 2048 key ht 802 bkt 0 flowid 1:a
  match 00b80000/00ff0000 at 0
filter parent 1: protocol all pref 4 u32
filter parent 1: protocol all pref 4 u32 fh 803: ht divisor 1
filter parent 1: protocol all pref 4 u32 fh 803::800 order 2048 key ht 803 bkt 0 flowid 1:14
  match 00680000/00ff0000 at 0