Tweaks for high-bandwidth tinc

[Guus Sliepen]

On Sat, Oct 23, 2010 at 11:19:30AM -0500, Brandon Black wrote:

> I've been using tinc to do some high bandwidth VPNs [...]

How high is the bandwidth exactly?

> 1) In tinc-up, users will want to adjust the txqueuelen of the tunnel
> device as appropriate with ifconfig.  The default it 500, which
> resulted in tons of overruns for me.  After trying 2000 for a while
> and still seeing overruns, I went with a value of 10000, which seems
> to be working well.

Is the traffic very bursty? Normally, any TCP traffic inside the tunnel should
adapt its bandwidth to that available, this means it also should not become
higher than tinc can handle. So if traffic is very regular, one wouldn't expect
many packets being queued in the tun device...

> 3) When looking at txqueuelen issues, I stumbled on a tun device flag
> IFF_ONE_QUEUE (and the corresponding TUN_ONE_QUEUE) in the Linux tun
> source.  Some other VPN software out there (openvpn for example) seems
> to set this on all tun devices.  Whether it helps or hurts in any
> given situation on a modern Linux host is unclear to me just from
> reading the source.  It's possible this might warrant an experimental
> config setting for Linux, or at least some more research and testing.

Interesting. I have seen that flag but never looked into what it does. It can
be made configurable.

> 4) tinc.conf needs some settings for tuning SO_SNDBUF and SO_RCVBUF on
> the tunnel traffic sockets.  I was getting tons of "Lost XXX packets
> from YYY" syslog messages initially before I worked around this by
> upping the default buffer sizes via
> /proc/sys/net/core/[rw]mem_default.  This raises the default for all
> sockets opened on the host though, which is pretty heavy-handed.
> Adding this seems like a no-brainer to me.

Ok.

> 5) Even with all of the above in place, I'm still getting some spam
> like this on the app servers:
> 
> Lost 133 packets from lb
[...]
> I think maybe the late-packet bitmap could be tuned to handle it
> better.  If I'm reading the code correctly, it only allows for a
> 128-packet window for tracking late/replay, and packets outside of
> that window are simply dropped, correct?  I'd rather expand this
> window (via a config parameter) and let the upper layers (the TCP
> sessions riding above the VPN) deal with the ordering issues than have
> the VPN drop the packets for relative tardiness.

It can currently be changed at compile time, but it can be made run-time
configurable.

> 6) While looking at the late-packet bitmap, I also noticed the packet
> seqs are 32-bit, and when they rollover they force a re-key, which in
> turn will restart the sequences at zero.  It's not completely clear to
> me yet whether this is an issue or not, but I'm probably rolling these
> counters over as fast as once a day in my worst cases (lots of small
> packets).

If it is once a day it is not a big problem; the default key renewal is once
per hour.

> Seem like this has the potential to cause a traffic hiccup
> at each rollover for packets that are already queued up with
> overflowed sequence numbers when the sudden key change and sequence
> reset happens.  Could we handle the rollover more gracefully (e.g.
> re-key well ahead of the sequence exhaustion and allow the remaining
> sequence numbers using the old keying to still flow through the
> buffers before the rollover happens), and/or extend the sequence to a
> 64-bit counter?

The size of the counter cannot be changed in tinc 1.x, but in principle the key
rollover could be handled more gracefully by keeping a copy of both the old and
new key around, and looking at the sequence number to decide which key to use
to decrypt packets.

> I'll write up some patches for 1.0.X for some of these issues if
> nobody beats me to it, but I wanted to see if anyone had more insight
> (or objections) first.

Go ahead, patches are welcome!

-- 
Met vriendelijke groet / with kind regards,
     Guus Sliepen <guus at tinc-vpn.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://www.tinc-vpn.org/pipermail/tinc-devel/attachments/20101024/0ce51ad1/attachment.pgp>