Tinc over 3g problems?

[Thomas Arvanitis]

Hello,

I am experiencing some weird problems in a setup with tinc where
communication between the 'server' and the 'clients' occur over 3g
connections.

Let me describe briefly the setup:

- The server, on a public IP, runs tinc 1.0.8, in router mode, and the
whole setup uses one VPN network. All client's VPN addresses are on
the same subnet, and each client has a seperate local private LAN
'behind' it. The server's tinc.conf:

Name = Pegasus
PrivateKeyFile = /etc/tinc/solarvpn/rsa_key.priv
Device = /dev/net/tun
Interface = vpn
TCPOnly = yes

Host file (Pegasus VPN i/f is 192.168.10.1):

Address = ccc.bbb.yyy.xx (Public)
Subnet = 192.168.10.0/24
Port = 657
-----BEGIN RSA PUBLIC KEY-----
....
-----END RSA PUBLIC KEY-----

- Every client in the field runs tinc 1.0.8, as well, and is behind
the NAT of the 3g mobile operator (his whole network being private)
and the NAT of the local router/modem at the remote location. A
client's tinc.conf:

Name = thiva
PrivateKeyFile = /etc/tinc/solarvpn/rsa_key.priv
Device = /dev/net/tun
Interface = vpn
TCPOnly = yes
ConnectTo = Pegasus
PingInterval = 120
PingTimeout = 60
MaxTimeout = 120

Host file:

Subnet = 192.168.10.5/32
Subnet = 192.168.102.0/24
-----BEGIN RSA PUBLIC KEY-----
....
-----END RSA PUBLIC KEY-----


We use TCPOnly, since there is no way to setup port forwarding in the
clients. That was the reason we ended up using VPNs from all our hosts
in the field, since there was no way for the server (Pegasus) to reach
them.

Here is a small graph of the network path for one client:

-----------        ---------------------       -----------     -------
| Pegasus |--------|Operator NAT router|-------|3g router|-----|thiva|
-----------        ---------------------       -----------     -------
                          NAT                      NAT

There are many clients in the field, and we use more than one 3g
provider.

All clients do connect and ip connectivity and setup works fine. When
the tinc connection is up, everything works fine. But that's where the
problem is. Connections are dropped frequently (even up to 10 times
per hour) and are not established deterministically. My initial
thoughts was that there is a problem with the 3g connection going up
and down. But I used a script on the client hosts to monitor
connection availability (by pinging and performing a wget of small
files from a public web server) and I was surprised to see that it is
very rare that the 3g router does not have a connection (once or twice
in 24h hours, while on those hours the tinc VPN had gone up and down
25-30 times!).

I tried to monitor packets arriving on Pegasus, but I wasn't able to
make much sense out of it. Analyzing the traffic with wireshark shows
a lot of packet re-ordering and retransmission though. And from what I
show there were times that a tinc client was trying to establish the
VPN with pegasus but that wasn't always successful!

I am already using, on a different server a similar setup over
standard leased lines and adsl lines, but never experienced anything
similar.

Is it possible that something related to the well known latency, rtt
times and buffering issues of 3g, leads tinc to such a behavior?

Any help/hint would be greatly appreciated.

Thanks in advance,
-Thomas.