Packet size issue with direct UDP connections

[Chris Clements]

Have an interesting issue that's cropped up for us.  We have a switch based vpn setup with multiple endpoints, some publicly accessible, some behind hide NAT.  What we've noticed is that every machine can reach each other's VPN IP's up to a certain packet size.  All machines can successfully ping each other up to 297 bytes.  At 298 bytes and beyond, only "forwarded" connections succeed.  Specifically, a machine behind NAT connected directly to a public machine via UDP cannot ping it with greater than 297 bytes.  In this case ICMP pings are an example, but we see the same behavior with other services.

Doing packet dumps on the VPN interfaces, we see that the pings >297 from direct connections do indeed arrive at the destination, but are dropped by the destination tinc interface as "ethertype unknown".  Again, forwarded connections show no issues at all.  If directly connected public endpoints are reconfigured to instead forward through other endpoints, they exhibit no issues with packet sizes.  There are no errors reported related to this we can identify at any debug level.

Examining the differences between the 297 and 298 ping packets:

Server (public) tinc mac addr: 72:49:ef:9b:99:b9
vm (natted) tinc mac addr: 26:ae:0d:bb:12:01

Works:
server pinging the vm with 'ping -s 297 12.12.0.55'
--------------------------------------------------------------------
   vm mac     | gw mac         | start of ip header
26ae 0dbb 1201 | 7249 ef9b 99b9 | 0800 4500


Doesn't Work:                                                                                         
server pinging the vm with 'ping -s 298 12.12.0.55'
--------------------------------------------------------------------
 ?  | vm mac         | gw mac         | start of ip header
0024 | 26ae 0dbb 1201 | 7249 ef9b 99b9 | 0800
====================================================================


So it appears that extra data "0024" is being pre-pended to packets larger than 297 bytes and this offset is why the kernel drops it as an unknown ethertype.


Any thoughts on this?  We are using tinc version 1.1pre11-143-gbfe231b at all points. We can share more config info if it would be helpful.  We will only have access to this env for another week if there are any troubleshooting suggestions.


Chris