Bridged nodes sharing local IP to be used as gateway for LXC

[Guus Sliepen]

On Tue, Mar 12, 2013 at 12:21:33AM +0100, Mike Machuidel wrote:

> I am running multiple nodes each having multiple LXC containers. On
> those nodes I am using Tinc to share the subnet 10.20.0.0. The LXC
> containers may migrate between nodes and should keep the same
> configuration.
> On all nodes traffic from the external interfaces (connected to the
> internet) is forwarded to the bridge interfaces and masqueraded. The
> LXC containers are using 10.20.0.1 as their default gateway which
> should stay the same independent of the node they are running on.

[...]

> on Node A:
> 
>   ip addr add 10.20.0.1/16 dev br0   # using same IP on node as node B
> so the LXC containers can use same gateway IP
> 
> on Node B:
> 
>   ip addr add 10.20.0.1/16 dev br0   # using same IP on node as node A
> so the LXC containers can use same gateway IP

[...]

> meaning that both nodes will share the same IP which the LXC container
> can use as their default gateway?
> 
> To be honest I already tried this configuration. It ran well for about
> 2 days till I noticed that sometimes LXC containers on Node B were
> using the default gateway on Node A and the other way around.

Well if you have two hosts with the same IP addresses on a network, the one
which responds first to an ARP request will win. The MAC address discovered
via ARP will be cached, so if a LXC container migrates to another node it will
not automatically change to the local gateway. And even if no containers
migrate, it could be that the local host has a higher load than the remote
node, and the remote node's ARP replies will arrive before the local node's.

You could use ebtables to filter ARP requests/replies going to/coming from the
remote gateway. There might be other solutions, but I cannot think of any right
now.

> Is it possible to force the 10.20.0.1 on the local interface to be
> used instead of the traffic being bridged over to the other side? Like
> setting some metrics?

I think the problem happens on the MAC layer, not on the IP layer, in which
case metrics will not help much.

-- 
Met vriendelijke groet / with kind regards,
     Guus Sliepen <guus at tinc-vpn.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20130314/0f9d322f/attachment.pgp>