Yet another tutorial to create a private network


#1

Like many of you, I needed to build a private network between several instances that don’t have public IP.
Several posts on this forum suggest to use vpn software like openvpn or tinc.
Even if those solutions are all relevants, I needed something more low-level without encryption overhead (eg. faster) and without limitation (eg. multicast support).

In this howto, I will explain how to use GRE tunnels to achieve this goal.

Architecture

One instance with a public IP will act as a switch and as a NAT gateway. One GRE tunnel per private intance will be set up. GRE endpoints on public instance will be bound to a bridge that will act as a switch.

Requirements

  • one instance with a public IP
  • several instances with private IP (named private instance in this document)
  • tested on Debian and Alpine Linux but should works on any modern linux distro.

You probably need to temporarily bind public ip on private instances to install dependencies and set root password.
In this example, the private subnet is 172.16.42.0/24 and the gateway is 172.16.42.254

Note about GRE tunnel on linux

GRE (Generic Routing Encapsulation) tunnel are really easy to set up but didn’t have any security mechanism like encryption or strong authentication.
GRE tunnel can operate at OSI layer 3 or layer 2. As we want switching between GRE tunnels, we need to operate at layer 2 and under linux, layer 2 GRE tunnels are named ‘gretap’.

‘public instance’ configuration

On this server, we need to create a bridge and create as many GRE endpoint as private instance.

Brigde

To create bridge interface, you can use brctl but in this example we use iproute2.

ip link add name br0 type bridge
ip addr add 172.16.42.254/24 dev br0
ip link set br0 up

GRE tunnels

For every private instance, create a new GRE endpoint:

ip link add tun-X type gretap remote 10.X.X.X ttl 64
ip link set dev tun-X up

10.X.X.X is the private address of the private instance. In tun-X replace the X by anything you want (ex. tun-1).

Then, add tun-X into the bridge :

ip link set dev tun-X master br0

As scaleway private addresses change when instances are restarted, you should use ‘Private DNS’ address to get instance private IP. The following script will do all the magic :

#!/bin/bash

REMOTE_ADDR="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX92.priv.cloud.scaleway.com
XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX93.priv.cloud.scaleway.com"

ip link add name br0 type bridge
ip addr add 172.16.42.254/24 dev br0
ip link set br0 up

for ADDR in $REMOTE_ADDR; do
        IP=$(getent hosts $ADDR | awk '{ print $1 }')
        IP_SANITIZED=$(echo $IP | sed 's/\./-/g')
        TUN_DEV="tun-"$IP_SANITIZED

        ip link add $TUN_DEV type gretap remote $IP ttl 64
        ip link set dev $TUN_DEV up
        ip link set dev $TUN_DEV master br0
done

‘private instance’ configuration

One each private instance, create the GRE endpoint :

ip link add tun0 type gretap remote 10.Y.Y.Y ttl 64
ip addr add 172.16.42.X/24 dev tun0
ip link set tun0 up

10.Y.Y.Y is the private IP of the public instance.
172.16.42.X/24 is obviously the private IP inside our new private network.

Now you can ping any 172.16.42.X/24 addresses and have a fully functionnal private LAN.

NAT

If we want our private instances access to internet through the public instance, we need to change the default route of all our private instance.

But before, we need to enable NAT on public instance :

Edit /etc/sysctl.conf like :
net.ipv4.ip_forward=1

Load the new configuration with sysctl -p

Then, create iptables masquerade (quick and dirty way) :
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Now we can edit private instance routing table. This is the tricky part, take care to launch commands in the right order otherwise your are good to hard reboot your server.

1) default routing table should look like :
# ip route show
default via 10.X.X3.94 dev eth0
10.X.X3.94/31 dev eth0 proto kernel scope link src 10.X.X3.95
172.16.42.0/24 dev tun0 proto kernel scope link src 172.16.42.2

2) In order to keep GRE tunnel working, we have to create a specific route to our public instance (10.Y.Y6.89):
# ip route add 10.Y.Y6.89 via 10.X.X3.94
# ip show route
default via 10.X.X3.94 dev eth0
10.Y.Y6.89 via 10.X.X3.94 dev eth0
10.X.X3.94/31 dev eth0 proto kernel scope link src 10.X.X3.95
172.16.42.0/24 dev tun0 proto kernel scope link src 172.16.42.2

3) replace default route
# ip route change 0.0.0.0/0 via 172.16.42.254
# ip route show
default via 172.16.42.254 dev tun0
10.Y.Y6.89 via 10.X.X3.94 dev eth0
10.X.X3.94/31 dev eth0 proto kernel scope link src 10.X.X3.95
172.16.42.0/24 dev tun0 proto kernel scope link src 172.16.42.2

You now have a fast NATted private LAN !

Things to do and security warning

All configurations in this tutorial are volatile and won’t survive to a server reboot. You need to make this persistent using startup scripts but take care of scaleway private IP and the default gateway that will change if servers are restarted.

As previously told, GRE tunnels are not encrypted and thus are prone to be sniffed and inspected. I personnaly think that you don’t take so much risk by using GRE tunnels inside scaleway private network but if you are paranoid, use an encrypted vpn.

You can use iptables to filter GRE connections (see. --protocol 47).

And be aware that GRE tunnel MTU is 1462 against 1500 on regular ethernet networks. You should dig this way if you meet unexplained networking troubles.

Hope that can help someone :slight_smile:


#2

Without encryption overhead.

In untrusted networks, as is any public network, encryption is required.
ipsec (libreswan) performance is very good in that regard.


#3

@PierreEmile Thanks for the tutorial! Can you please share how you setup the startup script? Via init.d or cloud-init?

Thanks