Double IP same Host

[alfred-stokespace]

/etc/nebula/nebula -version
Version: 1.6.1

I have an odd need to have a network where some physical hosts have more than one overly network ip (yup two ips for one host). I found this old issue #534 referencing this idea and tried it myself. I ran into an issue and maybe a solution but I'd like some clarity on why my solution "worked".

In my network I have a lighthouse host and 5 participant hosts
Lets ignore 4 of those participant hosts because they are all standard 1 host -to- 1 nebula ip.
YES! :) I have a cert & key file for the additional ip that I expect to double up on host-5 (hah! plus the initial ip's cert & key)

Lighthouse looks like ...

pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/host.crt
  key: /etc/nebula/host.key

static_host_map:
  "172.20.0.1": ["10.100.3.67:4242"]

lighthouse:
  am_lighthouse: true

listen:
  host: 0.0.0.0
  port: 4242

punchy:
  # Continues to punch inbound/outbound at a regular interval to avoid expiration of firewall nat mappings
  punch: true

relay:
  am_relay: false
  use_relays: true

# Configure the private interface. Note: addr is baked into the nebula certificate
tun:
  disabled: false
  dev: nebula1
  drop_local_broadcast: false
  drop_multicast: false
  tx_queue: 500
  mtu: 1300

logging:
  # panic, fatal, error, warning, info, or debug. Default is info
  level: info
  # json or text formats currently available. Default is text
  format: text

firewall:
  outbound_action: drop
  inbound_action: drop

  conntrack:
    tcp_timeout: 12m
    udp_timeout: 3m
    default_timeout: 10m

  outbound:
    # Allow all outbound traffic from this node
    - port: any
      proto: any
      host: any

  inbound:
    # Allow icmp between any nebula hosts
    - port: any
      proto: any
      host: any

Special host 5 (remember, we don't care about 1-4 since they are normal) looks like ...

config 1 ( signed for use of 172.20.0.50 )

pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/host.crt
  key: /etc/nebula/host.key

static_host_map:
  "172.20.0.1": ["10.100.3.67:4242"]

lighthouse:
  am_lighthouse: false
  interval: 60
  hosts:
    - '172.20.0.1'

listen:
  host: 0.0.0.0
  port: 4242

punchy:
  # Continues to punch inbound/outbound at a regular interval to avoid expiration of firewall nat mappings
  punch: true

relay:
  am_relay: false
  use_relays: true

# Configure the private interface. Note: addr is baked into the nebula certificate
tun:
  disabled: false
  dev: nebula1
  drop_local_broadcast: false
  drop_multicast: false
  tx_queue: 500
  mtu: 1300

logging:
  # panic, fatal, error, warning, info, or debug. Default is info
  level: info
  # json or text formats currently available. Default is text
  format: text

firewall:
  outbound_action: drop
  inbound_action: drop

  conntrack:
    tcp_timeout: 12m
    udp_timeout: 3m
    default_timeout: 10m

  outbound:
    # Allow all outbound traffic from this node
    - port: any
      proto: any
      host: any

  inbound:
    # Allow icmp between any nebula hosts
    - port: any
      proto: any
      host: any

and now for config 2 ( signed for use of 172.20.0.51 )

pki:
  ca: /etc/nebula/ca.crt
  cert: /var/tmp/crt
  key: /var/tmp/key

static_host_map:
  "172.20.0.1": ["10.100.3.67:4242"]

lighthouse:
  am_lighthouse: false
  interval: 60
  hosts:
    - '172.20.0.1'

listen:
  host: 0.0.0.0
  port: 4243

punchy:
  # Continues to punch inbound/outbound at a regular interval to avoid expiration of firewall nat mappings
  punch: true

relay:
  am_relay: false
  use_relays: true

# Configure the private interface. Note: addr is baked into the nebula certificate
tun:
  disabled: false
  dev: nebula2
  drop_local_broadcast: false
  drop_multicast: false
  tx_queue: 500
  mtu: 1300

logging:
  # panic, fatal, error, warning, info, or debug. Default is info
  level: info
  # json or text formats currently available. Default is text
  format: text

firewall:
  outbound_action: drop
  inbound_action: drop

  conntrack:
    tcp_timeout: 12m
    udp_timeout: 3m
    default_timeout: 10m

  outbound:
    # Allow all outbound traffic from this node
    - port: any
      proto: any
      host: any

  inbound:
    # Allow icmp between any nebula hosts
    - port: any
      proto: any
      host: any

and here is the diff between conf1 and conf2

root@host5:/var/tmp# diff config.yaml /etc/nebula/config.yaml
3,4c3,4
<   cert: /var/tmp/crt
<   key: /var/tmp/key
---
>   cert: /etc/nebula/host.crt
>   key: /etc/nebula/host.key
17c17
<   port: 4243
---
>   port: 4242
30c30
<   dev: nebula2
---
>   dev: nebula1 

Behavior with above config...

As soon as nebula2 (ie. 172.20.0.51) is started; pings from lighthouse to 172.20.0.50 stop, but pings to 51 work!

nebula2 starting...

/etc/nebula/nebula -config /var/tmp/config.yaml
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:outgoing endPort:0 groups:[] host:any ip: proto:0 startPort:0]"
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:incoming endPort:0 groups:[] host:any ip: proto:0 startPort:0]"
INFO[0000] Firewall started                              firewallHash=21716b47a7a140e448077fe66c31b4b42f232e996818d7dd1c6c4991e066dbdb
INFO[0000] Main HostMap created                          network=172.20.0.51/24 preferredRanges="[]"
INFO[0000] UDP hole punching enabled
INFO[0000] Loaded send_recv_error config                 sendRecvError=always
INFO[0000] Nebula interface is active                    build=1.6.1 interface=nebula2 network=172.20.0.51/24 udpAddr="0.0.0.0:4243"
INFO[0000] Handshake message sent                        handshake="map[stage:1 style:ix_psk0]" initiatorIndex=3193988034 udpAddrs="[10.100.3.67:4242]" vpnIp=172.20.0.1
INFO[0000] Handshake message received                    certName=lighthouse durationNs=5076643 fingerprint=ecbd0ac7d4d7b004f5c31fc5b2e0266daeed8a26f0c964ca3ec01ae697d56b89 handshake="map[stage:2 style:ix_psk0]" initiatorIndex=3193988034 issuer=745f1f9b9181012a4d4ba5a27c43dc68de6b09737a168a179df3ac9a41e9831e remoteIndex=3193988034 responderIndex=3912406011 sentCachedPackets=1 udpAddr="10.100.3.67:4242" vpnIp=172.20.0.1

and now from Lighthouse we see...

ping 172.20.0.50 -c 2 || ping 172.20.0.51
PING 172.20.0.50 (172.20.0.50) 56(84) bytes of data.
^C
--- 172.20.0.50 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1030ms

PING 172.20.0.51 (172.20.0.51) 56(84) bytes of data.
64 bytes from 172.20.0.51: icmp_seq=1 ttl=64 time=0.885 ms
64 bytes from 172.20.0.51: icmp_seq=2 ttl=64 time=0.493 ms
64 bytes from 172.20.0.51: icmp_seq=3 ttl=64 time=0.585 ms

if I kill nebula2 i have to also restart nebula1 in order to get ping back on nebula1.

What I discovered ...

if I disable the tun: on nebula2 config and start both nebula1 and 2, I'm able to ping both from the lighthouse host

ping 172.20.0.50 -c 2 && ping 172.20.0.51 -c 2
PING 172.20.0.50 (172.20.0.50) 56(84) bytes of data.
64 bytes from 172.20.0.50: icmp_seq=1 ttl=64 time=0.471 ms
64 bytes from 172.20.0.50: icmp_seq=2 ttl=64 time=0.508 ms

--- 172.20.0.50 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1032ms
rtt min/avg/max/mdev = 0.471/0.489/0.508/0.018 ms
PING 172.20.0.51 (172.20.0.51) 56(84) bytes of data.
64 bytes from 172.20.0.51: icmp_seq=1 ttl=64 time=2.08 ms
64 bytes from 172.20.0.51: icmp_seq=2 ttl=64 time=0.534 ms

for clarity... nebula1 has tun enabled and nebula2 has it disabled

Why? and was this intended? and how should I be doing this?

[NiceGuyIT]

Wow! You solved my problem with a similar setup. Since a lighthouse does not add itself to the DNS (issue #271), it was suggested to add a second Nebula instance to provide name resolution. This works except for the problem you describe. Whatever service is started last can be pinged and is reachable. If the lighthouse is restarted, it has an IP and is serviceable to all clients, but the node is unreachable. If the node is restarted, the lighthouse is no longer reachable and all clients disconnect, but the node is reachable. I have 2 lighthouses so discoverability is still there.

After re-reading issue #271, I noticed they mentioned that tun is disabled:

One configured as a lighthouse but with tun disabled

I had to make one more change on the node and that's to disable Prometheus stats because it couldn't bind to the IP:port. Checking the IPs shows why. The node (nebula1) with tun disabled does not have an IP while the lighthouse (nebula2) does have an IP. Switching the configs to enable tun for the node and disable tun for the lighthouse makes the IP for the node visible in ip addr.

What is the purpose of tun and why does it prevent 2 instances from running on the same host?

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 96:00:01:4e:64:35 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 96:00:01:4e:64:35 brd ff:ff:ff:ff:ff:ff
    inet *.*.*.*/32 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::9400:1ff:fe4e:6435/64 scope link
       valid_lft forever preferred_lft forever
60: nebula2: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1300 qdisc pfifo_fast state UNKNOWN group default qlen 500
    link/none
    inet 172.25.0.3/13 scope global nebula2
       valid_lft forever preferred_lft forever
    inet6 fe80::7df9:9d76:33a8:2ee6/64 scope link stable-privacy
       valid_lft forever preferred_lft forever

[NiceGuyIT]

One consequence of this is that while Nebula traffic flows, services on the host can only bind to the IP exposed to the host. I.e. services can bind only to the IP of the Nebula process that has tun enabled.

@alfred-stokespace If you intend to expose services on both IPs, you might need to find alternatives like the relay functionality, but that would imply both nodes (nebula processes on the host) are in the same Nebula network.

[alfred-stokespace]

It's like you were reading my mind! I was just about to add an additional comment here that I'm having trouble interacting with the 2nd ip...

my use case is rather odd in that I need to have port 22 (ssh) available for both of the nebula ip's. And I just realized that the second nebula ip isn't ssh-able so now i'm stuck.

[alfred-stokespace]

I guess it's time to checkout the repo and read the code.

[NiceGuyIT]

I thought SSH would work because it binds to 0.0.0.0 and Nebula traffic flows.

... But you're right. The connection times out.

This isn't that urgent for me to spend time troubleshooting it. Good luck!

[alfred-stokespace]

hmm... gonna guess this is why we see the behavior with only last process working and restart of first required to regain ...

https://github.com/slackhq/nebula/blob/master/overlay/tun_linux.go#L88

func newTun(l *logrus.Logger, deviceName string, cidr *net.IPNet, defaultMTU int, routes []Route, txQueueLen int, multiqueue bool) (*tun, error) {
	fd, err := unix.Open("/dev/net/tun", os.O_RDWR, 0)
	if err != nil {
		return nil, err
	}

If'n I'm reading that right... doesn't matter if you name your device something different you are always going to open /dev/net/tun/
and that seems like it won't work out like we expect.

[alfred-stokespace]

I don't know much about /dev/net/tun (seen here https://github.com/slackhq/nebula/blob/master/overlay/tun_linux.go#L88 ) but it seems it's not a typical file and as such allows multiple ReadWrite interactions.
I found this post useful for some better understanding of what nebula is doing and maybe diagnose the issue.
https://stackoverflow.com/a/35735842

I can confirm that with tun enabled on both of my ip's I see the following...

$> ip addr show
...
5: nebula2: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1300 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 172.20.0.51/24 scope global nebula2
       valid_lft forever preferred_lft forever
    inet6 fe80::cb56:4252:7b23:d99d/64 scope link stable-privacy
       valid_lft forever preferred_lft forever
6: nebula1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1300 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 172.20.0.50/24 scope global nebula1
       valid_lft forever preferred_lft forever
    inet6 fe80::ed05:80bb:dc8c:e4b0/64 scope link stable-privacy
       valid_lft forever preferred_lft forever

from that same host I can ping both addresses ...

root@MYHOST:/etc/nebula# ping -c 2 172.20.0.50 && ping -c 2 172.20.0.51
PING 172.20.0.50 (172.20.0.50) 56(84) bytes of data.
64 bytes from 172.20.0.50: icmp_seq=1 ttl=64 time=0.033 ms
64 bytes from 172.20.0.50: icmp_seq=2 ttl=64 time=0.041 ms

--- 172.20.0.50 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1018ms
rtt min/avg/max/mdev = 0.033/0.037/0.041/0.004 ms
PING 172.20.0.51 (172.20.0.51) 56(84) bytes of data.
64 bytes from 172.20.0.51: icmp_seq=1 ttl=64 time=0.029 ms
64 bytes from 172.20.0.51: icmp_seq=2 ttl=64 time=0.085 ms

--- 172.20.0.51 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1022ms
rtt min/avg/max/mdev = 0.029/0.057/0.085/0.028 ms

This makes me think that the problem might actually not be with this host but actually might be related to the lighthouse? Like maybe it only has a single place to relate a physical ip with a neb-ip with and we are just stomping on the previous value ?

[NiceGuyIT]

Re-reading your post above (in this thread), it sounds like if tun is enabled for both services, you can ping both nebula IPs from the host but not from other hosts. This is the same result I had with a node/lighthouse setup.

Does this matrix quantify the problem? Services on the host can only bind to the IPs shown in ip addr show.

Service 1	Service 2	tun enabled for both?	ip addr show	Ping from another host
node	node	Yes	2 Nebula IPs shown	Can ping the last service started
node	node	No; tun disabled for one	1 Nebula IP shown	Can ping both Nebula IPs
node	lighthouse	Yes	2 Nebula IPs shown	Can ping the last service started
node	lighthouse	No; tun disabled for one	1 Nebula IP shown	Can ping both Nebula IPs

[alfred-stokespace]

The first two rows look correct.
The next two are confusing, and if I'm interpreting them correctly, you're asking if the "Service 2" was attempted on the lighthouse host and no, I've not tried that.

[NiceGuyIT]

The last 2 rows are my setup. It looks like we are in the same boat. It doesn't matter if it's node/node or node/lighthouse combination on the host. tun has to be disabled on one nebula service for other Nebula nodes to ping the Nebula IPs. But that has the drawback that the host sees only 1 IP and other services on the host can bind to that 1 IP. This limits the functionality of multiple Nebula IPs on one host, either in node/node or node/lighthouse combination.

[alfred-stokespace]

A thing I want to try and maybe you already have?

To recreate the certificates such that the doubled up neb-ip's have non-overlapping cidr ranges. My theory is that the OS of the doubled-up host doesn't have a good way of disambiguating the two cidr ranges for routing logic.
I'm also skeptical about whether that makes any sense though.
As I understand it each process opens a port on 0.0.0.0

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      437/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1190/sshd: /usr/sbi
tcp6       0      0 :::22                   :::*                    LISTEN      1190/sshd: /usr/sbi
udp        0      0 127.0.0.53:53           0.0.0.0:*                           437/systemd-resolve
udp        0      0 10.100.3.187:68         0.0.0.0:*                           434/systemd-network
udp6       0      0 0.0.0.0:4242            :::*                                2828/nebula
udp6       0      0 0.0.0.0:4251            :::*                                2781/nebula

and if they have overlapping cidrs does that confuse whatever the thing is that routes packets?
command ran: ip addr show | grep "nebula\|inet 17"
output...

    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0  (IGNORE ME, GREP CAUGHT THIS)
5: nebula2: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1300 qdisc fq_codel state UNKNOWN group default qlen 500
    inet 172.20.0.51/24 scope global nebula2
6: nebula1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1300 qdisc fq_codel state UNKNOWN group default qlen 500
    inet 172.20.0.50/24 scope global nebula1

[alfred-stokespace]

nope... that didn't work ERRO[0000] static_host_map key is not in our subnet, invalid entry=1 network=172.20.0.50/32 vpnIp=172.20.0.1

/etc/nebula/nebula -config /etc/nebula/config.yaml
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:outgoing endPort:0 groups:[] host:anyip: proto:0 startPort:0]"
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:incoming endPort:0 groups:[] host:anyip: proto:0 startPort:0]"
INFO[0000] Firewall started                              firewallHash=21716b47a7a140e448077fe66c31b4b42f232e996818d7dd1c6c4991e066dbdb
INFO[0000] Main HostMap created                          network=172.20.0.50/32 preferredRanges="[]"
INFO[0000] UDP hole punching enabled
ERRO[0000] static_host_map key is not in our subnet, invalid  entry=1 network=172.20.0.50/32 vpnIp=172.20.0.1