Recovering after network disturbance

[bigretromike]

Long story short.

Peer after being cut of from network and reconnected, reconnected to few other peers but did not reconnect (automaticly) with server.
Had to manualy innernet down innnetwork innernet up innnetwork only then I was able to reconnect to server.

[mcginty]

Yeah, I totally believe it can get into weird situations with network reconnects. I think generally innernet up needs more sanity checks rather than assuming everything with innernet is functional if the interface is up already.

[bigretromike]

It toke me some time to notice that the issue is with innernet as I could ping peers without bigger issue but not the "server" itself, so innernet fetch also did not work because of that. A simple ping to server would be enough assuming that with working server service can fetch to restore connection to peers

[bigretromike]

Maybe someone find this helpful, this is a quick and dirty way to handle this:

#!/bin/bash

IP="__IP_OF_MAIN_SERVER___"
WAIT_TIME=10

while true; do
    if ping -c 1 -W 1 $IP > /dev/null; then
        WAIT_TIME=10
    else
        bash -c "innernet down __NAME_OF_NETWORK__"
        bash -c "innernet up __NAME_OF_NETWORK__"
        sleep $WAIT_TIME
        if [ $WAIT_TIME -lt 30 ]; then
            WAIT_TIME=$((WAIT_TIME + 10))
        fi
    fi
    sleep $WAIT_TIME 
done

Simple script to ping the innernet server, if ping does not work assume network is down and restart innernet network. Until something internal is not build in with watchdog I will be using something like this - or until there is some kind of mechanism build it to check network health.

[alerque]

I've been struggling with this a lot lately. The above mentioned work-arounds don't always work for me. I'm wondering if I have the same problem or just a related one. The complication I can think of of in my case is that most of my piers are behind double layers of NAT. Everything works great until a pier reboots or the top level NAT will get assigned a new address, and then getting all the piers talking to all the other ones becomes a huge fiasco. Each pier usually gets reconnected with the server and the server can usually reach all piers, and the piers behind the same NATs usually find each other, but until I shut down and restart interfaces on multiple pairs of interfaces sometimes multiple times I can't get them directly talking again.

What can I do that would be helpful to debug this and determine if it is the same root issue or some other one?

[strohel]

@alerque I think connectivity between hosts behind NATs is our biggest struggle too. 😅

The most reliable way is to set port forwarding, set fixed listen port, and set override endpoint (since 1.7.0 override just the port). But likely you are not in control of the NATs.

innernet could support UPnP ("automatic port forwarding", #110), but that is not usually enabled on business routers due to (valid) security concerns. This would also only help with a single NAT layer.

Sometimes IPv4 is NATed, but you get public IPv6: innernet can make use of it (thanks to "NAT traversal", a badly named feature) and act as a kind of IPv4 over IPv6 tunnel.

If you cannot afford anything of the above, then you're on the mercy of the NATs. Innernet currently doesn't do any advanced NAT holepunching (listening at many ports, "spraying" packets over a range of ports...) like WebRTC and I think tailscale does.

"Nice" NATs would prefer to keep the same internal and external port for connections, if that external port is available. Or at least keep the same port mapping for multiple destinations. "Naughty" NATs would force remapping the port, and randomize it for every target host. If at least one host in a given peer-to-peer pair is behind a "nice" NAT, the connection usually works. Unless you're unlucky and the port is taken, then innernet down && innernet up may help as that randomizes the port (if not fixed) -- we may try baking such behavior directly to innernet.

You can tell if a peer is behind a "nice" NAT by examining under what external endpoint that peer is seen by other peers that can connect to it (innernet show | grep -A4 examined-peer). If they all see it under the same endpoint (including port), then it is behind a "nice" NAT and connection should work.

So that would be the first think to determine. If you get "hey, this endpoint should work, but somehow innernet doesn't try to use it", then we have instructions I can share how to change the endpoint manually using wg.

[bigretromike]

It would be really great tool without all fancy webui, bloated service etc.... but unable to connect to 'main' server after network connectivity issues is big breaker....

I would assume that connectivity with server that manage peers is essential if there is no mesh connectivity - but I'm not sure if I would like to connect to server thru other peers...

Maybe it is just me and my use case - where innernet server is also a reverse proxy server so the connectivity is over innernet/wg tunnel and if that breaks peer <---/---> server then the whole service is down.

I should probably pick something more robust or relay on those hacky scripts

[alerque]

but unable to connect to 'main' server after network connectivity issues is big breaker....

Is that what this issue is about? In many years of running innernet across three continents I've never had an issue with piers recovering their connectivity with the main server. That seems like a different issue than the pier-to-pier links that have trouble recognizing when they are assigned new NAT ports, etc. Do we need to split this issue into two?

[bigretromike]

Do we need to split this issue into two?

I don't know, because my issue is that the server is also peer ?!
Or is it a mistake and it shouldn't be consider as one - but then would we need create another peer for server ?

The server is out there in public, with always public IP (many times static). Peer being unable to reconnect with server is a bit absurd.

I remember reading somewhere that if there is a wg-tunnel up and there is a ip-change (on peer side) the tunnel will not reconnect without restart/reload - I'm unable to test this but I read about it on d42 wiki few times.
That critical flaw would then lead to only use cached information on peer (as of 1.7.0) and ?never? reconnect for updated list and that would be security risk, would it ? If I cannot get information about revoked peers ?

I understand that this could be hard issue to bite, that's why I wanted to report it as this is not something that happens often, but when it does it hit hard.

[bigretromike]

Looks like I was hit with this issue again at night.

Can't tell how long I was struggeling (can't tell if it was because of 1.7.0 update or not) but nothing did help until I systemctl restart innernet-server@ on server. I had to systemctl daemon-reload because of the update. But then and only then peers started to talk with server. I know this is edge case where most (all) peers are behind same NAT so in my case if NAT get different IP (all peers are behind it) the whole network (connection to server) go silent, which would be understandable if the IP of server change (but not in this case).

I hope it was because of different version on client and server.