Hubble crashes on Ubuntu 23.10 (run with Podman)

[jgresham]

What is the bug?
Hubble crashes on Ubuntu 23.10 (run with Podman)

{
"level":50,"time":1698131311256,"pid":1,"hostname":"c3a58e5cf2b2",
"reason":"Uncaught exception","err":{"type":"Error","message":"Request failed: 500","stack":"Error: Request failed: 500\n    
    at ClientRequest.<anonymous> (file:///home/node/app/node_modules/@achingbrain/nat-port-mapper/dist/src/upnp/fetch.js:43:23)\n    
    at ClientRequest.emit (node:events:511:28)\n    
    at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:693:27)\n    
    at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17)\n    
    at Socket.socketOnData (node:_http_client:535:22)\n    at Socket.emit (node:events:511:28)\n    
    at addChunk (node:internal/streams/readable:332:12)\n    
    at readableAddChunk (node:internal/streams/readable:305:9)\n    at Readable.push (node:internal/streams/readable:242:10)\n    
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)"},
"msg":"shutting down hub"}

How can it be reproduced? (optional)
Run command

podman run -q -d --name hubble -v "/home/user/.config/nodes/hubble-1698132555/hub":/home/node/app/apps/hubble/.hub:z -v "/home/user/.config/nodes/hubble-1698132555/rocks":/home/node/app/apps/hubble/.rocks:z -p 2282:2282/tcp -p 2282:2282/udp -p 2281:2281 --user 0 docker.io/farcasterxyz/hubble:latest node --no-warnings build/cli.js start --ip 0.0.0.0  --gossip-port 2282 --http-api-port 2281 --network 1 -b /dns/nemes.farcaster.xyz/tcp/2282 --eth-mainnet-rpc-url https://eth-mainnet.g.alchemy.com/v2/<key> --l2-rpc-url https://opt-mainnet.g.alchemy.com/v2/<key>

Additional context (optional)
I haven't seen this before and I have tested on many other OS/archs, so I am thinking it is somehow related to Ubuntu 23.10, nonetheless it looks like Hubble might be crashing when a upnp map fails?

[sanjayprabhu]

We've seen this happen with certain routers. Passing in --announce-ip with the external ip will bypass this check but not sure if the hub will actually be reachable. You might need to forward ports manually with your router.

[jgresham]

It does seem to be my router or something with my home network.

Oddly, I see the entries in the Upnp table on my router get created.

I think newer versions of libp2p (& subsequently @achingbrain/nat-port-mapper) fix this. I think @achingbrain/[email protected] and newer may fix it.

[sds]

We've since updated certain parts of our use of libp2p since this issue was opened. Going to close, but if you're still seeing this and have readily-reproducible steps, let us know.