Packages cannot be downloaded on IPv6-only networks

[jeanas]

Posted by @jenslink on the pip tracker (pypa/pip#12486), moved here:

Description

Hi,

there is a small problem with IPv6 only environments with an IPv6 only resolver. You can not install anything:

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f835a601090>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/

I debugged this, and it all boils down to a CDN hosted by Fastly and the Fastly authoritative DNS Severs used do not have AAAA records. The solution is easy: Ask Fastly to change to different auth servers with AAAA records.

deb.debian.org (an also the OpenBSD CDN) had the same problem.

Temporary workaround: use a DNS dual-stacked DNS resolver.

Expected behavior

No response

pip version

all

Python version

all

OS

all

How to Reproduce

* Setup an IPv6 only VM

* configure a local resolver (or have another IPv6 only VM with an IPv6 only resolver

* install anything via pip

Output

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f835a601090>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/

Code of Conduct

* [x]  I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).

[Jonathan-Landeed]

+1

Important as AWS is going to start charging for IPv4 addresses and already (over)charges a lot for NAT gateways.

[JanneKiiskila]

I think there are issues even if you're on an IPv4+IPv6 network which has machines that prefer IPv6. We have some self-hosted GitHub runners and once we enabled IPv6 and the hosts had Ubuntu 22.04 as the base (which prefers IPv6 traffic by default) we saw some Python-related jobs start to take 10x more runtime than earlier. The delays were related to sudden stops in the runs in Python package installations taking several minutes, instead of seconds as earlier.

We did some trials on this and once we updated the /etc/gai.conf file to prefer IPv4 over IPv6 we got back to normal runtimes for the test jobs. In practice this meant for us having test jobs take 50..55 minutes instead of the normal 5 minutes.

This article might be of use to others as well:

• https://weblog.lkiesow.de/20220311-make-linux-prefer-ipv4.html

[di]

Can folks provide the results of the following commands on machines that hare having trouble connecting via IPv6?

• traceroute6 pypi.org
• curl -vvv -I --ipv6 https://pypi.org/
• echo -n | openssl s_client -6 -connect pypi.org:443

[JanneKiiskila]

Of course we can. These are from a machine hooked up to Google fiber in Texas, USA.

traceroute6

~# traceroute6 pypi.org
traceroute to pypi.org (2a04:4e42:600::223) from 2605:a601:a0f9:d502:216:3eff:fee8:d22, port 33434, from port 54610, 30 hops max, 60 bytes packets
 1  2605:a601:a0f9:d502::1 (2605:a601:a0f9:d502::1)  0.490 ms  0.226 ms  0.372 ms 
 2  * * *         
 3  2605:a601:ffff:9009:1e:f1ba:0:97 (2605:a601:ffff:9009:1e:f1ba:0:97)  16.551 ms  16.471 ms  16.578 ms 
 4  * * *         
 5  * * *         
 6  * * *         
 7  * * *         
 8  * * *         
 9  * * *         
10  * * *         
11  * * *         
12  * * *         
13  * * *         
14  * * *         
15  * * *         
16  * * *         
17  * * *         
18  * * *         
19  * * *         
20  * * *         
21  * * *         
22  * * *         
23  * * *         
24  * * *         
25  * * *         
26  * * *         
27  * * *         
28  * * *         
29  * * *         
30  * * * 

Hmmh... Not sure how useful that is?

curl

Well, curl then.

curl -vvv -I --ipv6 https://pypi.org/
*   Trying 2a04:4e42::223:443...

and it gets stuck there for several minutes... It then continues:

* connect to 2a04:4e42::223 port 443 failed: Connection timed out
*   Trying 2a04:4e42:400::223:443...
* After 82766ms connect time, move on!
* connect to 2a04:4e42:400::223 port 443 failed: Connection timed out
*   Trying 2a04:4e42:200::223:443...
* After 20691ms connect time, move on!
* connect to 2a04:4e42:600::223 port 443 failed: Connection timed out
* Failed to connect to pypi.org port 443 after 279309 ms: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to pypi.org port 443 after 279309 ms: Connection timed out

I think that proves there's a problem somewhere.
But not necessarily in pypi.org, though!

openssl

Similar story here, it gets stuck. It's actually not printing out anything.
UPDATE: Well, it turns out it did connect and this came out (but it took minutes).

~# echo -n | openssl s_client -6 -connect pypi.org:443
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign Atlas R3 DV TLS CA 2023 Q2
verify return:1
depth=0 CN = pypi.org
verify return:1
---
Certificate chain
 0 s:CN = pypi.org
   i:C = BE, O = GlobalSign nv-sa, CN = GlobalSign Atlas R3 DV TLS CA 2023 Q2
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 29 19:53:38 2023 GMT; NotAfter: May 30 19:53:37 2024 GMT
 1 s:C = BE, O = GlobalSign nv-sa, CN = GlobalSign Atlas R3 DV TLS CA 2023 Q2
   i:OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 18 03:36:52 2023 GMT; NotAfter: Jan 18 00:00:00 2025 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGfzCCBWegAwIBAgIQATphpUBWfreAY2OTrbDywTANBgkqhkiG9w0BAQsFADBY
MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEuMCwGA1UE
AxMlR2xvYmFsU2lnbiBBdGxhcyBSMyBEViBUTFMgQ0EgMjAyMyBRMjAeFw0yMzA0
MjkxOTUzMzhaFw0yNDA1MzAxOTUzMzdaMBMxETAPBgNVBAMMCHB5cGkub3JnMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsVzK0dZ2ALTL74lfSb8xvcqO
eEP1opP9RNEQXhsEAs5sI8hYBP/j8IJO4cBZN/585CTJ8SAMxXon7IC0FvXnhefc
bE0kyHQf1cchLuqi2cArNUCZd7kIwYCpIpDvYLTlcJP1ClwwlUdDfobcMMLlot14
NBY11KT1iuNdujm83xF4LvHZu1dNTNMkU0Qnzl26tVKzewC29nEELWwz0Navusc/
5RVUI8bWrY2ZEJyeJvbs6aHcNbPdhfb8JP8pzheLNDL0VjWnxJnIDWttprck5FJM
lueKFjrx4vGB8kWfFpLkOSoq4/VW75FvR78Zx2Bj8BroMnbUiQ/NLhlBFHCscQID
AQABo4IDiDCCA4QwPgYDVR0RBDcwNYIIcHlwaS5vcmeCCioucHlwaS5vcmeCDHd3
dy5weXBpLm9yZ4IPZG9uYXRlLnB5cGkub3JnMA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwHQYDVR0OBBYEFH5KpEjjaz1tMuya
qyjnyKenhRycMFcGA1UdIARQME4wCAYGZ4EMAQIBMEIGCisGAQQBoDIKAQMwNDAy
BggrBgEFBQcCARYmaHR0cHM6Ly93d3cuZ2xvYmFsc2lnbi5jb20vcmVwb3NpdG9y
eS8wDAYDVR0TAQH/BAIwADCBngYIKwYBBQUHAQEEgZEwgY4wQAYIKwYBBQUHMAGG
NGh0dHA6Ly9vY3NwLmdsb2JhbHNpZ24uY29tL2NhL2dzYXRsYXNyM2R2dGxzY2Ey
MDIzcTIwSgYIKwYBBQUHMAKGPmh0dHA6Ly9zZWN1cmUuZ2xvYmFsc2lnbi5jb20v
Y2FjZXJ0L2dzYXRsYXNyM2R2dGxzY2EyMDIzcTIuY3J0MB8GA1UdIwQYMBaAFMai
YRNFFcyQj7rBrLzMOw0O0N4GMEgGA1UdHwRBMD8wPaA7oDmGN2h0dHA6Ly9jcmwu
Z2xvYmFsc2lnbi5jb20vY2EvZ3NhdGxhc3IzZHZ0bHNjYTIwMjNxMi5jcmwwggF/
BgorBgEEAdZ5AgQCBIIBbwSCAWsBaQB2AHb/iD8KtvuVUcJhzPWHujS0pM27Kdxo
Qgqf5mdMWjp0AAABh86UAM4AAAQDAEcwRQIhAMEeNv9F3vkrUFb8UDUnBgcJ2ed2
u0Y2h4qXuv4SyBcCAiAdoR3AfpZboQtcbZ4F5kfLAIaEZ48avbJTJGrTg0Z0dwB2
ADtTd3U+LbmAToswWwb+QDtn2E/D9Me9AA0tcm/h+tQXAAABh86UAXkAAAQDAEcw
RQIhAL+PCGdwFgr/6mjPltxIkO2pSZ6eDY2sv8n+cW5hXCmzAiAL3XKCZqwJGJXE
kvt1ZJkMoqLzAy5CM2ANzSIXQni6wgB3AHPZnokbTJZ4oCB9R53mssYc0FFecRkq
jGuAEHrBd3K1AAABh86UAcgAAAQDAEgwRgIhALbbX2nIHt3iNRL+nMyqN0CUwxDz
BaSjdQhfBDPciwKbAiEAov0IJaslITt1Mxtyh4Ped9ZOkST5A0K0/Fdbk64zI+sw
DQYJKoZIhvcNAQELBQADggEBAGodVH7XXcayedVPuAMF5agiOBVJp7evMWz2R8xK
ON0jovv1oItNrXvt3tVWvPUl4Dm/AIbCrAHw/d1cz1loot8Nv2tRI8vRpBZ6bnnQ
wWKVlJmz3BRr42wV5HMiLjk0cUFgL+BDcoqpcEKUXsuNgTaevvLkx38Oo8b4Y9PT
CBkcuiqBfqViBAHB9mUcJCtpXW5waX+DwvbkkTULNTdYq5CZLENSPa2H22r/V0SI
gu08iGpOOdMJaaFBQp35ECW+UTdYSa+9tMx+B3GzpgxXVcW/P/ZCKcThequGVDbr
7hy8HRcXVP/+ejef0co8/E9MR5/EYrICq5M7JWJunSDgFP8=
-----END CERTIFICATE-----
subject=CN = pypi.org
issuer=C = BE, O = GlobalSign nv-sa, CN = GlobalSign Atlas R3 DV TLS CA 2023 Q2
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3387 bytes and written 374 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 69746A55416BFDFB6B85A4B3B4FA238B6E9049795B7A1A72850D52B2169CBA1E
    Session-ID-ctx: 
    Resumption PSK: C3FA56B07F20D8A73CCDE33E05B916265E9F79ACC8EA1CC5F6FC88B83F61A4FD
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 3600 (seconds)
    TLS session ticket:
    0000 - 9a 1c cd 8a f6 1f 21 51-30 23 31 0d 5a cd 94 2b   ......!Q0#1.Z..+
    0010 - e4 81 e2 9e ac 7c e0 13-7d 64 7a 20 6a 0f 1d 03   .....|..}dz j...
    0020 - 4f 9d 00 42 a9 57 83 13-00 69 b4 13 b7 5f 57 15   O..B.W...i..._W.
    0030 - fb 12 19 5c 1d 1b ad 2a-b9 49 dd 6f cf ec 8a ae   ...\...*.I.o....
    0040 - 31 d4 a1 07 8a b5 47 27-f7 1c f8 02 f3 42 a1 ce   1.....G'.....B..
    0050 - fb 78 d9 c0 f1 26 e5 bb-67 14 1a fc 00 34 61 56   .x...&..g....4aV
    0060 - aa 35 fb b1 de 9f c6 0e-b4 c9 fc 12 88 39 d8 84   .5...........9..
    0070 - 84 a7 60 e6 53 6e 2a 27-9d ea 3f 45 73 64 2b 76   ..`.Sn*'..?Esd+v
    0080 - 64 99 90 94 ad f8 48 a3-d0 ca 20 49 c4 a4 ff 7b   d.....H... I...{
    0090 - 04 12 0e e9 c6 68 24 79-7d 66 2b c1 df 0b 83 7f   .....h$y}f+.....

    Start Time: 1713363532
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
DONE

When running the 2nd time, it finished in <1 second. 3rd time, stuck again, etc. Completely random behaviour - sometimes <1 second, sometimes minutes.

Summary

Connection problem, but where? Let's get some alternative results from a machine hooked up via DNA cable modem in Oulu, Finland.

traceroute

$ traceroute6 pypi.org
traceroute to pypi.org (2a04:4e42:600::223), 30 hops max, 80 byte packets
 1  drlxtrygckcts--x85n4t-3.rev.dnainternet.fi (2001:14ba:7267:700:a240:a0ff:fe7c:caf1)  1.393 ms  1.367 ms  1.264 ms
 2  * * *
 3  oul2-sr2.dnaip.fi (2001:14b8:1040:104::44:0)  10.709 ms  10.858 ms  10.724 ms
 4  * * *
 5  * * *
 6  * * *
 7  2620:11a:c000:127:fa57::1 (2620:11a:c000:127:fa57::1)  22.610 ms  14.698 ms  21.638 ms
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

curl

This comes in <1 second.

curl -vvv -I --ipv6 https://pypi.org/
*   Trying 2a04:4e42::223:443...
* Connected to pypi.org (2a04:4e42::223) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=pypi.org
*  start date: Apr 29 19:53:38 2023 GMT
*  expire date: May 30 19:53:37 2024 GMT
*  subjectAltName: host "pypi.org" matched cert's "pypi.org"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Atlas R3 DV TLS CA 2023 Q2
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x64e652f83eb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> HEAD / HTTP/2
> Host: pypi.org
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
HTTP/2 200 
< content-type: text/html; charset=UTF-8
content-type: text/html; charset=UTF-8
< etag: "vDDx/F+BCvG0drFU8ErU9g"
etag: "vDDx/F+BCvG0drFU8ErU9g"
< content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ https://api.github.com/search/issues https://*.google-analytics.com https://*.analytics.google.com https://*.googletagmanager.com fastly-insights.com *.fastly-insights.com *.ethicalads.io https://api.pwnedpasswords.com https://cdn.jsdelivr.net/npm/mathjax@3.2.2/es5/sre/mathmaps/ https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self' https://checkout.stripe.com; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://pypi-camo.freetls.fastly.net/ https://*.google-analytics.com https://*.googletagmanager.com *.fastly-insights.com *.ethicalads.io ethicalads.blob.core.windows.net; script-src 'self' https://*.googletagmanager.com https://www.google-analytics.com https://ssl.google-analytics.com *.fastly-insights.com *.ethicalads.io 'sha256-U3hKDidudIaxBDEzwGJApJgPEf2mWk6cfMWghrAa6i0=' https://cdn.jsdelivr.net/npm/mathjax@3.2.2/ 'sha256-1CldwzdEg2k1wTmf7s5RWVd7NMXI/7nxxjJM2C4DqII=' 'sha256-0POaN8stWYQxhzjKS+/eOfbbJ/u4YHO5ZagJvLpMypo='; style-src 'self' fonts.googleapis.com *.ethicalads.io 'sha256-2YHqZokjiizkHi1Zt+6ar0XJ0OeEy/egBnlm+MDMtrM=' 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=' 'sha256-JLEjeN9e5dGsz5475WyRaoA4eQOdNPxDIeUhclnJDCE=' 'sha256-mQyxHEuwZJqpxCw3SLmc4YOySNKXunyu2Oiz1r3/wAE=' 'sha256-OCf+kv5Asiwp++8PIevKBYSgnNLNUZvxAp4a7wMLuKA=' 'sha256-h5LOiLhk6wiJrGsG5ItM0KimwzWQH/yAcmoJDJL//bY='; worker-src *.fastly-insights.com
content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ https://api.github.com/search/issues https://*.google-analytics.com https://*.analytics.google.com https://*.googletagmanager.com fastly-insights.com *.fastly-insights.com *.ethicalads.io https://api.pwnedpasswords.com https://cdn.jsdelivr.net/npm/mathjax@3.2.2/es5/sre/mathmaps/ https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self' https://checkout.stripe.com; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://pypi-camo.freetls.fastly.net/ https://*.google-analytics.com https://*.googletagmanager.com *.fastly-insights.com *.ethicalads.io ethicalads.blob.core.windows.net; script-src 'self' https://*.googletagmanager.com https://www.google-analytics.com https://ssl.google-analytics.com *.fastly-insights.com *.ethicalads.io 'sha256-U3hKDidudIaxBDEzwGJApJgPEf2mWk6cfMWghrAa6i0=' https://cdn.jsdelivr.net/npm/mathjax@3.2.2/ 'sha256-1CldwzdEg2k1wTmf7s5RWVd7NMXI/7nxxjJM2C4DqII=' 'sha256-0POaN8stWYQxhzjKS+/eOfbbJ/u4YHO5ZagJvLpMypo='; style-src 'self' fonts.googleapis.com *.ethicalads.io 'sha256-2YHqZokjiizkHi1Zt+6ar0XJ0OeEy/egBnlm+MDMtrM=' 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=' 'sha256-JLEjeN9e5dGsz5475WyRaoA4eQOdNPxDIeUhclnJDCE=' 'sha256-mQyxHEuwZJqpxCw3SLmc4YOySNKXunyu2Oiz1r3/wAE=' 'sha256-OCf+kv5Asiwp++8PIevKBYSgnNLNUZvxAp4a7wMLuKA=' 'sha256-h5LOiLhk6wiJrGsG5ItM0KimwzWQH/yAcmoJDJL//bY='; worker-src *.fastly-insights.com
< referrer-policy: origin-when-cross-origin
referrer-policy: origin-when-cross-origin
< accept-ranges: bytes
accept-ranges: bytes
< date: Wed, 17 Apr 2024 14:20:30 GMT
date: Wed, 17 Apr 2024 14:20:30 GMT
< x-served-by: cache-iad-kjyo7100172-IAD, cache-hel1410023-HEL
x-served-by: cache-iad-kjyo7100172-IAD, cache-hel1410023-HEL
< x-cache: HIT, HIT
x-cache: HIT, HIT
< x-cache-hits: 31, 1
x-cache-hits: 31, 1
< x-timer: S1713363631.613015,VS0,VE1
x-timer: S1713363631.613015,VS0,VE1
< vary: Cookie, Accept-Encoding
vary: Cookie, Accept-Encoding
< strict-transport-security: max-age=31536000; includeSubDomains; preload
strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-frame-options: deny
x-frame-options: deny
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-permitted-cross-domain-policies: none
x-permitted-cross-domain-policies: none
< permissions-policy: publickey-credentials-create=(self),publickey-credentials-get=(self),accelerometer=(),ambient-light-sensor=(),autoplay=(),battery=(),camera=(),display-capture=(),document-domain=(),encrypted-media=(),execution-while-not-rendered=(),execution-while-out-of-viewport=(),fullscreen=(),gamepad=(),geolocation=(),gyroscope=(),hid=(),identity-credentials-get=(),idle-detection=(),local-fonts=(),magnetometer=(),microphone=(),midi=(),otp-credentials=(),payment=(),picture-in-picture=(),screen-wake-lock=(),serial=(),speaker-selection=(),storage-access=(),usb=(),web-share=(),xr-spatial-tracking=()
permissions-policy: publickey-credentials-create=(self),publickey-credentials-get=(self),accelerometer=(),ambient-light-sensor=(),autoplay=(),battery=(),camera=(),display-capture=(),document-domain=(),encrypted-media=(),execution-while-not-rendered=(),execution-while-out-of-viewport=(),fullscreen=(),gamepad=(),geolocation=(),gyroscope=(),hid=(),identity-credentials-get=(),idle-detection=(),local-fonts=(),magnetometer=(),microphone=(),midi=(),otp-credentials=(),payment=(),picture-in-picture=(),screen-wake-lock=(),serial=(),speaker-selection=(),storage-access=(),usb=(),web-share=(),xr-spatial-tracking=()
< content-length: 20202
content-length: 20202

< 
* Connection #0 to host pypi.org left intact

openssl

This comes in <1 second.

 echo -n | openssl s_client -6 -connect pypi.org:443
Connecting to 2a04:4e42:400::223
CONNECTED(00000003)
depth=2 OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign
verify return:1
depth=1 C=BE, O=GlobalSign nv-sa, CN=GlobalSign Atlas R3 DV TLS CA 2023 Q2
verify return:1
depth=0 CN=pypi.org
verify return:1
---
Certificate chain
 0 s:CN=pypi.org
   i:C=BE, O=GlobalSign nv-sa, CN=GlobalSign Atlas R3 DV TLS CA 2023 Q2
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 29 19:53:38 2023 GMT; NotAfter: May 30 19:53:37 2024 GMT
 1 s:C=BE, O=GlobalSign nv-sa, CN=GlobalSign Atlas R3 DV TLS CA 2023 Q2
   i:OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 18 03:36:52 2023 GMT; NotAfter: Jan 18 00:00:00 2025 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGfzCCBWegAwIBAgIQATphpUBWfreAY2OTrbDywTANBgkqhkiG9w0BAQsFADBY
MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEuMCwGA1UE
AxMlR2xvYmFsU2lnbiBBdGxhcyBSMyBEViBUTFMgQ0EgMjAyMyBRMjAeFw0yMzA0
MjkxOTUzMzhaFw0yNDA1MzAxOTUzMzdaMBMxETAPBgNVBAMMCHB5cGkub3JnMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsVzK0dZ2ALTL74lfSb8xvcqO
eEP1opP9RNEQXhsEAs5sI8hYBP/j8IJO4cBZN/585CTJ8SAMxXon7IC0FvXnhefc
bE0kyHQf1cchLuqi2cArNUCZd7kIwYCpIpDvYLTlcJP1ClwwlUdDfobcMMLlot14
NBY11KT1iuNdujm83xF4LvHZu1dNTNMkU0Qnzl26tVKzewC29nEELWwz0Navusc/
5RVUI8bWrY2ZEJyeJvbs6aHcNbPdhfb8JP8pzheLNDL0VjWnxJnIDWttprck5FJM
lueKFjrx4vGB8kWfFpLkOSoq4/VW75FvR78Zx2Bj8BroMnbUiQ/NLhlBFHCscQID
AQABo4IDiDCCA4QwPgYDVR0RBDcwNYIIcHlwaS5vcmeCCioucHlwaS5vcmeCDHd3
dy5weXBpLm9yZ4IPZG9uYXRlLnB5cGkub3JnMA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwHQYDVR0OBBYEFH5KpEjjaz1tMuya
qyjnyKenhRycMFcGA1UdIARQME4wCAYGZ4EMAQIBMEIGCisGAQQBoDIKAQMwNDAy
BggrBgEFBQcCARYmaHR0cHM6Ly93d3cuZ2xvYmFsc2lnbi5jb20vcmVwb3NpdG9y
eS8wDAYDVR0TAQH/BAIwADCBngYIKwYBBQUHAQEEgZEwgY4wQAYIKwYBBQUHMAGG
NGh0dHA6Ly9vY3NwLmdsb2JhbHNpZ24uY29tL2NhL2dzYXRsYXNyM2R2dGxzY2Ey
MDIzcTIwSgYIKwYBBQUHMAKGPmh0dHA6Ly9zZWN1cmUuZ2xvYmFsc2lnbi5jb20v
Y2FjZXJ0L2dzYXRsYXNyM2R2dGxzY2EyMDIzcTIuY3J0MB8GA1UdIwQYMBaAFMai
YRNFFcyQj7rBrLzMOw0O0N4GMEgGA1UdHwRBMD8wPaA7oDmGN2h0dHA6Ly9jcmwu
Z2xvYmFsc2lnbi5jb20vY2EvZ3NhdGxhc3IzZHZ0bHNjYTIwMjNxMi5jcmwwggF/
BgorBgEEAdZ5AgQCBIIBbwSCAWsBaQB2AHb/iD8KtvuVUcJhzPWHujS0pM27Kdxo
Qgqf5mdMWjp0AAABh86UAM4AAAQDAEcwRQIhAMEeNv9F3vkrUFb8UDUnBgcJ2ed2
u0Y2h4qXuv4SyBcCAiAdoR3AfpZboQtcbZ4F5kfLAIaEZ48avbJTJGrTg0Z0dwB2
ADtTd3U+LbmAToswWwb+QDtn2E/D9Me9AA0tcm/h+tQXAAABh86UAXkAAAQDAEcw
RQIhAL+PCGdwFgr/6mjPltxIkO2pSZ6eDY2sv8n+cW5hXCmzAiAL3XKCZqwJGJXE
kvt1ZJkMoqLzAy5CM2ANzSIXQni6wgB3AHPZnokbTJZ4oCB9R53mssYc0FFecRkq
jGuAEHrBd3K1AAABh86UAcgAAAQDAEgwRgIhALbbX2nIHt3iNRL+nMyqN0CUwxDz
BaSjdQhfBDPciwKbAiEAov0IJaslITt1Mxtyh4Ped9ZOkST5A0K0/Fdbk64zI+sw
DQYJKoZIhvcNAQELBQADggEBAGodVH7XXcayedVPuAMF5agiOBVJp7evMWz2R8xK
ON0jovv1oItNrXvt3tVWvPUl4Dm/AIbCrAHw/d1cz1loot8Nv2tRI8vRpBZ6bnnQ
wWKVlJmz3BRr42wV5HMiLjk0cUFgL+BDcoqpcEKUXsuNgTaevvLkx38Oo8b4Y9PT
CBkcuiqBfqViBAHB9mUcJCtpXW5waX+DwvbkkTULNTdYq5CZLENSPa2H22r/V0SI
gu08iGpOOdMJaaFBQp35ECW+UTdYSa+9tMx+B3GzpgxXVcW/P/ZCKcThequGVDbr
7hy8HRcXVP/+ejef0co8/E9MR5/EYrICq5M7JWJunSDgFP8=
-----END CERTIFICATE-----
subject=CN=pypi.org
issuer=C=BE, O=GlobalSign nv-sa, CN=GlobalSign Atlas R3 DV TLS CA 2023 Q2
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3387 bytes and written 380 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 77A3896F009A9960B29252EA34BC1B80461531269BA5C7D04CC177514EE4836F
    Session-ID-ctx: 
    Resumption PSK: 5A2888B3D4E9FCB00732BF10500E2358FE56D239C12F7CC2A88C1921B62370A1
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 3600 (seconds)
    TLS session ticket:
    0000 - 9a 1c cd 8a f6 1f 21 51-30 23 31 0d 5a cd 94 2b   ......!Q0#1.Z..+
    0010 - eb f6 b9 f7 df bf c9 f5-e1 93 35 19 25 5d 45 ca   ..........5.%]E.
    0020 - f4 70 11 b9 e8 5f b1 67-be 2a a8 fa ef fa d6 60   .p..._.g.*.....`
    0030 - da 60 83 64 34 2c dd 6c-cd 20 b8 a0 f4 08 69 f0   .`.d4,.l. ....i.
    0040 - 9e 68 e9 97 52 fd d9 60-ae 1e 87 74 98 2d fd ca   .h..R..`...t.-..
    0050 - 15 1a 74 46 a6 ef ba 26-11 e7 02 ea 7c 75 9d e1   ..tF...&....|u..
    0060 - e1 63 37 de b4 e2 d6 49-51 1a 95 02 ff fa a4 f4   .c7....IQ.......
    0070 - 1c 5c 81 ff 61 4c 7e 76-8c be ce eb dd 39 1f 15   .\..aL~v.....9..
    0080 - 96 4a 64 8c c6 7f af c5-51 32 f8 f6 fd 58 62 65   .Jd.....Q2...Xbe
    0090 - 2f 14 e1 a4 7e a6 53 34-57 e4 83 44 b6 79 a1 f0   /...~.S4W..D.y..

    Start Time: 1713363666
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
DONE

This also comes consistently at <1 second in Oulu, Finland. Repeated >10 times.

Conclusion

We might have been barking up the wrong tree alltogether, the issue is possibly elsewhere than pypi.org?

[JanneKiiskila]

This is a fun experiment.

time for i in {1..100}; do time echo -n | openssl s_client -6 -connect pypi.org:443 >>pypilog.txt; done

In Oulu I get that 100 repeats in:

real	0m11.156s

In the Texas Google fibre machine it's still running it (and this is 3 hours ago already).

[jenslink]

Can folks provide the results of the following commands on machines that hare having trouble connecting via IPv6?

* `traceroute6 pypi.org`

At least for my original problem, pypi.org is the wrong target. files.pythonhosted.org is the correct one. And traceroute / curl won't work because it's an DNS problem or rather an authoritative DNS server not having IPv6 problem.
deb.debian.org had the same problem some time ago.

$ host files.pythonhosted.org. 
files.pythonhosted.org is an alias for dualstack.python.map.fastly.net.
dualstack.python.map.fastly.net has address 146.75.116.223
dualstack.python.map.fastly.net has IPv6 address 2a04:4e42:8d::223

$ host deb.debian.org
deb.debian.org is an alias for debian.map.fastlydns.net.
debian.map.fastlydns.net has address 199.232.190.132
debian.map.fastlydns.net has IPv6 address 2a04:4e42:8d::644

You'll notice that they use different domains: fastly.net and fastlydns.net.

$ for i in $(dig ns fastly.net +short); do dig aaaa $i +short; done
$ 
for i in $(dig ns fastlydns.net +short); do dig aaaa $i +short; done
2a04:4e47::32
2a04:4e47:1::32
2a04:4e47:2::32
2a04:4e47:3::32

The solution seems easy: move to the other DNS server.

[jenslink]

# traceroute6 pypi.org
traceroute to pypi.org (2a04:4e42:600::223) from 2605:a601:a0f9:d502:216:3eff:fee8:d22, port 33434, from port 54610, 30 hops max, 60 bytes packets
1  2605:a601:a0f9:d502::1 (2605:a601:a0f9:d502::1)  0.490 ms  0.226 ms  0.372 ms 
2  * * *         
 3  2605:a601:ffff:9009:1e:f1ba:0:97 (2605:a601:ffff:9009:1e:f1ba:0:97)  16.551 ms  16.471 ms  16.578 ms 
 4  * * *         

Looks like broken IPv6. Is ping -6 www.google.com working? Or ping 2600::? Or traceroute -6 www.google.com?
You don't notice that from a browser because of https://en.wikipedia.org/wiki/Happy_Eyeballs.

[JanneKiiskila]

Looks like broken IPv6. Is ping -6 www.google.com working? Or ping 2600::?

$ ping -6 www.google.com
PING www.google.com(rs-in-f147.1e100.net (2607:f8b0:4023:1000::93)) 56 data bytes
64 bytes from rs-in-f147.1e100.net (2607:f8b0:4023:1000::93): icmp_seq=1 ttl=57 time=7.84 ms
64 bytes from rs-in-f147.1e100.net (2607:f8b0:4023:1000::93): icmp_seq=2 ttl=57 time=8.00 ms
64 bytes from rs-in-f147.1e100.net (2607:f8b0:4023:1000::93): icmp_seq=3 ttl=57 time=7.86 ms
64 bytes from rs-in-f147.1e100.net (2607:f8b0:4023:1000::93): icmp_seq=4 ttl=57 time=8.46 ms
^C
--- www.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 7.838/8.037/8.455/0.249 ms

and

 ping 2600::
PING 2600::(2600::) 56 data bytes
64 bytes from 2600::: icmp_seq=1 ttl=53 time=34.1 ms
64 bytes from 2600::: icmp_seq=2 ttl=53 time=37.9 ms
64 bytes from 2600::: icmp_seq=3 ttl=53 time=34.6 ms
64 bytes from 2600::: icmp_seq=4 ttl=53 time=34.7 ms
^C
--- 2600:: ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 34.077/35.312/37.871/1.496 ms

plus the traceroute:

~$ traceroute -6 www.google.com
traceroute to www.google.com (2607:f8b0:4023:1000::69), 30 hops max, 80 byte packets
 1  2605:a601:a0f9:d502::1 (2605:a601:a0f9:d502::1)  1.340 ms  0.416 ms  0.498 ms
 2  * * *
 3  2605:a601:ffff:9009:1e:f1ba:0:67 (2605:a601:ffff:9009:1e:f1ba:0:67)  7.292 ms  7.194 ms  37.092 ms
 4  2605:a600:201::bd (2605:a600:201::bd)  7.019 ms  7.079 ms  7.423 ms
 5  2607:f8b0:825e::1 (2607:f8b0:825e::1)  7.375 ms 2607:f8b0:80e5::1 (2607:f8b0:80e5::1)  10.936 ms 2607:f8b0:8327::1 (2607:f8b0:8327::1)  6.423 ms
 6  2001:4860:0:1::5708 (2001:4860:0:1::5708)  7.220 ms 2001:4860:0:1::18c0 (2001:4860:0:1::18c0)  10.561 ms 2001:4860:0:1::5700 (2001:4860:0:1::5700)  7.835 ms
 7  2001:4860:0:1::26f6 (2001:4860:0:1::26f6)  7.912 ms * *
 8  * 2001:4860::c:4001:e55b (2001:4860::c:4001:e55b)  8.715 ms *
 9  2001:4860::c:4001:f895 (2001:4860::c:4001:f895)  12.204 ms * 2001:4860::c:4002:17b1 (2001:4860::c:4002:17b1)  9.762 ms
10  2001:4860::cc:4002:c2d6 (2001:4860::cc:4002:c2d6)  7.907 ms 2001:4860::cc:4001:f891 (2001:4860::cc:4001:f891)  8.809 ms  9.031 ms
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  rs-in-f105.1e100.net (2607:f8b0:4023:1000::69)  8.449 ms * *

Work as far as I can tell.

[jenslink]

Then you should ask your provider.

[JanneKiiskila]

Then you should ask your provider.

Who should ask? People responsible for the pypi.org DNS or files.pythonhosted.org?

Or me and my colleagues? I'm not sure I understand your reply.

janne@mercury:~$ dig ns pypi.org

; <<>> DiG 9.18.18-0ubuntu0.22.04.2-Ubuntu <<>> ns pypi.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61917
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;pypi.org.			IN	NS

;; ANSWER SECTION:
pypi.org.		172647	IN	NS	ns-96.awsdns-12.com.
pypi.org.		172647	IN	NS	ns-1264.awsdns-30.org.
pypi.org.		172647	IN	NS	ns-897.awsdns-48.net.
pypi.org.		172647	IN	NS	ns-1702.awsdns-20.co.uk.

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Wed Apr 17 17:59:38 UTC 2024
;; MSG SIZE  rcvd: 173

and

$ dig ns files.pythonhosted.org

; <<>> DiG 9.18.18-0ubuntu0.22.04.2-Ubuntu <<>> ns files.pythonhosted.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55198
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;files.pythonhosted.org.		IN	NS

;; ANSWER SECTION:
files.pythonhosted.org.	86013	IN	CNAME	dualstack.python.map.fastly.net.

;; AUTHORITY SECTION:
fastly.net.		30	IN	SOA	ns1.fastly.net. hostmaster.fastly.com. 2017052201 3600 600 604800 30

;; Query time: 24 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Wed Apr 17 17:59:51 UTC 2024
;; MSG SIZE  rcvd: 157

Results for the

for i in $(dig ns fastly.net +short); do dig aaaa $i +short; done

Is nothing - both in Oulu, FIN and Texas, USA - same as yours @jenslink.

For the other one, in Oulu, FI:

$ for i in $(dig ns fastlydns.net +short); do dig aaaa $i +short; done
2a04:4e47:1::32
2a04:4e47:3::32
2a04:4e47:2::32
2a04:4e47::32

and in Texas, USA:

$ for i in $(dig ns fastlydns.net +short); do dig aaaa $i +short; done
2a04:4e47:2::32
2a04:4e47:3::32
2a04:4e47::32
2a04:4e47:1::32

Slightly different output.

[jenslink]

Then you should ask your provider.

Who should ask? People responsible for the pypi.org DNS or files.pythonhosted.org?

Or me and my colleagues? I'm not sure I understand your reply.

My first guess: Google Fiber as you can't connect from there.

You can also try running traceroute6 -I files.pythonhosted.org. Traceroute via ICMP seems to be working in this case. Standard UDP traceroute not. :-(

We have totally different problems. Mine a sever not speaking IPv6 yours looks like a routing problem.

[sdstrowes]

Hi folks; wearing my Fastly hat:

@jenslink is right, there are two separate issues being discussed here.

The first issue is that the python names are served from the fastly.net zone, and the fastly.net zone has v4-only nameserver addresses. We don’t publish v6 addresses for that zone. The fastlydns.net zone on the other hand does have dual-stacked nameservers, and we can create a new configuration in that zone that Python can point into. We've opened a ticket internally to start that work, and then Python will be able to either create a new name or point an existing name into the fastlydns zone.

The second issue does look like some intermittent routing issue with Google Fiber. I'm afraid I have no insight into the Google Fiber network, but I ran traceroutes to a few Fastly targets from that network using RIPE Atlas probes, and they have reachability.

In order to remove the ambiguities from resolving domain names, IPs that you ought to be able to reach Python's services on include:

2a04:4e42::223
2a04:4e42:200::223
2a04:4e42:400::223
2a04:4e42:600::223
2a04:4e42:1000::223
2a04:4e42:2000::223
2a04:4e42:3000::223
2a04:4e42:4000::223
2a04:4e42:5000::223
2a04:4e42:7000::223

Chances are that if traceroute -nI <addr> to any of these works, all will work. (If some consistently fail, that's interesting.) If traceroute succeeds to these addresses, then curl calls to https://python.org/ should also be good.