Summary
Idna.domainToAsciiForUrl (via its asciiLenientDomainToAscii helper) decides, label by label,
whether an all-ASCII label gets kept verbatim (the WHATWG web-compat leniency for an invalid xn--
label) or has to run the full UTS-46 pipeline. That decision is made by splitting the domain on a
literal . before UTS-46 mapping runs.
The UTS-46 mapping table also maps three other code points to . (U+002E):
- U+3002 IDEOGRAPHIC FULL STOP
- U+FF0E FULLWIDTH FULL STOP
- U+FF61 HALFWIDTH IDEOGRAPHIC FULL STOP
These are legitimate, browser-accepted domain-label separators — that's exactly why the mapping
table maps them to . in the first place. Because the per-label split runs on the raw text before
that mapping happens, a label boundary formed by one of these three characters is invisible to the
leniency decision: the two labels either side of it get merged into a single non-ASCII "label" by
the naive splitter and handed to the strict pipeline as one unit, which fails outright the moment it
hits an invalid xn-- label instead of applying the intended per-label leniency to it.
Trigger
// literal ASCII dot: per-label leniency applies correctly
Idn.toAscii("xn--a.bücher") // => Ok(xn--a.xn--bcher-kva)
// ideographic full stop (U+3002) instead of '.': leniency doesn't kick in
Idn.toAscii("xn--a。bücher") // => Err(InvalidHost(..., IdnaFailed))
xn--a is invalid Punycode either way; the only difference is which separator joins it to the
neighboring Unicode label. The first case is the correctly-fixed one; the second rejects the whole
domain over the same invalid label, which is the exact failure mode the per-label leniency is meant
to avoid when a sibling label is non-ASCII.
Where it comes from
Idna.asciiLenientDomainToAscii splits on LABEL_SEPARATOR (.) directly over the raw domain
parameter, before any UTS-46 mapping has run. domainToAscii (the strict pipeline it delegates
non-ASCII spans to) does the equivalent split correctly, because by the time it calls
splitLabels, mapAll has already turned any of the three variants above into a literal .. The
mismatch is that the outer per-label ASCII/non-ASCII gate runs before that normalization, while the
inner strict-path split runs after it.
This isn't a new regression — the pre-existing whole-domain-ASCII gate had the identical outcome for
this input (a domain containing any non-ASCII text, including these separators, went through the
strict pipeline with no leniency at all). It's a gap in the per-label leniency fix's completeness for
these three characters specifically.
Proposal
Normalize (or at least recognize) the full UTS-46 label-separator set — U+002E, U+3002, U+FF0E, and
U+FF61 — when deciding label boundaries for the leniency decision, rather than splitting on . only
before mapping has run. The simplest fix is probably to run mapAll over the domain first and split
on the result, mirroring what domainToAscii already does internally.
Summary
Idna.domainToAsciiForUrl(via itsasciiLenientDomainToAsciihelper) decides, label by label,whether an all-ASCII label gets kept verbatim (the WHATWG web-compat leniency for an invalid
xn--label) or has to run the full UTS-46 pipeline. That decision is made by splitting the domain on a
literal
.before UTS-46 mapping runs.The UTS-46 mapping table also maps three other code points to
.(U+002E):These are legitimate, browser-accepted domain-label separators — that's exactly why the mapping
table maps them to
.in the first place. Because the per-label split runs on the raw text beforethat mapping happens, a label boundary formed by one of these three characters is invisible to the
leniency decision: the two labels either side of it get merged into a single non-ASCII "label" by
the naive splitter and handed to the strict pipeline as one unit, which fails outright the moment it
hits an invalid
xn--label instead of applying the intended per-label leniency to it.Trigger
xn--ais invalid Punycode either way; the only difference is which separator joins it to theneighboring Unicode label. The first case is the correctly-fixed one; the second rejects the whole
domain over the same invalid label, which is the exact failure mode the per-label leniency is meant
to avoid when a sibling label is non-ASCII.
Where it comes from
Idna.asciiLenientDomainToAsciisplits onLABEL_SEPARATOR(.) directly over the rawdomainparameter, before any UTS-46 mapping has run.
domainToAscii(the strict pipeline it delegatesnon-ASCII spans to) does the equivalent split correctly, because by the time it calls
splitLabels,mapAllhas already turned any of the three variants above into a literal.. Themismatch is that the outer per-label ASCII/non-ASCII gate runs before that normalization, while the
inner strict-path split runs after it.
This isn't a new regression — the pre-existing whole-domain-ASCII gate had the identical outcome for
this input (a domain containing any non-ASCII text, including these separators, went through the
strict pipeline with no leniency at all). It's a gap in the per-label leniency fix's completeness for
these three characters specifically.
Proposal
Normalize (or at least recognize) the full UTS-46 label-separator set — U+002E, U+3002, U+FF0E, and
U+FF61 — when deciding label boundaries for the leniency decision, rather than splitting on
.onlybefore mapping has run. The simplest fix is probably to run
mapAllover the domain first and spliton the result, mirroring what
domainToAsciialready does internally.