Repository navigation
yesod-core: gate encoding dependency off Windows - #1924
Merged
Merged
Conversation
The `encoding` package (added in #1894 for typedContentToSnippet's CJK charset decoding) fails to build on Windows: it drags in regex-compat -> regex-posix (needs POSIX regex.h, missing on the ghc-8.8 mingw64 toolchain) and itself needs system_encoding.h (missing on the ghc-9.8 clang64 toolchain). This broke the lts-16 and lts-23 Windows CI jobs. Gate the dependency behind `if !os(windows)` in all three yesod-core stanzas, pairing it with a -DWITH_DATA_ENCODING cpp flag so the source guard always matches whether the package was linked. On Windows the four CJK charsets (GB18030/windows-1251/Shift_JIS/Windows-31J) fall back to utf-8-lenient decoding in the content snippet; behavior is unchanged on all other platforms, where CJK decoding is fully preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
parsonsmatt
marked this pull request as ready for review
June 18, 2026 21:50
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move all of the WITH_DATA_ENCODING CPP into a new internal module,
Yesod.Core.Internal.ContentCharset, which exposes a CPP-free API:
lookupExtraCharsetDecoder :: ByteString -> Maybe (L.ByteString -> TL.Text)
It returns Nothing for charsets the build cannot handle (the CJK charsets
on Windows), so callers fall back to utf-8-lenient decoding without any
CPP of their own.
TypedContent.decoderForCharset now just consults that lookup via a pattern
guard, dropping its encoding imports and #ifdef branches. The Content test
becomes CPP-free too: its fixtures only ever encoded the ASCII payload
"[1,2,3]", which is byte-identical under GB18030/windows-1251/Shift_JIS/
Windows-31J and utf-8, so the CJK-label cases now run on every platform
(exercising the real decoder off-Windows and the fallback on Windows).
The 'encoding' dependency is consequently dropped from the tests stanza
(it builds against the packaged library, not src). The library and
test-routes stanzas keep the `if !os(windows)` gate, since they compile
ContentCharset from source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the Maybe: extraCharsetDecoder now returns the decoder directly, owning the utf-8-lenient fallback for unrecognized/unsupported charsets. The call site in TypedContent.decoderForCharset becomes a plain `otherwise` branch instead of a pattern guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Windows CI jobs for lts-16 and lts-23 have been failing. Both trace to a single dependency:
encoding, added in #1894 fortypedContentToSnippet's CJK charset decoding.encoding→regex-compat→regex-posix, which needs POSIXregex.h— absent in that Windows mingw64 toolchain.encodingitself needssystem_encoding.h— absent in the clang64 toolchain.The other Windows configs (lts-18/20/22) happened to build it, so it's a brittle, toolchain-dependent dependency.
Fix
Gate
encodingbehindif !os(windows)in all threeyesod-corestanzas (library,test-routes,tests), each pairing the dependency with a-DWITH_DATA_ENCODINGcpp flag so the source guard can never drift from whether the package was linked.TypedContent.hs: the four CJK charset imports and decoder branches are#ifdef-guarded; on Windows those charsets fall through to the existing utf-8-lenientotherwisebranch.Content.hs(test): the CJK imports, helpers, and four test cases are guarded the same way.CJK decoding (GB18030/windows-1251/Shift_JIS/Windows-31J) is fully preserved on Linux/macOS. Only Windows degrades to utf-8-lenient for those four charsets, and only in the debug content-snippet display — actual response serving is untouched.
Verification
yesod-corebuilds;tests374/374 andtest-routes111/111 pass.TypedContent.hsand the test typecheck cleanly withWITH_DATA_ENCODINGundefined, confirming neither referencesencodingon that path.🤖 Generated with Claude Code