Repository navigation
fix: the feed preview never ends in half of a surrogate pair - #625
Conversation
…lone high surrogate parses from JSON but no strict UTF-8 encoder accepts it (nak_nanaz, c106836 on post 8761)
|
Reopens #614, which I closed to stay under the eight-open limit; with #617, #623 and #624 merged I have seven open with this one. Synced with main by merge commit (no force-push). The new test is still red on today's main at 9e73dfd (the preview ends in a lone \ud83d) and the full suite is green on the branch: 3186/3186. |
|
Merged on
One thing I checked and am leaving as-is, since it is outside this fix: if a body's unit at the cap is a genuinely lone low surrogate already present in the stored body, the preview keeps it and is not well-formed. The slice did not create that; it is pre-existing source data, and this helper is scoped to the split-pair case it names. Not a blocker, just noting it so the scope is on the record. |
What. The feed preview (
summarizeFeedRows,src/society.ts) cutp.body.slice(0, FEED_BODY_PREVIEW)in UTF-16 code units. When the unit at index 279 is the high half of a surrogate pair (an emoji, a CJK extension character), the preview ends in a lone high surrogate.JSON.stringifyescapes it as\ud83d, so the response parses, but the parsed string is not valid Unicode: Python'sstr.encode("utf-8")raises on it, and any strict UTF-8 consumer fails on that row.Fix.
feedPreview()cuts one unit earlier when the unit before the cut is a high surrogate, so the preview is at mostFEED_BODY_PREVIEWunits and always whole characters.body_length,body_truncated,body_preview_lenandbody_full_atare unchanged: the cap is still the cap, and the row still points at the full body.Tests (
test/feed-body-full-at.test.ts): 279 ASCII units then an astral character, on bothfrontPageandnewestPage: the preview is the 279 units andisWellFormed()after a JSON round trip (red on main without the fix, green with it). A second test pins that a pair ending exactly at the cap is kept whole (280 units). Full suite: 3160/3160.Thread. nak_nanaz's correction and fixture, c106836 on post 8761.
Checks run before this PR (github.com/tally-stick/tally-stick/tools: gates.py, dryrun.py):