fix(cli): decode manual ! command output with the OEM code page fallback on Windows#871
Open
abhay-codes07 wants to merge 1 commit into
Open
fix(cli): decode manual ! command output with the OEM code page fallback on Windows#871abhay-codes07 wants to merge 1 commit into
abhay-codes07 wants to merge 1 commit into
Conversation
Output of manual ! commands was decoded as hardcoded UTF-8 with errors=replace. On Windows, console programs emit the OEM code page (cp949 on Korean systems, cp850/cp437 on many European ones), so any non-ASCII output rendered as replacement characters, for example garbled Korean cmd error messages. Add IncrementalSafeDecoder, a streaming counterpart of decode_safe: it decodes optimistically as UTF-8 and, on the first invalid byte, switches to the Windows OEM code page (or the locale encoding) without losing bytes buffered across chunk boundaries. The one-shot bash tool already resolves encodings this way via decode_safe(from_subprocess=True); this brings the streaming path in line. Fixes mistralai#777
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #777
Output of manual
!commands in the TUI is decoded as hardcoded UTF-8 witherrors="replace". Windows console programs emit the OEM code page instead (cp949 on Korean systems, cp850/cp437 on many others), so any non-ASCII output renders as replacement characters. The report in #777 shows a cmd error message in Korean turning into garbage.The one-off bash tool already solved this: it decodes through
decode_safe(..., from_subprocess=True), which prefers the Windows OEM code page. The streaming!path just never got the same treatment.This adds
IncrementalSafeDecodertovibe.core.utils.io, a streaming counterpart ofdecode_safe: it decodes optimistically as UTF-8 and on the first invalid byte switches permanently to the OEM code page (or the locale encoding) with undecodable bytes replaced. The strict UTF-8 incremental decoder leaves its internal buffer untouched when it raises, so bytes buffered across chunk boundaries are re-decoded by the fallback codec and nothing is lost at the switch point. On systems where the locale is UTF-8 the behavior is identical to before._bash_read_streamin the TUI now uses it withfrom_subprocess=True.Testing:
Verified on a Windows 11 machine. @elGrogz tagging you for a review pass when convenient.