fix(io): sniff BOM before plain UTF-8 so SKILL.md files with a BOM load#869
Open
abhay-codes07 wants to merge 1 commit into
Open
fix(io): sniff BOM before plain UTF-8 so SKILL.md files with a BOM load#869abhay-codes07 wants to merge 1 commit into
abhay-codes07 wants to merge 1 commit into
Conversation
A SKILL.md saved with a UTF-8 BOM decodes successfully as plain utf-8, which leaves U+FEFF at the start of the returned text. The frontmatter parser then rejects the file because the leading boundary no longer matches, so the skill is reported as invalid. Trying the BOM-derived codec first returns utf-8-sig for these files, stripping the marker while keeping round-trip writes byte-identical via the reported encoding. Fixes mistralai#701
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #701
A SKILL.md saved with a UTF-8 BOM (common when editing on Windows, Notepad and some editors add it silently) is reported as invalid with "Missing or invalid YAML frontmatter" even though the file looks perfectly fine.
Root cause:
decode_safetries plainutf-8before sniffing the BOM. A UTF-8 BOM file decodes successfully as utf-8, so the BOM survives as U+FEFF at the start of the returned text. The frontmatter splitter then sees---instead of---on the first line and rejects the file (str.strip()does not remove U+FEFF, it is not whitespace).Fix: try the BOM-derived codec first. For these files the codec is now
utf-8-sig, which strips the marker. Round-trip writes stay byte-identical because the edit path reuses the reported encoding, so the BOM is preserved on write. The utf-16/32 BOM cases behave exactly as before since plain utf-8 decoding already failed on those.This also fixes the same silent failure for any other file read through
read_safe, AGENTS.md included.Testing:
@michelTho tagging you for review when you have a moment.