Skip to content

feat(mojibake): new rule MD083 for mojibake detection#753

Open
guillp wants to merge 8 commits into
rvben:mainfrom
guillp:feat/mojibake
Open

feat(mojibake): new rule MD083 for mojibake detection#753
guillp wants to merge 8 commits into
rvben:mainfrom
guillp:feat/mojibake

Conversation

@guillp

@guillp guillp commented Jul 22, 2026

Copy link
Copy Markdown

What

Add a rule for mojibake detection, as discussed in #687

I made the rule opt-in, but it is probably worth enabling it by default. You decide @rvben :)

@rvben

rvben commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Thanks for this, the detection genuinely works. I built the branch and ran it over a deliberately mojibaked corpus and it caught every case, including the mixed é/â€" families and doubly-encoded Cyrillic. Opt-in by default and FixCapability::Unfixable are both the right calls, and 45 tests is a solid base.

Before it can land I need the false positives closed. Clean prose in Portuguese, Spanish, German, Greek, Russian, Danish and Icelandic is untouched, and so are symbols, maths, box drawing and typographic punctuation, so this is narrow rather than pervasive. Three things:

  1. [Œœ][^A-Za-z] fires on ordinary French. Any œ or Œ not followed by an ASCII letter matches, so a sentence discussing the ligature itself trips it: La ligature œ est utilisée, Le caractère œ se compose, and the same for the capital. Four hits in one page of normal French typography prose.
  2. [a-z]\s?[ÃÂ][\s] fires on letter enumerations. Romanian sunt cuvinte cu  și Î and Vietnamese nguyên âm có dấu mũ: a  e Ê o Ô both match. Naming a letter is a normal thing to do in a document about that language.
  3. No code-block exemption. Mojibake is flagged inside inline code spans and fenced and indented blocks, which is a problem for exactly the documents that discuss encoding. The clearest evidence is that this PR's own docs/md083.md trips MD083 ten times.

There is a structural reason for 1 and 2. These patterns come from ftfy's BADNESS_RE, where they are weighted contributions to a score compared against a threshold. Ported to binary matching, a single weak signal becomes a finding on its own, so the false positive rate goes up by construction. Either restore something threshold-like, or drop the individual patterns that cannot carry a finding alone.

Also: from_config ignores the config, so MD083 has no options. Since the rule is heuristic, users need an escape hatch narrower than disabling it outright.

One process note: no CI has run yet, since fork PRs need workflow approval. I'll approve a run once the above is addressed.

@guillp

guillp commented Jul 23, 2026

Copy link
Copy Markdown
Author

@rvben 👍🏻 Thanks for the feedback, all very valid points.

  • about french œ : I should know, I'm french 🇫🇷 . I edited the pattern [Œœ][^A-Za-z] to allow a more characters after the ligature than just letters, i.e. spaces and most kinds of quotes.
  • about single letters: I removed the pattern [a-z]\s?[ÃÂ][\s] completely.
  • I made the rule configurable with both an allow-list of ignored mojibake, and an option to ignore code blocks, that is enabled by default.

@rvben

rvben commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Thanks, fast turnaround. Approved CI and verified against a build. Letter enumerations and the config (ignore + ignore-code-blocks) are both solid now.

CI is red on two new-rule bookkeeping items:

  1. Rule-count docs out of sync (MD083 makes it 77 rules, docs still say 76, RULE_MAX still MD082). make sync-rule-docs updates the sentinels; then add the MD083 rows to docs/rules.md by hand.
  2. test_all_53_rules_systematic_coverage fails: MD083 has no arm in get_test_content_for_rule (tests/cli/cli_lsp_fix_consistency.rs), tripping assert!(test_content_missing < 15). Add e.g. "MD083" => Some("Broken dash – here"),.

Two false positives still open:

  1. French œ: [Œœ][^A-Za-z\s"'»”’] still fires before ordinary punctuation, e.g. la ligature (œ), œ,, œ., œ;. Extending the exclusion set will keep leaking. Since real œ mojibake only appears inside “ (already caught elsewhere), I'd drop the standalone œ pattern rather than enumerate trailing chars. Your call.
  2. Inline code spans aren't exempted (only blocks are), so `Café` still flags and docs/md083.md itself trips 9 times. Exempt inline code, or move the doc examples into fenced blocks.

Once 1-3 land and you've decided on 4, I'll merge. Happy to push the two bookkeeping fixes myself if you'd rather focus on the regex.

@guillp

guillp commented Jul 24, 2026

Copy link
Copy Markdown
Author
  1. fixed (please review the opt-in rationale in docs/rules.md)
  2. fixed
  3. I removed the oe specific pattern
  4. I am actually not sure how to exclude inline code blocks. Is there some built-in features in Rumdl LintContext to obtain those code blocks, or should I detect them myself in the rule ? This looks a bit too complex when I look at rule MD048...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants