feat(mojibake): new rule MD083 for mojibake detection#753
Conversation
|
Thanks for this, the detection genuinely works. I built the branch and ran it over a deliberately mojibaked corpus and it caught every case, including the mixed Before it can land I need the false positives closed. Clean prose in Portuguese, Spanish, German, Greek, Russian, Danish and Icelandic is untouched, and so are symbols, maths, box drawing and typographic punctuation, so this is narrow rather than pervasive. Three things:
There is a structural reason for 1 and 2. These patterns come from ftfy's Also: One process note: no CI has run yet, since fork PRs need workflow approval. I'll approve a run once the above is addressed. |
|
@rvben 👍🏻 Thanks for the feedback, all very valid points.
|
|
Thanks, fast turnaround. Approved CI and verified against a build. Letter enumerations and the config ( CI is red on two new-rule bookkeeping items:
Two false positives still open:
Once 1-3 land and you've decided on 4, I'll merge. Happy to push the two bookkeeping fixes myself if you'd rather focus on the regex. |
|
What
Add a rule for mojibake detection, as discussed in #687
I made the rule opt-in, but it is probably worth enabling it by default. You decide @rvben :)