Fix multibyte character corruption in post summaries #1995
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes character encoding issues where Greek text like "σου" was being corrupted to "σ?" in generated post summaries. The root cause was the sequence of html_entity_decode() followed by wp_strip_all_tags() which doesn't handle multibyte characters correctly. Changes: - Reorder text processing to use wp_strip_all_tags() before html_entity_decode() - Replace byte-based string functions with multibyte-safe equivalents (mb_strlen, mb_substr, etc.) - Add proper UTF-8 encoding parameters throughout the text processing pipeline - Improve word boundary detection for better truncation points - Add test case for Greek text to prevent regressions
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
Thanks for the feedback! I've addressed the redundant condition issue. Regarding the magic numbers (12, 0.4, 0.8): I'm keeping these as inline values rather than constants because:
Constants would be more appropriate if these values needed to be configurable or reused across multiple functions. |
The shortcode test was expecting truncation at 'dolor' but the improved algorithm correctly includes 'sit' within the 25-character limit, providing better text truncation behavior.
Keep the essential encoding fixes (reorder processing, use multibyte functions) but revert to the simpler wordwrap-based truncation approach. The complex word boundary detection was an improvement but not necessary to fix the original character corruption issue.
There was a problem hiding this comment.
Pull Request Overview
This PR addresses multibyte character corruption in post summaries by replacing byte-based string functions with multibyte-safe equivalents throughout the codebase. The fix ensures that Greek text and other non-ASCII characters are properly handled during text processing operations.
- Replaces
strlen()withmb_strlen()for accurate character counting - Reorders HTML entity decoding and tag stripping operations for proper encoding handling
- Adds UTF-8 encoding parameters to
html_entity_decode()calls - Updates regex patterns with Unicode modifiers for multibyte character support
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| includes/functions.php | Core fix in generate_post_summary() function replacing byte-based operations with multibyte-safe equivalents |
| includes/transformer/class-post.php | Updates HTML entity decoding in image processing methods to preserve UTF-8 encoding |
| tests/includes/class-test-functions.php | Adds Greek text test case to verify multibyte character handling |
| tests/includes/class-test-shortcodes.php | Updates expected test output to reflect corrected excerpt behavior |
| .github/changelog/1995-from-description | Adds changelog entry documenting the bug fix |
pfefferle
reviewed
Jul 24, 2025
pfefferle
previously approved these changes
Jul 24, 2025
Co-authored-by: Matthias Pfefferle <[email protected]>
pfefferle
previously approved these changes
Jul 24, 2025
pfefferle
reviewed
Jul 24, 2025
Co-authored-by: Matthias Pfefferle <[email protected]>
pfefferle
approved these changes
Jul 24, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1948
Proposed changes:
generate_post_summary()function by reordering text processing operations.Other information:
Testing instructions:
Τι μπορεί να σου συμβεί σε μια βόλτα για να αγοράσεις μια βαλίτσα για τα ταξίδια σου; Όλα είναι πιθανά αν έχεις ανοιχτές τις "κεραίες" σου\!npm run env-test -- --filter test_generate_post_summaryChangelog entry
Changelog Entry Details
Significance
Type
Message
Fix multibyte character corruption in post summaries, preventing Greek and other non-ASCII text from being garbled during text processing.