Missing character when saving after using multiple add_redact_annot & insert_htmlbox

[Nader-Khalil]

Description of the bug

Hello there,
I'm trying to use the library for pdf translation process,
I managed to do lots of stuff but i stuck in a problem
This is the an original page of a pdf that contains 9 pages :

This is the result after translation to Arabic :
you can notice lots of missing characters in a lot of words [ I already pointed to some with a red arrow ]

so after some tracing and trials, i tried to make the pdf with this page only to make the tracing more simple :
and the surprise the letters appear! 😅 not all There are still 2 missing but lots have been shown correctly now
you can note the correction in the middle squares compared to the previous image

I need help handling this, whenever the page contains fewer blocks it goes well, but when the page contains lots of blocks the problem occurs ,

I have also another question the pdf after saving had its size increased alot is there something to do to reduce this to be close to the orginal size ?? cause it's more than x10

Thanks in advance

How to reproduce the bug

i'm using python code in these steps :

python in Google Colab

Define translate_sent(sent, source, trgt) function:
Translate text from source language to target language

Define check_table(page, text, bbox) function:
Determine if the specified text in a bbox is a table by checking alignment

Define process_table_text(page, bbox, text, src_lang, trgt_lang, css_style) function:
## Can be Neglected for now nothing wrong with it
Translate and replace table text on the page within the specified bbox using provided styles

The exact function to handle the page is as follows:

def process_page(page, src_lang, trgt_lang):
    modifications = []
    for block in page.get_text("dict")["blocks"]:
        if block["type"] == 0:  # Text block
            block_text = "\n".join(''.join(span['text'] for span in line['spans']) for line in block['lines']).strip()
            if block_text:
                bbox = block['bbox']
                is_table = check_table(page, block_text, bbox)
                color = block.get('lines', [{}])[0].get('spans', [{}])[0].get('color', 0)
                size = block.get('lines', [{}])[0].get('spans', [{}])[0].get('size', 12)  # Default size
                font = block.get('lines', [{}])[0].get('spans', [{}])[0].get('font', 'Arial')  # Default font
                color_css = int_to_css_color(color)
                css_style = f"* {{font-size: {size}px; font-family: '{font}'; color: {color_css}; direction: rtl; text-align: right;}}"
                modifications.append((bbox, block_text, is_table, css_style))

    for bbox, text, is_table, css_style in modifications:
        translated_text = translate_sent(sent=text, source=src_lang, trgt=trgt_lang)
        print(translated_text)
        print("******")

        if is_table:
            process_table_text(page, bbox, text, src_lang, trgt_lang, css_style)
        else:
            page.add_redact_annot(bbox)
            page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE)
            page.insert_htmlbox(bbox, translated_text, css=css_style)
            page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE)

for running :

def run_pdf_page_range(path, trgt_lang, src_lang, start_page, end_page, edit=False):
    doc = fitz.open(path)
    
    for page_number in range(start_page, end_page + 1):
        if 0 <= page_number < len(doc):
            process_page(doc.load_page(page_number), src_lang, trgt_lang)
        else:
            print(f"Invalid page number: {page_number}")

    if edit:
        doc.save("replaced.pdf", garbage=3, deflate=True)

PyMuPDF version

1.23.26

Operating system

Other

Python version

3.10

[JorjMcKie]

Highly complex post - and apparently not a bug, but a call for help.
Transferring this to the "Discussions" tab.

[JorjMcKie]

1. I do not understand why you execute apply_redactions() two times with insert_htmlbox in between:
   The first one already removes text and all redaction annotations.

2. Try not to supply any font like Arial. Use all the defaults and clean the result. Then continue ...

[Nader-Khalil]

I appreciate your help thanks , but
1 - it was an extra trial from my side to add apply_redaction() after the insert , but even if i removed it nothing changed the characters still missing, if you look at the two squares in the middle in the last two images you can note the different, the third one has the full charterers shown, but the second has a lot of missing characters
The only difference between producing the second screenshot and the third is just for the third i created a pdf contains only this page not but the second the page was in a middle of pdf containing lots of pages
2 - I tried this before also I already commented the whole line of the Css styling , and nothing changes [ btw the line just to preserve the original fonts and formatting and the Arial is just a default value if no font found ]

can you share with me an email and I can share the whole notebook with you if you want to run it?

[JorjMcKie]

I do not have the time to deal with details of your script.
Can you instead please share an example of Arabian text that is not correctly written by insert_htmlbox?