Pdf to Markdown duplication issue

[AbzyS1]

Question

I am new to Docling and have been experimenting. This is how I have currently implemented Docling to parse my PDF:

```python
from docling.datamodel.base_models import InputFormat
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.pipeline_options import PdfPipelineOptions

pipeline_options = PdfPipelineOptions(force_ocr=True)
converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
    }
)
source = "data/my-document.pdf"
result = converter.convert(source)
print(result.document.export_to_markdown())

The issue is in the output, every sentence is duplicated.

Here is an example of what I mean:

4. Characteristics of Investment Agency 4. Characteristics of Investment Agency
* 4/1 Investment agency contracts, whether remunerated or unremuner4/1 Investment agency contracts, whether remunerated or unremunerated, are binding on institutions because they are invariably fixed ated, are binding on institutions because they are invariably fixed term contracts in which both parties agree not to terminate within term contracts in which both parties agree not to terminate within a specified period.
* 4/2 Where the parties agreed to terminate for a specified period, it is 4/2 Where the parties agreed to terminate for a specified period, it is permissible for the contract to stipulate the right of one of the parties permissible for the contract to stipulate the right of one of the parties to terminate the contract unilaterally in specific circumstances. to terminate the contract unilaterally in specific circumstances.
* 4/3 When the term of an agency expires, the agent is required not to enter 4/3 When the term of an agency expires, the agent is required not to enter into new investment activities, but may not liquidate ongoing existing into new investment activities, but may not liquidate ongoing existing investments. investments.

As you can see, it happens in both titles and the text itself. Any ideas why? I have tried a few things (i.e., changing force_ocr = False, etc.) but can't seem to pinpoint the issue.

All help would be appreciated.

[vagenas]

@AbzyS1 I ran your snippet against a document that matched your pasted content on Google (I assume it's the same, otherwise please share your document if it is possible) and it works without duplication issues.

Can you share the outputs of docling --version and python -c "import platform; print(platform.platform())"?

In case you have an old Docling version, I would update to the latest and try again.

[PeterStaar-IBM]

@AbzyS1 Can you add an example pdf where this happens?

[AbzyS1]

Shariaa-Standards-extract-ENG.pdf

The above is an extract from the document in question.

Following from Vagenas's questions here are the outputs:

Docling version: 2.8.3
Docling Core version: 2.14.0
Docling IBM Models version: 2.0.8
Docling Parse version: 2.1.2

---

Python 3.11.5

---

macOS-14.5-arm64-arm-64bit

[PeterStaar-IBM]

@AbzyS1 Thank you, I can reproduce it and know how to fix it (the bold is done by replicating text), fix coming in soon!

[PeterStaar-IBM]

This is resolved here: docling-project/docling-parse#91