Fix pdf parsing bug

[markenki]

The line
text = page.get_text("text", sort=True)
in readers.py doesn't respect multiple columns. For example, applied to pasa.pdf (in tests/stub_data), the first line of text is extracted as "We introduce PaSa, an advanced Paper Search Academic paper search lies at the core of research" but the first half of that comes from the first column while the second half comes from the second column.

Replacing that line of code with

# Extract text blocks, which are already in the correct order, from the page
blocks = page.get_text("blocks", sort=False)

# Concatenate text blocks into a single string
text = "\n".join(block[4] for block in blocks)

extracts this text: "We introduce PaSa, an advanced Paper Search\nagent powered by large language models.", which is correct.

[Copilot]

Pull Request Overview

This pull request fixes a PDF parsing bug by updating the text extraction logic to correctly handle multi-column layouts.

• Replaces the use of page.get_text("text", sort=True) with a blocks-based extraction
• Concatenates text blocks in the order provided by the PDF parser

[jamesbraza]

What do we lose with sort=False? I am wondering why we had sort=True originally (it predates my time at FutureHouse)

[markenki]

The problem wasn't sort=True. The problem was getting "text" rather than "blocks".

[jamesbraza]

If the problem isn't sort=False, can you revert to sort=True there? Just to keep diff smaller

[markenki]

Using sort=False retains the correct order of blocks in two-column pdfs (as well as one-column pdfs).

[jamesbraza]

We'll need lint workflow and tests to pass. Our CI doesn't work for contributors since the OpenAI API key secret doesn't propagate outside of the FutureHouse org.

That being said, looks like failing tests (locally for me) are:

• test_get_directory_index
• test_get_directory_index_w_manifest

Can you:

1. Get these to pass (adjusting the assertions)
2. Expand them to account for pasa.pdf

[jamesbraza]

If the problem isn't sort=False, can you revert to sort=True there? Just to keep diff smaller

[markenki]

We'll need lint workflow and tests to pass. Our CI doesn't work for contributors since the OpenAI API key secret doesn't propagate outside of the FutureHouse org.

That being said, looks like failing tests (locally for me) are:

• test_get_directory_index
• test_get_directory_index_w_manifest

Can you:

1. Get these to pass (adjusting the assertions)
2. Expand them to account for pasa.pdf

Thanks, @jamesbraza. I fixed the failing unit tests.

[markenki]

@jamesbraza could you kick off the workflow, please. Thanks!