Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update: change pdf text parser to pymupdf4llm #139

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

update: change pdf text parser to pymupdf4llm #139

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b3f7e00
update: change pdf text parser to pymupdf4llm
tungsten106 Dec 19, 2024
df5f14e
Merge remote-tracking branch 'origin/main' into dev
tungsten106 Dec 23, 2024
263e0b5
update: add parameter "method" for PdfConverter
tungsten106 Dec 23, 2024
d4d11a8
Merge branch 'main' into dev
tungsten106 Dec 24, 2024
797e0d4
Merge branch 'dev' of https://github.com/tungsten106/markitdown into dev
tungsten106 Dec 24, 2024
ba5df9b
update: changed "method" parameter fro PdfConverter to "pdf_engine" f…
tungsten106 Dec 24, 2024
e808548
update: adding named parameter `pdf_engine` to .conver(); adding tes…
tungsten106 Dec 25, 2024
565ef05
update: Add`engine_kwargs` for customize parameters. Update PdfConver…
tungsten106 Dec 26, 2024
bd95fb0
update: adding `tests/out` folder to save test files
tungsten106 Dec 26, 2024
8482477
update: use smaller test-pdf size
tungsten106 Dec 26, 2024
f07ea3e
bugfix: fixing test_markitdown.py; updating exception messages
tungsten106 Dec 27, 2024
a1766c5
update: cli options added for engine selection
tungsten106 Jan 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ dependencies = [
"pandas",
"openpyxl",
"pdfminer.six",
"pymupdf4llm",
"puremagic",
"pydub",
"youtube-transcript-api",
Expand Down
25 changes: 17 additions & 8 deletions src/markitdown/_markitdown.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
import traceback
import zipfile
from xml.dom import minidom
from typing import Any, Dict, List, Optional, Union
from typing import Any, Dict, List, Optional, Union, Literal
from pathlib import Path
from urllib.parse import parse_qs, quote, unquote, urlparse, urlunparse
from warnings import warn, resetwarnings, catch_warnings
Expand All @@ -24,6 +24,7 @@
import pandas as pd
import pdfminer
import pdfminer.high_level
import pymupdf4llm
import pptx

# File-format detection
Expand Down Expand Up @@ -676,19 +677,27 @@ def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]:

class PdfConverter(DocumentConverter):
"""
Converts PDFs to Markdown. Most style information is ignored, so the results are essentially plain-text.
Converts PDFs to Markdown. Most style information is ignored, so the results are essentially plain-text.
"""

def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]:
def convert(self, local_path, pdf_engine: Literal['pdfminer', 'pymupdf4llm']='pdfminer', **kwargs) -> Union[None, DocumentConverterResult]:
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think pdf_engine should be a named parameter in the function instead of being accessed via kwargs. This provides more clarity and ensures better handling of default values

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for you suggestion. I have added it to a named parameters ( I was meant to align with other converters' parameter definition), and added the exception case when pdf_engine is not valid. The new test cases could be seen on my new commit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is optional, but adding one more named parameter could make this more customizable, like:

def convert(self, local_path, engine: Literal['pdfminer', 'pymupdf4llm']='pdfminer', engine_kwargs=None, **kwargs) -> Union[None, DocumentConverterResult]:

Example:
>>> source = "https://arxiv.org/pdf/2308.08155v2.pdf"
>>> markitdown.convert(source, pdf_engine="pymupdf4llm")
"""
# Bail if not a PDF
extension = kwargs.get("file_extension", "")
if extension.lower() != ".pdf":
return None

return DocumentConverterResult(
title=None,
text_content=pdfminer.high_level.extract_text(local_path),
)
if pdf_engine == "pdfminer":
text_content = pdfminer.high_level.extract_text(local_path)
elif pdf_engine == "pymupdf4llm":
text_content = pymupdf4llm.to_markdown(local_path, show_progress=False)
else:
# return None # unknown method
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this

raise FileConversionException("'pdf_engine' not valid. Please choose between ['pdfminer', 'pymupdf4llm'].")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest checking the engine first. You could use _engines to define the allowed engines

_engines: Mapping[str, Any] = {
        "pdfminer": pdfminer,
        "pymupdf4llm": pymupdf4llm,
    }
    
###
if engine is not None and engine not in self._engines:
    raise

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a good idea for the ease of adding further engines. I have updated those changes in the latest commit. An example of using engine_kwargs for pdf imeges extraction could also be found in the test file.

return DocumentConverterResult(title=None, text_content=text_content)


class DocxConverter(HtmlConverter):
Expand Down
Binary file added tests/test_files/2308.08155v2.pdf
View file
Open in desktop
Binary file not shown.
18 changes: 17 additions & 1 deletion tests/test_markitdown.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
import requests

from warnings import catch_warnings, resetwarnings

import sys
sys.path.insert(0, "/home/yxl/Projects/markitdown/src")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think don't need this. Let me know if you need help setting up the test!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will remove this, it won't be used

from markitdown import MarkItDown

skip_remote = (
Expand Down Expand Up @@ -299,6 +300,20 @@ def test_markitdown_llm() -> None:
for test_string in ["red", "circle", "blue", "square"]:
assert test_string in result.text_content.lower()

def test_markitdown_pdf() -> None:
markitdown = MarkItDown()

# I test by local pdf, using PDF_TEST_URL may also be fine.

# By pymupdf4llm
result = markitdown.convert(os.path.join(TEST_FILES_DIR, "2308.08155v2.pdf"), pdf_engine="pymupdf4llm")
for test_string in PDF_TEST_STRINGS:
assert test_string in result.text_content

# By pdfminer
result = markitdown.convert(os.path.join(TEST_FILES_DIR, "2308.08155v2.pdf"), pdf_engine="pdfminer")
for test_string in PDF_TEST_STRINGS:
assert test_string in result.text_content

if __name__ == "__main__":
"""Runs this file's tests from the command line."""
Expand All @@ -307,3 +322,4 @@ def test_markitdown_llm() -> None:
test_markitdown_exiftool()
test_markitdown_deprecation()
test_markitdown_llm()
test_markitdown_pdf()