Add PDFParser

[merveenoyan]

This PR adds PDFParsingTool with GOT-OCR 2.0.

Note that this model isn't yet integrated with transformers (there's a PR for it), I don't know when it will be merged hence loading with trust_remote_code and not using PipelineTool. (they implemented custom methods to do forward that abstracts away a bunch of things, I used it)
How should we take file inputs? Now we only take path.
An issue with this model is that it comes with two extra dependencies: tiktoken and verovio.
We could add extra dependencies for this tool like pip install smolagents["pdf"].

WDYT? @aymeric-roucher

[merveenoyan]

Partially fixes #176

[aymeric-roucher]

@merveenoyan thank you for this addition!
Does this tool require additional imports?
Also, couldn't the tool name be PdfReader to be more explicit?

[merveenoyan]

I will test this with GOT-OCR and report here

[merveenoyan]

@aymeric-roucher I wrapped it up. I'm a bit confused about a proper way of giving agent the PDF, i.e. should it be PIL, should it be PDF, etc. model itself handles things in PIL/numpy etc so I went with that.

agent gets a bit confused despite giving AgentImage as type 😮‍💨🥱
Here's how I test:

from smolagents.default_tools import PDFParsingTool
from smolagents import CodeAgent, HfApiModel
from PIL import Image
from pdf2image import convert_from_path

pil_image_lst = convert_from_path("/home/merve/smolagents/smolagents/test.pdf")
pil_image = pil_image_lst[0]

pdf_tool = PDFParsingTool()

model = HfApiModel("Qwen/Qwen2.5-Coder-32B-Instruct")
agent = CodeAgent(tools=[pdf_tool], model=model, 
                  additional_authorized_imports=["math, PIL, pdf2image"])

agent.run(
    "Given a PDF file opened as PDF image, use PDF parsing tool to render pillow image it into markdown, and answer this question: How much is emissions in total in Europe. Use math to sum up the numbers of member states if necessary. Do not use web search.",
    additional_args={"image": pil_image}
)

here's the doc: https://www.europarl.europa.eu/pdfs/news/expert/2018/7/story/20180706STO07407/20180706STO07407_en.pdf

agent expects path:

 ─ Executing this code: ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
  markdown_content = pdf_parser(image="path/to/local/document.pdf")                                                                                                                                     
  print(markdown_content)                                                                                                                                                                               
 ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
[Errno 2] No such file or directory: 'path/to/local/document.pdf'
[Step 0: Duration 27.79 seconds| Input tokens: 2,135 | Output tokens: 155]

so my question is, how do we force the agent without prompting? seems brittle
it's a good doc to test because agent has to use the tool to parse the charts inside doc and then pass to LLM.

[aymeric-roucher]

Actually it's also confusing to me! Given that pdf is a file format, talking about pdf feels to me as if it hints to having a local file. If this tools takes images as input, could you not rename it ImageParser? Or OCRTool?

[merveenoyan]

@aymeric-roucher sure let me try

[merveenoyan]

here's what I do

pil_image = ...
ocr_tool = OCRTool()

model = HfApiModel("Qwen/Qwen2.5-Coder-32B-Instruct")
agent = CodeAgent(tools=[ocr_tool], model=model, 
                  additional_authorized_imports=["math, PIL, pdf2image"])

agent.run(
    "Given a PDF file opened as pillow image, use OCR tool to render pillow image it into markdown, and answer this question: How much is emissions in total in Europe. Use math to sum up the numbers of member states if necessary. Do not use web search.",
    additional_args={"image": pil_image}
)

now I don't get any errors, but agent makes up data and attempts to answer from there lol

[merveenoyan]

going back to debugging let's see lol

[merveenoyan]

I just noticed I think the issue was rather a bug on my end, fixed it. now only issue is that I need to handle multiple images and multiple decoded text. perhaps makes sense to store in state

[aymeric-roucher]

@merveenoyan I thought that this model needed additional pip installs, is that not the case?