fix(markdown): add support for HTML content

[vagenas]

Addresses #734.

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

🟢 Require two reviewer for test updates

Wonderful, this rule succeeded.

When test data is updated, we require two reviewers

• #approved-reviews-by >= 2

[PeterStaar-IBM]

🚀

[dolfim-ibm]

I suggest making a reminder issue about html features which cannot be converted to markdown, i.e. we will have to deal with tables having merged cells, which are only possible via html code blocks.

[vagenas]

I suggest making a reminder issue about html features which cannot be converted to markdown, i.e. we will have to deal with tables having merged cells, which are only possible via html code blocks.

@dolfim-ibm can you elaborate? Unless you are you referring to Markdown export in general, this PR does not involve HTML-to-Markdown; what it does (when needed) is Markdown-to-HTML & then HTML parsing, in order to streamline HTML content processing.

[dolfim-ibm]

I suggest making a reminder issue about html features which cannot be converted to markdown, i.e. we will have to deal with tables having merged cells, which are only possible via html code blocks.

@dolfim-ibm can you elaborate? Unless you are you referring to Markdown export in general, this PR does not involve HTML-to-Markdown; what it does (when needed) is Markdown-to-HTML & then HTML parsing, in order to streamline HTML content processing.

I'm referring to Markdown documents using html <table> to represent tables which are not supported by the native markdown format, e.g. with merged cells.

We would enhance the export to markdown in the DoclingDocument with an option to potentially output html tables if we detect that the markdown format would be "too lossy". Something like tables_as_html: bool = False.