[bug] markitdown._markitdown.UnsupportedFormatException

[doggy8088]

Here is the steps to reproduce the issue:

OS: Windows

Shell: PowerShell v7.4.6

pip install markitdown
curl -s https://www.duotify.com -o a.htm
type a.htm | markitdown

The error message:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Python312\Scripts\markitdown.exe\__main__.py", line 7, in <module>
  File "C:\Python312\Lib\site-packages\markitdown\__main__.py", line 38, in main
    result = markitdown.convert_stream(sys.stdin.buffer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\markitdown\_markitdown.py", line 1142, in convert_stream
    result = self._convert(temp_path, extensions, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\markitdown\_markitdown.py", line 1260, in _convert
    raise UnsupportedFormatException(
markitdown._markitdown.UnsupportedFormatException: Could not convert 'C:\Users\user\AppData\Local\Temp\tmpzn_zxenq' to Markdown. The formats [] are not supported.

[doggy8088]

I found that as long as the first few characters of the downloaded webpage are blank or line breaks, the program will crash, always showing the same error message.

It might be an issue with the logic for determining file type.

[afourney]

I am investigating. I think the problem must be lower in the stack, perhaps puremagic or similar. We shouldn't need to trim the input stream.

UPDATE: the problem is indeed here:

markitdown/src/markitdown/_markitdown.py

Line 1596 in 731b39e

guesses = puremagic.magic_file(path)

The magic_file method does not detect the file as HTML, since it doesn't start with an html or similar tag.

I would really rather not change the file (e.g, by trimming it), just so detection works. By all rights, those spaces might be meaningful in some files. I would like to investigate other approaches to addressing this problem so that any corrections are narrow

[afourney]

Fixed in #260