Image export fails with Chromium 132

[mschoettle]

Somewhat related to #211, with alpine3.21 chromium v132 is now installed which fails with the following error:

Chromium 132.0.6834.83 Alpine Linux
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<Browser._open_async() done, defined at /usr/local/lib/python3.12/site-packages/choreographer/browser.py:246> exception=BrowserFailedError('The browser seemed to close immediately after starting. Perhaps adding debug_browser=True will help.')>
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/choreographer/browser.py", line 271, in _open_async
    await self.populate_targets()
  File "/usr/local/lib/python3.12/site-packages/choreographer/browser.py", line 603, in populate_targets
    response = await self.browser.send_command("Target.getTargets")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
choreographer.browser.BrowserClosedError: Command not completed because browser closed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/choreographer/browser.py", line 274, in _open_async
    raise BrowserFailedError("The browser seemed to close immediately after starting. Perhaps adding debug_browser=True will help.") from e
choreographer.browser.BrowserFailedError: The browser seemed to close immediately after starting. Perhaps adding debug_browser=True will help.
Traceback (most recent call last):
  File "/app/test.py", line 4, in <module>
    fig.write_image("/output/figure.png")
  File "/usr/local/lib/python3.12/site-packages/plotly/basedatatypes.py", line 3827, in write_image
    return pio.write_image(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/plotly/io/_kaleido.py", line 266, in write_image
    img_data = to_image(
               ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/plotly/io/_kaleido.py", line 143, in to_image
    img_bytes = scope.transform(
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kaleido/scopes/plotly.py", line 235, in transform
    img = kaleido.to_image_block(spec, Path(self._tempfile.name).absolute(), self._topojson, self._mapbox_access_token, debug=debug, tmp_path=self._tmp_path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kaleido/__init__.py", line 43, in to_image_block
    return asyncio.run(to_image(spec, f, topojson, mapbox_token, debug=debug, tmp_path=tmp_path))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kaleido/__init__.py", line 53, in to_image
    Browser(headless=True, debug=debug, debug_browser=debug, tmp_path=tmp_path) as browser,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
choreographer.browser.BrowserFailedError: Close() was called before the browser finished opening- maybe it crashed?

Kaleido: v1.0.0rc0
choreographer: 0.99.7

I pinned choreographer to 0.99.7 because with 1.0.3 I get:

Chromium 132.0.6834.83 Alpine Linux
Traceback (most recent call last):
  File "/app/test.py", line 4, in <module>
    fig.write_image("/output/figure.png")
  File "/usr/local/lib/python3.12/site-packages/plotly/basedatatypes.py", line 3827, in write_image
    return pio.write_image(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/plotly/io/_kaleido.py", line 266, in write_image
    img_data = to_image(
               ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/plotly/io/_kaleido.py", line 132, in to_image
    raise ValueError(
ValueError: 
Image export using the "kaleido" engine requires the kaleido package,
which can be installed using pip:
    $ pip install -U kaleido

For that error I created the following issue: plotly/choreographer#218 but maybe it is a Plotly/kaleido issue?

[mschoettle]

Updated my MRE repo: https://github.com/mschoettle/plotly-image-export-alpine

/cc @gvwilson @ayjayt

[breJcharAff]

I managed to avoid these two errors and make it work by using Kaleido 0.1.0.post1 with choreographer 0.99.7 and then changing cli.append("--headless=old") with cli.append("--headless") inside Lib/site-packages/choreographer/chrome_wrapper.py

[ayjayt]

Hi @breJcharAff , we don't recommend pinning the old version. Kaleido 0.1.0.post1 doesn't use choreographer at all afaik. (That said, it's not a bad idea right now during the transition).

So @mschoettle, thanks so much for your patience, two things:

chromium version

The newer versions of Kaleido are going to, instead of automatically downloading the latest version of chromium, use a last known good version. This is a fix in choreographer v1.0.4- I'm not sure if Kaleido v1.0.0rc0 is compat w/ choreographer v1.*.*, but i think so, so I'll take a look at that error.

These "last known good versions" can be downloaded through the cli kaleido_get_chrome --help or through a function await kaleido.get_chrome()

These new versions of Kaleido (> Kaleido v1.0.0rc0) are not yet published on pypi and the API is slightly different.

Version Changes

Kaleido v1.0.0.rc0 was the old api with the new engine, but strategy has changed, and all Kaleido v1 will be new api new engine. We haven't pushed anything > rc0 yet because we're still working on documentation and because plotly needs to be aware of the new Kaleido API. You can use the new kaleido as a package by itself, but the integration with plotly and kaleido for the fig.write_image() shortcut is on-going.

Here is v1.0.0rc6 (current master branch), here is v1.0.0rc7. The api is slightly different and my personal reference is here: (for rc7) https://geopozo.github.io/Kaleido/

I know this is a lot. Once the documentation is done, v1 of Kaleido will 100% use the new API and engine.

@gvwilson I think this is what support is going to look like during the migration.

Closing, but open for discussion.

[mrdobalina2k]

I got the same error on Debian Bullseye in docker. Setting log level to DEBUG revealed that alot of dependencies for chrome was missing. Once that was solved I was able to get Kaleido running. Not sure if all are needed but this is what worked for me:

apt install -y \
    libasound2 \
    libatk-bridge2.0-0 \
    libatk1.0-0 \
    libatomic1 \
    libatspi2.0-0 \
    libc6 \
    libcairo2 \
    libcups2 \
    libdbus-1-3 \
    libdouble-conversion3 \
    libdrm2 \
    libevent-2.1-7 \
    libexpat1 \
    libflac8 \
    libfontconfig1 \
    libfreetype6 \
    libgbm1 \
    libgcc-s1 \
    libglib2.0-0 \
    libgtk-3-0 \
    libjpeg62-turbo \
    libjsoncpp24 \
    liblcms2-2 \
    libminizip1 \
    libnspr4 \
    libnss3 \
    libopenjp2-7 \
    libopus0 \
    libpango-1.0-0 \
    libpng16-16 \
    libpulse0 \
    libre2-9 \
    libsnappy1v5 \
    libstdc++6 \
    libwebp6 \
    libwebpdemux2 \
    libwebpmux3 \
    libwoff1 \
    libx11-6 \
    libx11-xcb1 \
    libxcb1 \
    libxcomposite1 \
    libxdamage1 \
    libxext6 \
    libxfixes3 \
    libxkbcommon0 \
    libxml2 \
    libxnvctrl0 \
    libxrandr2 \
    libxslt1.1 \
    libxtst6 \
    zlib1g

[dlogozz0]

So what was the proposed workaround for this while we wait? @ayjayt

[ayjayt]

@mrdobalina2k how did you install chrome in this case?

[ayjayt]

Hi @dlogozz0 , this what I recommended elsewhere. Let me know if it works for you: mschoettle/plotly-image-export-alpine#1

That solution should basically be permanent.

edit: One thing to note, if you're using alpine/musl, you have to install chromium from their package manager. if you're using a standard win/mac/linux distribution, you can use kaleido_get_chrome which will install a tested version of chromium.

[mrdobalina2k]

@mrdobalina2k how did you install chrome in this case?

I used the method in the docs, kaleido.get_chrome_sync. It is a method imported from choreographer, and that was version 1.0.5

[dlogozz0]

@ayjayt thanks for the suggestions. I've made a couple edits to our functions to use kaleido.write_fig_sync(fig_spec, path=file_path, opts=opts) but I run into something strange:

  File "/opt/airflow/pipeline/ppt_reports/ppt_creator.py", line 187, in execute
    ).update()
  File "/opt/airflow/pipeline/ppt_reports/updaters/slide_updaters.py", line 109, in update
    ).append()
  File "/opt/airflow/pipeline/ppt_reports/appenders/map_appenders.py", line 54, in append
    go.Figure(data=go.Choropleth(locations=self.data[FEATURE_VALUE],
  File "/home/airflow/.local/lib/python3.10/site-packages/plotly/basedatatypes.py", line 3827, in write_image
    return pio.write_image(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/plotly/io/_kaleido.py", line 266, in write_image
    img_data = to_image(
  File "/home/airflow/.local/lib/python3.10/site-packages/plotly/io/_kaleido.py", line 132, in to_image
    raise ValueError(
ValueError: 
Image export using the "kaleido" engine requires the kaleido package,
which can be installed using pip:
    $ pip install -U kaleido

I do have Kaleido installed in the Pipfile as you recommended: kaleido = {git = "https://github.com/plotly/kaleido", ref = "latest-tag", subdirectory = "src/py"} and I have confirmed kaleido is installed in the Pipfile.lock

Do you have any ideas why plotly is not recognizing kaleido? I have the following versions installed:

kaleido: '1.0.0rc9'
plotly: '6.0.0'
choreographer: '1.0.5'

[ayjayt]

@mrdobalina2k I'm glad you were able to resolve the issue by installing dependencies. It is unfortunate that you need to do that, however.

@dlogozz0 plotly will not recognize kaleido until plotly 6.1.0, this is the pr.

A permanent solution is to use the kaleido library directly each time, it is also more efficient since you can control the lifetime of the engine.

[mschoettle]

if you're using a standard win/mac/linux distribution, you can use kaleido_get_chrome which will install a tested version of chromium.

Since my example is using a Dockerfile: Is kaleido_get_chrome suitable to be used during docker build?

[ayjayt]

@mschoettle in general yes but your docker examples use alpine which uses musl and kaleido_get_chrome uses a repo with no musl builds, only gnu libc