Building multiple languages at once

[m-aciek]

Is your feature request related to a problem? Please describe.
There's a discussion around optimisation of build time of Python docs, which consist of 13 languages and 6 versions currently. The whole process takes now more than 24 hours.

Describe the solution you'd like
I wonder if it wouldn't be feasible to enable invoking Sphinx builds for multiple languages at once. While I'm sadly not very familiar with Sphinx internals, I wonder if internally having a "documents structure" in memory, it wouldn't be faster to make use of it for multiple languages at once instead of invoking builds for separate languages synchronously. Could you please validate my way of thinking?

Describe alternatives you've considered
∅

Additional context

• Full build with PDF is taking more than 24h python/docsbuild-scripts#169

[picnixz]

I'll answer you by the end of the day but I'll put some labels in the meantime.

[picnixz]

(So it wasn't by the end of the day but the week).

I looked a bit at how Sphinx does things and here is what I understood (I'm no expert in that area since I've never worked with that component in depth).

• You have some document with translatable strings.
• You run make gettext to get .pot files and then use sphinx-intl to create the corresponding .po files. Then, you ask the universe to help you and fill these files.
• Then you can do make '-D language=...' html to get your translated build.

Now I assume that you want to speed up the build because running all make -D language=... for all your languages, you want to do it more or less once and have a worflow "filling up" what remains with the proper translations. I looked at the Python issue and what I observed is that most of the computation time is because of the LaTeX build. In particular, builds that require XeLaTeX or LuaTeX are de facto slow compared to the pdflatex.

When considering parallel builds with multiple languages, the translation is applied at a very early stage of the build, but after the directives being read if I'm not wrong. So, we could perhaps do the following:

• Run everything before the translation step as a single task, until the first localization transformation is applied.
• Unfortunately, after that, the doctrees are all distinct so it might happen that they will be transformed again but differently (like, there is a transformation that, e.g., removes weird symbols in texts. But with that, it may act differently depending on the locale).

As such, I'm not sure we could really improve the build time. Personally, I would like to know why the LaTeX build takes so much time. Is it because of references? if we knew which document is taking a long time to do, we could speed the things by redesigining the latex builder. For instance, LaTeX is generally slow because it is not parallelizable in general (pages are rendered in sequential order).

[jayaddison]

See also: #788

[AA-Turner]

Closing as duplicate of #788.

A