Skip to content

Added lazy_py_modules #95

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 23, 2024
Merged

Added lazy_py_modules #95

merged 1 commit into from
Apr 23, 2024

Conversation

WebReflection
Copy link
Contributor

@WebReflection WebReflection commented Apr 19, 2024

This MR would like to enable/allow our users to eventually lazily load modules as opposite of having these pre-fetched and already available via the config.

It exposes a polyscript.lazy_py_modules(...packages) callback that's going to be awaited so that modules can land on the current session lazily, as opposite of on bootstrap.

The idea is to forward this to the terminal or any other PyScript related session so that an asnc method could always lazily load a module avoiding slow bootstrap when a module is needed only in certain occasions.

I don't know if this is desirable but we already had some user asking for this, and internally neither Pyodide nor MicroPython allow to reach the loadModule or pyimport internally, for what I could test, so this uses the exposed JS API so that modules can land later on.

please see the test file attached

/cc @ntoll @JeffersGlass

@WebReflection
Copy link
Contributor Author

@JeffersGlass I am not sure where you comment ended up (email only or removed) but this is about loading Python modules within the running interpreter, otherwise not possible as from Python code, if I am not mistaken, there's no way to otherwise lazy load from the network or Pyodide wheels modules outside the static, pre-defined, config.toml or .json ... I hope this is clear and it answered your question.

Copy link
Member

@ntoll ntoll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see @JeffersGlass's question that you reference @WebReflection. :-/ I assume you've answered to Jeff's satisfaction.

This looks like a good first step because:

  1. It's a small change.
  2. It works with both MicroPython and Pyodide.
  3. You have to be explicit when using it (i.e. default behaviour of PyScript doesn't change).

Next steps (if required):

  1. Lazy loading of JS modules in a similar fashion.
  2. Lazy loading of our built-in PyScript "stuff" pydom and pyweb spring to mind.

We should ensure this is documented too. 👍

@WebReflection
Copy link
Contributor Author

@ntoll about this:

Lazy loading of JS modules in a similar fashion.

to be honest, JS modules in workers are already lazy loaded, the dance is made synchronous by the implementation, like everything else is in workers, but on top of that, import js; await js.import("module") is already possible ... maybe we can just avoid import name error and provide a lazy_js_modules too for symmetry? The gotcha there is that JS modules usually are better off on the main thread, which is why we have this disambiguation between js_modules.main and js_modules.worker so we need to be careful in what we expose/offer that, as the Python code never needs such disambiguation.

For pydom or pyweb we can indeed start simple here and maybe automatically orchestrate a CDN download from our own repository when any of these two names is required, bypassing micropip or mpip out of the box? maybe @fpliger has other ideas? The fact these are async might not be as straight forward as story for those modules, we need to better think if we just want special treatment via config directly ...

@WebReflection
Copy link
Contributor Author

btw ...

We should ensure this is documented too. 👍

good point, I was thinking this should be documented in PyScript once it lands but it's actually a Polyscript feature worth documenting in here too ... I will do if I have time, and before merging it.

@WebReflection
Copy link
Contributor Author

I've added minimal docs around lazy_py_modules + an example on how it works.

@WebReflection WebReflection merged commit d49e2ef into main Apr 23, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants