Proposal: Getting rid of GuppyModule

[mark-koch]

Goal

Get rid of the user-facing GuppyModule and only use module-free @guppy decorators. Things should just behave like Python:

• When writing a Guppy function, another Guppy object (e.g. another function, type, constant, ...) is in scope iff the object were also in scope if it was all Python
• Python modules may be used to structure Guppy functions, but they no longer correspond to Guppy compilation units
  • Complation entry points are on the functions themselves (e.g. main.compile() instead of module.compile())
  • Compiling an object should only compile the objects that are referenced by it, not the entire Python module
  • Extra benefit: No more module.unregister(...). If a faulty function is not used, it's not going to be compiled

Examples

Python style late binding

@guppy
def bar() -> int:
    return 1

@guppy
def foo() -> int:
    return bar()

@guppy
def bar() -> int:
    return 2

foo.compile()  # foo will return 2

Similarly, as in Python, it's fine to import late

@guppy
def foo() -> int:
    return bar()

from other_module import bar

foo.compile()

Functions can be dynamically created:

def make_bar(i):
    @guppy
    def bar() -> int:
        return comptime(i)

bar1 = make_bar(1)
bar2 = make_bar(2)

@guppy
def foo() -> int:
    x = bar()  # error: not in scope
    return bar1() + bar2()

How to do it (draft)

For each invocation of the guppy decorator:

• Store a definition in a global mapping indexed by unique DefId (no names!)
• Traverse the AST to store any names referenced from the outside (i.e. uses of names that never occur on the LHS of an assignment) as captured
• Also store the f_locals of the frame in which the function was defined. Any mutation of captured Python variables after the function was defined will be visible through f_locals!

Get rid of guppy.compile_module. Instead, offer .check() and .compile() methods on all guppy decorated objects. When calling those, do the following:

• Infer the dependencies of the to be compiled object (i.e. all other functions/types/... that are transitively used)
• We can do this by walking over the captured names and looking them up in the corresponding f_locals and __globals__ (in that order). If they contain any guppy definitions, mark them as dependencies and proceed recursively.
• At that point, we could also do some per-definition caching since we can store the captured values and only recompile if they have changed
• We should also update captured to store the actual values for easy lookup later without having to look again into f_locals. This also means, we can drop f_locals to get rid of some reference cycles and help the GC
• Finally, type check and compile all dependencies

When type checking a function body:

• When we encounter a name: If it's in captured, load the value. Otherwise, check if it's a previously defined local, else emit a user error.
• When we encounter expr.name, we check if expr is a captured Python module. In that case, we'll try to look up name from that module

Notes:

• Relying on __closure__ would not be enough since it doesn't contain values captured in the type signature. That's why we have to store the entire f_locals dict. However, this might change with Python 3.14 annotation descriptors: https://peps.python.org/pep-0649/

[ss2165]

The examples look good.

It is not clear to me what the impact of this would be on the ability to produce separately compiled guppy modules. Does a single compile invocation always produce a single hugr module with all used objects compiled in to that module? Are we relying on python module structure for namespacing and so is that namespace information gone in the HUGR output?

[mark-koch]

With this approach, every Guppy definition would stand on its own. If we want to do individual caching of compilation results, I would do it at the function level: every function would to be stored as a separate Hugr with decls for all of its dependencies.

If we want a notion of Guppy library (instead of a binary) that is compiled in one go and turned into a single Hugr that can be distributed and linked, I would want users to explicitly list all the functions that should be contained in the library (maybe similar to the __all__ in Python). Maybe something like:

@guppy
def foo() -> int:
    return 1

bar1 = make_bar(1)
bar2 = make_bar(2)

from other_file import baz

lib = guppy.declare_library(foo, bar1, baz)

Then, we could say that depening on either one of foo, bar1, baz will automatically add dependencies to all the others and they are cached in a single Hugr module. The point is, no more implicit conflation between Python modules and Guppy compilation units: baz from another file would be contained in the library, but bar2 would not.

But not sure if it would be worth adding this since it also comes with some complications: Any files not explicitly added to a library/compilation unit would stand on their own. E.g. if you would redeclare foo afterwards

lib = guppy.declare_library(foo, bar1, baz)

@guppy
def foo() -> int:
    return 2

the library would still point to the old version. But in compliance with Python scoping rules, any use of foo in bar1 or bar2 would refer to the new version which is now a free function. You'd need to explicitly add it to the library to become part of the compilation unit:

lib.add(foo)

[ss2165]

Ok understood.

no more implicit conflation between Python modules and Guppy compilation units

this I like a lot

If we want a notion of Guppy library ... that can be distributed and linked

yes, I think this is quite important. I am happy to lose the "implicit" module way of achieving this, but there should be a way that is ergonomic and sound (verbosity is not an issue).

Can we combine foo = guppy.declare_library with some kind of import mechanism? If you import a pre-declared library it is not injected but just referred to. import is a concept in hugr-model (@zrho may have thoughts here).

[tatiana-s]

What would be the difference between importing Guppy libraries as Python imports and the import mechanism tied to declare_library?

I think one of the most confusing things with the current module system is the module.load so it would be good to avoid anything in that direction.

[mark-koch]

I think one of the most confusing things with the current module system is the module.load so it would be good to avoid anything in that direction.

Agree! The only purpose for this library thing is that all contained definitions are checked and can be distributed together. There should be no explicit loading to use a library: Just do a Python import and use the function, then all other stuff in the library will be implicitly loaded

[tatiana-s]

A bit of a secondary concern, but just thinking of modules in the context of language services, specifically the difference between wanting to say run check() on all functions in a file vs. a user compiling from a single function entry-point. I suppose this would make it easier than before, since you could try to extract each function separately? But then I guess caching would be important since a lot of them will refer to each other.

[mark-koch]

since you could try to extract each function separately?

Exactly. The module.unregister thing is very annoying imo and would be solved by only compiling the entry point and its dependencies.

run check() on all functions in a file

This would no longer be possible since we would stop to associate Guppy functions to Python files. The closest analogue would be the library idea discussed above which would be checked together (but they need to be listed explicitly and don't need to be in the same file)

But then I guess caching would be important since a lot of them will refer to each other.

Agreed 👍

[ss2165]

Completed with #983