Skip to content

Introduce the astcompile pass - #649

Merged
antocuni merged 83 commits into
mainfrom
astcompile
Aug 10, 2026
Merged

Introduce the astcompile pass#649
antocuni merged 83 commits into
mainfrom
astcompile

Conversation

@antocuni

@antocuni antocuni commented Aug 10, 2026

Copy link
Copy Markdown
Member

This PR introduces a new pass astcompile in the pipeline.
astcompile takes a "parsed" AST and produce a new AST which is more suited for execution.

In particular, it handles:

  • ast.Name specialization: depending on the symbol, it becomes ast.NameLocalDirect, ast.NameOuterCell, etc.
  • ast.Assign specialization: similar to above, it becomes ast.AssignLocal, ast.AssignOuter, etc.
  • for loop desugaring
  • augmented assign desugaring

Previously, all these transformation were done at runtime by ASTFrame, and then saved on the ugly dicts ASTFrame._specialized_names, etc., so that they could be reused by DopplerFrame.

Having a separate pass makes it easier to inspect the result and to add more desugaring when needed.

While we were at it, we also simplified the relationship between ast.Assign and ast.AssignExpr. Now all the logic is done by the various AssignExpr variants, and the statement versions are just thin wrappers over them. This makes it possible to reduce code duplication and make the whole logic easier.

For example, given:

X = 1
def main() -> None:
    y = X
❯ spy astcompile /tmp/x.spy 
Module(
    stage='astcompiled',
    filename='/tmp/x.spy',
    docstring=None,
    decls=[
        ...
        GlobalFuncDef(
            funcdef=FuncDef(
                stage='astcompiled',
                ...
                body=[
                    AssignLocal(
                        expr=AssignExprLocal(
                            target=StrLiteral(value='y'),
                            sym=Symbol('y', 'const', 'direct'),
                            value=NameOuterDirect(
                                sym=Symbol('X', 'const', 'direct'),
[...]

Note that:

  • we have a new stage attribute which keeps track of which lowering we did so far
  • we have specialized AssignLocal and NameOuterDirect nodes

We can also use the spy backend, useful for e.g. this:

def main() -> None:
    for i in range(10):
        print(i)
❯ spy astcompile --format spy /tmp/x.spy 
def main() -> None:
    _$iter0 = range(10).__fastiter__()
    while _$iter0.__continue_iteration__():
        i = _$iter0.__item__()
        _$iter0 = _$iter0.__next__()
        print(i)

antocuni added 30 commits August 9, 2026 17:12
using e.g. @astnode("parsed") or @astnode("< redshifting"), etc.
This commit breaks the world, but test_simple[interp] passes.
…ler. TestBasic::test_assignexpr_basic[interp] passes
…ic::test_assignexpr_updates_cell[interp] passes
@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown
Documentation Preview
Preview documentation for PR #649.
Last Updated: 2026-08-10 22:20 UTC

@antocuni
antocuni marked this pull request as ready for review August 10, 2026 22:26
@antocuni
antocuni merged commit 73c3de8 into main Aug 10, 2026
8 checks passed
@antocuni
antocuni deleted the astcompile branch August 10, 2026 22:26
@antocuni antocuni mentioned this pull request Aug 10, 2026
@antocuni antocuni changed the title WIP: introduce the astcompile pass Introduce the astcompile pass Aug 10, 2026
antocuni added a commit that referenced this pull request Aug 11, 2026
This should have belonged to #649 , but too late and too bad.

The hard part is how to write readable tests, because comparing the
generated AST is a pain. We ideally want to use the SPy backend to
generated "astcompiled source code", but then all the nuances of
NameLocalDirect, AssignLocal, etc, disappears.

The solution was to introduce a new option for the SPy backend
`ast_format="full"`, which annotates some selected nodes with extra info
needed for tests.
This is now exposed also from the `spy` cli.
E.g.:

```
❯ cat /tmp/x.spy
def main() -> None:
    x = 1
    print(x)
```

with the default "human-readable" format, we don't see any difference:

```
❯ spy astcompile -f spy /tmp/x.spy
def main() -> None:
    x = 1
    print(x)
```

But with `--full-ast`, we do!
```
❯ spy astcompile -f spy /tmp/x.spy --full-ast
def main() -> None:
    AssignLocal(x := 1)
    ImportRef(print)(LocalDirect(x))
``
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant