Making our AST builder more powerful by using an inference library

[tristanlatr]

This issue is related to #348 and #591 and #623.

I'm currently looking at building a library that would provide a replacement method for the ast.parse() function, outputting ast nodes that stays 100% compatible with the standard library one with some bonuses. Before starting that, I spend a little bit of time trying to use astroid, but I realized that they were too many disparities with the standard library and it would be too much of work adjusting pydoctor to astroid. So I'm basically adjusting astroid to pydoctor, or to be exact to the standard library AST nodes, and only the required subs tract in order to infer literal expressions and have better typing informations.

This is the repository to find the code I'm currently working on: https://github.com/tristanlatr/astuce

Is there any other existing tool that could have done the job? I've looked at many AST inference and tooling library and basically all of them that provides inference uses custom AST nodes that are not compatible with the standard library (like jedi, astroid, mypy, etc). Libraries that works with standard ast nodes like astor, astmonkey, does not provide inference utilities.

So this is why I'de suggest we use that new package once we have a first stable version, then we can use it to infer the __all__ values and variable type information when it's not documented. These are the two main goals of this new library.

If we decide to go forward with astuce, we'll need to add this new package to the Debian tracker as well.

Tell me what you think @adiroiban @tijuca

[tristanlatr]

The tricky thing to get right is the scope of this new ast analysis library. It should not duplicate bunch of pydoctor code and add extra on top of it like I did with pydocspec.

It should be as minimalistic as possible and the type of expression that we want to have inference for needs to be specified from the beginning if we want it to be useful for other users and stable in the time. This library should not be updated as often as pydoctor, to keep the maintenance low. Unfortunately it needs a bit of complexity to reason about branch reachability etc. So it should be battle tested from the go to suffer less possible bug fixes.

[tijuca]

I can't speak for any Python technical stuff you are thinking of, but I can package some required new package(s) for Debian for sure.

[tristanlatr]

Thanks @tijuca, I will communicate with you if there is a new requirement for pydoctor that is not already in the tracker.

On a technical note, I just found this package: https://github.com/SMAT-Lab/Scalpel with looks very promising.

[tristanlatr]

I’ve continued my search for inference libraries, and I’ve found this one: https://github.com/usagitoneko97/klara which offers the same kind of inference system as astroid, plus with the assumptions related to each inferred nodes, meaning we could filter nodes that assumes TYPE_CHECKING is True or other kind of checks such as sys.version_info checks very easily.