[VSC-31] Remove intermediate conversion to S-exps

[alexd1971]

Problem: The current process for creating a JS representation of the AST requires an intermediate conversion to S-expression. This hurts performance and makes it impossible to annotate all nodes with types.

Solution: Implement a library for converting the original AST directly to JS, eliminating the intermediate step.

[cla-idx-bot]

Dear @alexd1971,

In order to potentially merge your code in this open-source repository and therefore proceed with your contribution, we need to have your approval on DFINITY's CLA.

If you decide to agree with it, please visit this issue and read the instructions there. Once you have signed it, re-trigger the workflow on this PR to see if your code can be merged.

— The DFINITY Foundation

[rvanasa]

Thanks for the improvements! Apologies for the delay in reviewing; a lot is happening internally at the moment.

It could be worth investigating whether it's possible to cache / share type representations to reduce the memory and conversion overhead. This would be relevant for larger-scale projects which tend to have a lot of duplicated complex type nodes.

[alexd1971]

Thanks for the improvements! Apologies for the delay in reviewing; a lot is happening internally at the moment.

It could be worth investigating whether it's possible to cache / share type representations to reduce the memory and conversion overhead. This would be relevant for larger-scale projects which tend to have a lot of duplicated complex type nodes.

Thank you for your review and your suggestion for further improvement. We think that introducing typeRep cache is reasonable and actually will allow us to reduce consuming resources. We could implement it as follows:

1. Introduce a JS Map as a typeRep cache storage.
2. Use string representation of type as a key and typeRep object as a value.
3. During serialization if we already have a key in the cache Map we don't serialize type representation again.
4. To make AST compatible with current implementation we need to define typeRep as the Node object property with the getter taking the value from the cache.

However, it sounds like a non-trivial change that is quite large and requires thorough testing. That's why we propose postponing this change until later and merging this PR now, given that it is already an improvement on its own and satisfies the corresponding acceptance criterion from the first milestone.