diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1621317..8c82e71 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -18,7 +18,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - smalltalk: [ Pharo64-13 ] + smalltalk: [ Pharo64-13, Pharo64-14 ] name: ${{ matrix.smalltalk }} steps: - uses: actions/checkout@v4 diff --git a/README.md b/README.md index 97d701b..2aa4019 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,30 @@ This project provides **Pharo bindings to [Tree-sitter](https://tree-sitter.github.io/tree-sitter/)**. It allows you to analyze and highlight code using existing Tree-sitter parsers developed by the community. + + + +- [Pharo Tree-Sitter](#pharo-tree-sitter) + - [Installation](#installation) + - [Native Libraries Required](#native-libraries-required) + - [Automated Library Generation](#automated-library-generation) + - [How It Works](#how-it-works) + - [Required Tools for MacOS and Linux](#required-tools-for-macos-and-linux) + - [Required Tools for Windows](#required-tools-for-windows) + - [Manual Installation](#manual-installation) + - [macOS](#macos) + - [Building Tree-sitter Shared Library](#building-tree-sitter-shared-library) + - [Archlinux](#archlinux) + - [Quick Example](#quick-example) + - [Moose: TreeSitter to build FAST and Famix models](#moose-treesitter-to-build-fast-and-famix-models) + - [Utilities and Debugging](#utilities-and-debugging) + - [Pre-built Libraries (Temporary Solution)](#pre-built-libraries-temporary-solution) +- [Create a new FASTLanguageMetamodel using PharoTreeSitter](#create-a-new-fastlanguagemetamodel-using-pharotreesitter) + + + +## Installation + To install it in **Pharo**, run the following script in a Playground: ```smalltalk @@ -16,6 +40,14 @@ Metacello new load. ``` +To add this project to your baseline you can copy paste: + +```Smalltalk +spec + baseline: 'TreeSitter' + with: [ spec repository: 'github://Evref-BL/Pharo-Tree-Sitter:main/src' ] +``` + ## Native Libraries Required To function correctly, the following native libraries are required depending on your OS: @@ -131,24 +163,6 @@ So you only have to install tree-sitter and the grammar you wanna use with your yay tree-sitter tree-sitter-grammars ``` - -## Debug - -You might want to debug a tree sitter parser from Pharo. -Whereas we did not find _yet_ a way to use the pharo debugger. -You can create a logger attached to the parser. - -```st -callback := (TSLogCallback on: [ :payload :log_type :buffer | Transcript crShow: buffer ] ). -logger := TSLogger new log: callback . -parser logger: logger. -``` - -## Pre-built Libraries (Temporary Solution) - -If you face issues building the libraries, you can download pre-built libraries (for Windows and macOS) here: -https://doi.org/10.5281/zenodo.15423234 - ## Quick Example Here is a basic usage example in Pharo: @@ -161,6 +175,28 @@ tree := parser parseString: string. tree rootNode ``` +## Moose: TreeSitter to build FAST and Famix models + +[`Moose`](https://github.com/moosetechnology/Moose)is a software analysis platform. It can manage two kinds of models: +- Famix been a dependency model +- FAST (Famix AST) been an AST representation of the language + +It exist some utilities to help building a Famix model for a language supported by TreeSitter. You can find this project here: [https://github.com/moosetechnology/TreeSitterFamixIntegration](https://github.com/moosetechnology/TreeSitterFamixIntegration) + +And this project provides a way to easily create a FAST importer. You can find the documentation here: [Documentation](resources/doc/fast_importer.md) + +## Utilities and Debugging + +This project provides a few utilities to help you debug a tree sitter parser or to help you with its manipulation. + +Those utilises are documented here: [Documentation](resources/doc/ts_utilities.md) + +## Pre-built Libraries (Temporary Solution) + +If you face issues building the libraries, you can download pre-built libraries (for Windows and macOS) here: +https://doi.org/10.5281/zenodo.15423234 + + # Create a new FASTLanguageMetamodel using PharoTreeSitter This library allows you to create a first version of any FASTLanguageMetamodel, in condition that this language is supported by treesitter and by PharoTreeSitter. To do that you can follow a detailed documentation [here](https://modularmoose.org/blog/2025-09-16-generation-of-fast-metamodel-using-treesitter/). diff --git a/resources/doc/TSNodeFinderVisitor.png b/resources/doc/TSNodeFinderVisitor.png new file mode 100644 index 0000000..9873847 Binary files /dev/null and b/resources/doc/TSNodeFinderVisitor.png differ diff --git a/resources/doc/TSSymbolsBuilderVisitor.png b/resources/doc/TSSymbolsBuilderVisitor.png new file mode 100644 index 0000000..8cf660d Binary files /dev/null and b/resources/doc/TSSymbolsBuilderVisitor.png differ diff --git a/resources/doc/basicMM.png b/resources/doc/basicMM.png new file mode 100644 index 0000000..d45c2e6 Binary files /dev/null and b/resources/doc/basicMM.png differ diff --git a/resources/doc/fast_importer.md b/resources/doc/fast_importer.md new file mode 100644 index 0000000..27f6fb3 --- /dev/null +++ b/resources/doc/fast_importer.md @@ -0,0 +1,808 @@ +# Create a FAST importer with TreeSitter + +This section covers the content of the package `TreeSitter-FAST-Utils` and explain how to use it to create a [FAST model](https://modularmoose.org/users/ast/fast/) for the [`Moose plateform`](https://github.com/moosetechnology/Moose). + +> [!NOTE] +> Here is some context for this documentation: It was produced after a new iteration over the FAST importer of TreeSitter and the production of [FAST-Python](https://github.com/moosetechnology/FAST-Python). Multiple utilities got extracted from `FAST-Python` to `pharo-tree-sitter`. This documentation explains as if all those utilities were present in `pharo-tree-sitter` before `FAST-Python` was started and most of the examples provided here will be from `FAST-Python`. + + + +- [Create a FAST importer with TreeSitter](#create-a-fast-importer-with-treesitter) + - [Some context](#some-context) + - [Get a working baisc FAST importer](#get-a-working-baisc-fast-importer) + - [Install TreeSitter](#install-treesitter) + - [Generate the base of the Metamodel](#generate-the-base-of-the-metamodel) + - [FAST importer](#fast-importer) + - [Customize your metamodel and visitor](#customize-your-metamodel-and-visitor) + - [Improve relations management](#improve-relations-management) + - [Implement a specialized visitor](#implement-a-specialized-visitor) + - [Specialized visit methods](#specialized-visit-methods) + - [Change the class produced by a node](#change-the-class-produced-by-a-node) + - [Handle the parent/children relation yourself](#handle-the-parentchildren-relation-yourself) + - [The context management](#the-context-management) + - [Customize fields relation name matching](#customize-fields-relation-name-matching) + - [Manage properties](#manage-properties) + - [Add regression tests to your project](#add-regression-tests-to-your-project) + - [TSFASTAbstractImporterTest](#tsfastabstractimportertest) + - [Regression tests generation](#regression-tests-generation) + - [Customize the test generation](#customize-the-test-generation) + - [Regenerate all tests](#regenerate-all-tests) + - [Generate multiple tests](#generate-multiple-tests) + - [Error handling](#error-handling) + - [Error node](#error-node) + - [Error report](#error-report) + - [Utilities](#utilities) + - [Cyril's tips on how to work](#cyrils-tips-on-how-to-work) + + + + +## Some context + +This documentation assume you are already familiar with: + +- Tree-Sitter +- Pharo-Tree-Sitter +- FAST +- Metamodel generators + +> [!IMPORTANT] +> If you are not, the following provides a general introduction to these tools/libraries along with references for further details: +> - **[Tree-Sitter](https://tree-sitter.github.io/tree-sitter/)** is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited. It is able to parse a large variety of programming languages such as Java, C++, C#, Python and many others. +> - **[Pharo-Tree-Sitter](https://github.com/Evref-BL/Pharo-Tree-Sitter/)** the project containing the FAST importer infrastructure. It is developed in Pharo and integrates the original Tree-Sitter parsers and allows visualizing their results (such as ASTs) directly in Pharo. It relies on the FFI protocol, which requires the corresponding libraries depending on the OS (.dll, .so, or .pylib) to be present in Pharo’s VM folders. +> The project supports parsing several languages, and for some of them (like Python, TypeScript, and C), the library generation is automated. You can find more details in the README. +> This is the project that we will use to generate a new FAST-Language metamodel, so you need to download it into your Pharo image. +> - **[FAST](https://github.com/moosetechnology/fast)** means Famix AST. Contrary to Famix that represent application at a high abstraction level, FAST uses a low-level representation: the AST. +> FAST defines a set of traits that can be used to create new meta-models compatible with Moose tools. +> When developing a new FAST-Language metamodel, you will rely on these FAST traits to structure your metamodel. +> - **[Metamodel generator](https://modularmoose.org/developers/create-new-metamodel/)** is a Pharo library used to create new metamodels such as FAST-Java, Famix-Java, or FAST-Fortran. +> The generation of any new version of a FAST-Language metamodel can only be achieved through the metamodel generator. +> As you will see in this post, Pharo-Tree-Sitter enables you to define a new metamodel generator. Once executed, it produces the corresponding FAST-Language metamodel. We will explain this process in more detail in the following sections. + + +## Get a working baisc FAST importer + +### Install TreeSitter + +First you need to create a Moose image and download [Pharo-Tree-Sitter](https://github.com/Evref-BL/Pharo-Tree-Sitter/): + +```smalltalk +Metacello new + baseline: 'TreeSitter'; + repository: 'github://Evref-BL/Pharo-Tree-Sitter:main/src'; + load. +``` + +Once you project has a baseline, you need to add tree sitter as a dependency: + +```smalltalk +spec + baseline: 'TreeSitter' + with: [ spec repository: 'github://Evref-BL/Pharo-Tree-Sitter:main/src' ] +``` + +Once downloaded, you need to make sure that `Pharo-Tree-Sitter` is able to parse the language that you intend to create the metamodel for. +If it is not included, you need to follow the instructions in the readme file of this repository and add the new language. +For this documentation we will assume that the language is already supported and we will continue with "Python" 🐍🐍🐍. + +To be able to continue, and if this is the first time you're using `pharo-tree-sitter`, you need to launch the tests of python in package `TreeSitter-Tests` class `TSParserPythonTest`. +This is needed to launch the process of downloading the original **[tree-sitter](https://github.com/tree-sitter/tree-sitter)** and **[tree-sitter-python](github.com/tree-sitter/tree-sitter-python)** projects from GitHub, generating the correspondent libraries. + +Now that you have the libraries, you can parse python code and get an AST corresponding to the tree-sitter grammar, but not FAST-Python model. The next step will be to generate a basic FAST metamodel for our language. + +### Generate the base of the Metamodel + +It is possible to generate a first basic version of the FAST model using this the `TSFASTBuilder` like this: + +```smalltalk +TSFASTBuilder new + languageName: 'Python'; + tsLanguage: TSLanguage python; + build. +``` + +The language name is used for the prefix of the class names. They are on the form of `FASTPrefixSymbol` like `FASTPyFunctionDefinition`. + +The `tsLanguage` is the tree sitter language to use to retrieve the possible symbols. + +Once this code is executed, a new FAST generator will be in your image and you can generate the classes with it: + +```smalltalk +FASTPythonMetamodelGenerator generate +``` + +Example of generated code: + +![Screenshot of the basic MM](basicMM.png) + +In this generated metamodel, only one relation exist between any entities from #`genericChildren` to #`genericParent`. This relation is used to manage all relations in this basic importer. We will be able to improve this later. + +> [!TIP] +> The metamodel generated is only a base. It has a class for each element of the syntax but it has no hierarchy, no trait usage, no specific relations and no properties. We provide an explanation of the possible customizations in the section: [customize your metamodel and visitor](#customize-your-metamodel-and-visitor). + +This builder will also generate the base of an importer. We will explore that in the next section. + +### FAST importer + +Now that we have a basic metamodel, we can import a FAST model like this: + +```Smalltalk +| parser tsLanguage importer | + +parser := TSParser new. +tsLanguage := TSLanguage python. +parser language: tsLanguage. +string := 'if x > 0: + if x < 10: + 1 + else: + 2 +else: + 3'. + +importer := TSFASTImporter new. +importer tsLanguage: tsLanguage. +importer languageName: 'Python'. +importer originString: string. + +^ importer import: (parser parseString: string) rootNode +``` + +This is not really practical so it is recommended to implement a subclass of `TSFASTAbsractImporter`. With the current version of Tree Sitter, this should now happens automatically using `TSFASTBuilder`. + +For example for Python we now have a `FASTPythonImporter`. + +And the previous script can now be replaced by: + +```Smalltalk +FASTPythonImporter parse: 'if x > 0: + if x < 10: + 1 + else: + 2 +else: + 3' +``` + +or: + +```Smalltalk +FASTPythonImporter parseFile: 'myFile.py' +``` + +## Customize your metamodel and visitor + +> [!WARNING] +This documentation expects the reader to know how to develop a Moose metamodel generator. If this is not the case, you will need to learn how to handle entities hirerachy, relations between entities, properties declaration and hanlding of traits in order to fully understant this section. [See documentation.](https://modularmoose.org/developers/create-new-metamodel/) + +> [!WARNING] +When you update your generator (`FASTPythonMetamodelGenerator` in this documentation), do not forget to regenerate your metamodel. + +We are now able to have a FAST model, but this models has multiple limitations such has: +- There is no management of inheritance and usage of traits in FAST entities +- The relations are all stored in #`genericChildren` and #`genericParent` relation +- There is no property +- The AST of the tree-sitter grammar does not necessarily make a good AST and some nodes could gain to be customized + +In some cases, the model produce can be enough. This project was originally done in order to do pattern matching and have a good AST is not necessary. But, in order to use most Moose's tooling, we need to improve the generated code. + +Thus, this project provides a set a tools to customize our Metamodel and Importer. + +### Improve relations management + +A first way to improve the model is to improve the FAST generator and use specific relations instead of the generic one we generated. + +In order to create a FAST model, the importer will first use tree sitter to obtain a `TSTree`. This tree represent the AST of the source code parsed. We then visit its nodes to generate the FAST model. + +The `TSNodes` of the tree are grouping their children by `fields`. + +For example, we can see on the result of a tool we will explain later in the documentation: + +![Inspector](TSSymbolsBuilderVisitor.png) + +Here we see `if_statement` nodes have 4 fields: +- `condition` that always have 1 child +- `consequence` that always have 1 child +- `alternative` that is optional and can have multiple children +- an unnamed field that is optional and can have multiple children + +If the name a field matches the name of a Fame property (in the Moose description), then this relation will be used. + +In the case of our `if_statement`, the FAST node representing it should use `FASTTWithCondition` and store the condition expression into the `condition` relation. + +If order to do this, we need to retrieve `FASTTWithCondition` in our generator and make the `ifStatment` use it. + +```Smalltalk +FASTPythonMetamodelGenerator>>defineTraits + + super defineTraits. + + tWithCondition := self remoteTrait: #TWithCondition withPrefix: #FAST +``` + +```Smalltalk +FASTPythonMetamodelGenerator>>defineHierarchy + + super defineHierarchy. + + ifStatment --|> tWithCondition +``` + +> [!TIP] +> Do not forget to regenerate your metamodel afterward by executing `FASTPythonMetamodelGenerator generate` + +> [!NOTE] +> The importer will check if the relation used is multivalued or not. In case it is monovalue, it will use the setter. If it is multivalued, it will use `#addRelationName:`. + +This method is good because we just need to improve the metamodel generator to make it work. But it has its limits: +- Some fields are unnamed and cannot be mapped directly to a fame relation +- Some fileds are good names for a basic AST, but not good in an AST specialized for software analysis such as in Moose. For example, the left side of an asssignment can be named `left` in tree sitter, but should be named `variable` in a FAST model. Or a field can be named `expression` in tree sitter because it can contain one or multiple expression. But in Moose, a multivalued relation should be plurial and we need the relation to be named `expressions`. + +In order to make further improvements, we can use `TSFASTCustomizableVisitor`. + +### Implement a specialized visitor + +In order to apply more customizations to your FAST model, we will need to subclass `TSFASTCustomizableVisitor` like this: + +```Smalltalk +TSFASTCustomizableVisitor << #FASTPythonVisitor + slots: {}; + package: 'FAST-Python-Tools' +``` + +It should have an initialize method to specify the language: + +```Smalltalk +initialize + + super initialize. + self languageName: 'Python' +``` + +Then you need to indicate to your importer that it needs to use this visitor. + +```Smalltalk +FASTPythonImporter>>visitorClass + + ^ FASTPythonVisitor +``` + +We now have the possibility to apply a new set of customizations that are explained in the next sections. + +> [!NOTE] +> Each section will assume that you have read the previous sections. This means that everything in the examples provided will have been explained. + +#### Specialized visit methods + +By default, `TSVisitor` has only one visit method: `#visitNode:`. This is a limitation while developping an importer. Your new visitor is able to lift this limitation. + +We can use the type of a `TSNode` to implement a custom visit method. The visit method you implement should begin by `visit`, then the type in camel case format and the method takes one parameter: the `TSNode`. + +For example, in python the `TSNode` for a string is of type `string` and has 3 children: +- `string_start` +- `string_content` +- `string_end` + +The generator has `FASTPythonString`, `FASTPythonStringStart`, `FASTPythonStringContent` and `FASTPythonStringEnd`. + +Those nodes are useless to us since we just need the String node. + +In that case, we can remove the declaration of the 3 children in `FASTPtyhonMetamodelGenerator>>#defineClasses` and regenerate and we can modify the visitor to not produce any node for those 3 `TSNodes`. + +```Smalltalk +FASTPythonVisitor>>visitStringStart: aNode + "We only want one node for the string, my parent. So I do nothing" + +``` + +```Smalltalk +FASTPythonVisitor>>visitStringContent: aNode + "We only want one node for the string, my parent. So I do nothing" + +``` + +```Smalltalk +FASTPythonVisitor>>visitStringEnd: aNode + "We only want one node for the string, my parent. So I do nothing" + +``` + +We are able to implement one visit method for each node type. Here it was used to skip some nodes in the `TSTree`. Much more usages will be shown in the next sections. + +If you did not implement a specialized visit method, the visitor will backup to the basic visit: + +```Smalltalk +TSFASTVisitor>>visitNode: aTSNode + + ^ self handleNode: aTSNode +``` + +#### Change the class produced by a node + +In some cases it can be better to update the name of an entity. + +For example, the python `TSTree` represent attribute access with a node named `attribute`. This name is missleading in the context of `FAST`. We can rename the node in the generator and explicitly provide the class to the visitor. + +In `FASTPythonMetamodelGenerator`, the line: + +```Smalltalk +attribute := builder newClassNamed: #Attribute. +``` + +by: + +```Smalltalk +attribute := builder newClassNamed: #AttributeAccess. +``` + +Now that the metamodel is updated, we need to adapt the visitor: + +```Smalltalk +FASTPythonVisitor>>visitAttribute: aTSNode + + ^ self handleNode: aTSNode kind: FASTPythonAttributeAccess +``` + +#### Handle the parent/children relation yourself + +By default, when we are trying to set the parent of an entity, we are invoking the method `#setParentTo:`. This method takes as parameter the FAST entity represented by the node and will add itself in the children of the parent FAST entity. + +This method will check if we have a fame property of the same name as the field in which the node is. If yes, it will use this fame relation property to set the child in the right slot. Else it will use `#addGenericChild:`. + +Sometimes, we might need or prefer to set the parent ourselves. This is possible using: +- `#handleNode:parentBlock:` +- `#handleNode:kind:parentBlock:` + +For example, in python all entities can have a comment. Comments are entities annoying to manage because they are everywhere. We could manage them all at once. + +We can add this line: + +```Smalltalk +FASTPythonMetamodelGenerator>>#defineTraits + [...] + tWithComment := self remoteTrait: #TWithComments withPrefix: #FAST. +``` + +And this line: + +```Smalltalk +FASTPythonMetamodelGenerator>>#defineHierarchy + [...] + entity --|> tWithComment +``` + +Now in the visitor we can tell it that it will use `#addComment:` on its parent. + +```Smalltalk +FASTPythonVisitor>>visitComment: aTSNode + + ^ self handleNode: aTSNode parentBlock: [ :entity | self topFastEntity addComment: entity ] +``` + +or + +```Smalltalk +FASTPythonVisitor>>visitComment: aTSNode + + ^ self handleNode: aTSNode kind: FASTPythonComment parentBlock: [ :entity | self topFastEntity addComment: entity ] +``` + +Passing explicitly the class or not depend on your preference. Specifying it is a little bit faster during runtime. But the gain is often unsignificant. + +> [!NOTE] +> `self topFastEntity` will return the parent of the FAST entity we are creating. The name of the method refer to the concept of context that will be explaine if the section [The context management](#the-context-management). + + +#### The context management + +In order to be effective in our customizations, it is good to understand the context system of the visitor. + +Your visitor is maintaining a `Stack` named `context`. Its elements are `TSFASTContextEntry`. Each entry represent a node parent to the node we are visiting until we arrive at the root of the model. + +The entry will save multiple interesting informations such has: +- The fast entity produced by the node +- The `TSNode` corresponding to the context +- The filed name containing the nodes been pushed later on the stack +- The fm property corresponding to the field been visited. If it is not nil while `setParentTo:` is been invoked, it will be use the set the parent/child relation. + +This context stack explain why `topFastEntity` returns the parent of the node we are visiting. We get the top entry and we return its FAST entity. + +Those info in the context will be useful to import and also to customize the importers with `TSFASTCustomizableVisitor`. + +For example, in Python both method definitions and function definitions are managed by a node of type `function_definition`. But in our AST we want to distinguish them. + +For that we can add a new entity: + +```Smalltalk +FASTPythonMetamodelGenerator>>defineClasses + [...] + methodDefinition := builder newClassNamed: #MethodDefinition. +``` + +And we can customize our function definition visit: + +```Smalltalk +FASTPythonVisitor>>visitFunctionDefinition: aNode + "TreeSitter has the same nodes for method and function definitions but in FAST we want to disambiguate." + + ^ (context anySatisfy: [ :entry | entry fastEntity isClassDefinition ]) + ifTrue: [ self handleNode: aNode kind: FASTPyMethodDefinition ] + ifFalse: [ self handleNode: aNode kind: FASTPyFunctionDefinition ] +``` + +We can access all informations about the parent chain both in FAST entities and TS nodes. + +#### Customize fields relation name matching + +As we saw in a previous sections, the way we automatically resolve the relation to use in the parent/child relations have limits. The bigest one been that field names in tree sitter and relation names in FAST needs to be the same. + +The customizable visitor can use the context management to define some aliases for some fields and resolve this problem. + +For example in Python the `FASTTBinaryExpression` defines `leftOperand` and `rightOperand` but the fields in the node of type `binary_operator` are called `left` and `right`. + +Once the trait has been added to `FASTPythonBinaryOperator` in the generator, here is how to manage this: + +```st +FASTPythonImporter>>visitBinaryOperator: aTSNode + + | fastEntity | + fastEntity := self instantiateFastEntity: FASTPythonBinaryOperator from: aTSNode. + + self setParentOf: fastEntity. + + self pushContext: fastEntity node: aTSNode during: [ + context top add: 'leftOperand' asAliasOfField: 'left'. + context top add: 'rightOperand' asAliasOfField: 'right'. + self visitChildren: aTSNode in: fastEntity ]. + + ^ fastEntity +``` + +We cans do even better by using `#onNextContextDo:`. This method allow to apply a customization to the next context we create. Allowing to add the aliases before calling the normal node method handling: + +```Smalltalk +visitBinaryOperator: aTSNode + + self onNextContextDo: [ :entry | + entry + add: 'leftOperand' asAliasOfField: 'left'; + add: 'rightOperand' asAliasOfField: 'right' ]. + + ^ self handleNode: aTSNode kind: FASTPyBinaryOperator +``` + +A last problem solved by this alias mechanism is the case of unnamed fields. + +You can define an alias to it using `#aliasUnnamedFieldAs:`. For example, in python an assert statement has unnamed children. If we declare in the generator that an assert statement has a relation named `#expressions`, we can define: + +```Smalltalk +FASTPythonVisitor>>visitAssertStatement: aTSNode + + self onNextContextDo: [ :entry | entry aliasUnnamedFieldAs: #expressions ]. + + ^ self handleNode: aTSNode kind: FASTPyAssertStatement +``` + +#### Manage properties + +This mecanism of specialized visits is also useful to be able to handle some properties. + +For example, in python, multiple `TSNodes` have a child that can only be an `identifier` node and that represent the name of en entity. + +I did not wish to have too many `FASTPythonIdentifier` nodes representing names while FAST provides a `FASTTNamedEntity` trait with a `name` property. + +In order to do this, I first use this trait from FAST in the generator: + +```Smalltalk +FASTPythonMetamodelGenerator>>defineTraits + [...] + + tNamedEntity := self remoteTrait: #TNamedEntity withPrefix: #FAST. +``` + +Then we can use it: + +```Smalltalk +FASTPythonMetamodelGenerator>>defineHierarchy + [...] + + attribute --|> tNamedEntity. + + classDefinition --|> tNamedEntity. + + functionDefinition --|> tNamedEntity. + + genericType --|> tNamedEntity. + + keywordPattern --|> tNamedEntity. + + methodDefinition --|> tNamedEntity. + + namedExpression --|> tNamedEntity. + + splatType --|> tNamedEntity. +``` + +> [!NOTE] +> If you check the finished FAST-Python you will notice that it is not `classDefinition`, `methodDefinition` and `functionDefinition` using this trait because I introduced a new trait `tDefinition` to share some behavior between all behavioural definitions. + +Now that our named entities are using `FASTTNamedEntity`, they have a `name` property we need to fill instead of creating an identifier node by updating the visitor. + +A first version I implemented was: + +```Samlltalk +FASTPythonVisitor>>visitIdentifier: aTSNode + (self topFastEntity isOfType: FASTTNamedEntity) ifTrue: [ ^ self topFastEntity name: (self sourceCodeOf: aNode) ]. + + ^ self handleNode: aTSNode +``` + +Here, if the parent has the trait `FASTTNamedEntity` we do not visit it but set the `name` property instead by retrieving the source code of this specific node. + +> [!TIP] +> The method `#sourceCodeOf:` takes a `TSNode` and return the source code corresponding. This method is planned to be performant since source manipulation can be a performance bottleneck. If possible, use this method instead of doing a customized source management. + +Now, this method I implemented was too naive because the identifier node is not always representing the name of an entity. + +I order to manage this, I improved the code to also check the field in some cases: + +```Smalltalk +FASTPythonVisitor>>visitIdentifier: aTSNode + self shouldIdentifierSetNameProperty ifTrue: [ ^ self topFastEntity name: (self sourceCodeOf: aNode) ]. + + ^ self handleNode: aTSNode +``` + +```Smalltalk +FASTPythonVisitor>>shouldIdentifierSetNameProperty + + ({ FASTPyGenericType. FASTPySplatType } anySatisfy: [ :class | context top isOfFASTType: class ]) ifTrue: [ ^ true ]. + + (context top field = #attribute and: [ context top isOfFASTType: FASTPyAttributeAccess ]) ifTrue: [ ^ true ]. + + ^ context top field = #name and: [ context top isOfFASTType: FASTTNamedEntity ] +``` + +This is a way to handle properties. + +Now a second example: In the node `augmented_assignment` we do not have any node to represent the operator used. In our FAST model we want this operator. + +We can add a property in our FAST entity like this: + +```Smalltalk +FASTPythonMetamodelGenerator>>defineProperties + [...] + + augmentedAssignment property: #operator type: #String. +``` + +And we can add this property in the visitor: + +```Smalltalk +visitAugmentedAssignment: aTSNode + + | fastEntity | + + fastEntity := self handleNode: aTSNode kind: FASTPyAugmentedAssignment. + + "TreeSitter does not seems to manage operator, so I try to het this property manually." + fastEntity operator: (fastEntity sourceText copyFrom: fastEntity left endPos + 1 to: fastEntity right startPos - 1) trim. + + ^ fastEntity +``` + +## Add regression tests to your project + +Writting an importer is not an easy task. If you have experience in that field, you know that each change can have unexpected side effects and that the number of pattern possible in the code are huge. + +This makes having tests really important, but it takes time to implement them and it's hard to test all possibilities. + +This work is simplified by `TSFASTAbstractImporterTest` and `TSFASTImporterTestGenerator`. + +### TSFASTAbstractImporterTest + +We provide an abstract class to subclass to implement your tests. It will retrieve the importer based of its name if the prefix are the same and allow to use the `#parse:` method to create a FAST model from a piece of code. + +But the most interesting part is the regression test generation. + +### Regression tests generation + +I implement with `TSFASTImporterTestGeneration` a way to generate regression tests. + +In order to work, I need the importer to be able to handle the code to parse. If it works, I can give the piece of code to the generator and generate a test. +The test builds a model and assert a lot of things based on Fame properties (the moose descriptions). + +For example I can do: + +```Smalltalk +code := 'yield from x'. +FASTPythonImporterTest generateTestNamed: 'yieldFrom' fromCode: code protocol: 'tests'. +``` + +And this will generate: + +```Smalltalk +testYieldFrom + + "Generated as regression test by FASTPyImporterTestGenerator>>#generateTestNamed:fromCode:protocol: + Regenerate executing: self regenerateTest: #testYieldFrom " + + self parse: 'yield from x'. + + self assert: (fast allWithType: FASTPyIdentifier) size equals: 1. + self assert: (fast allWithType: FASTPyModule) size equals: 1. + self assert: (fast allWithType: FASTPyYield) size equals: 1. + + stack push: fast rootEntities anyOne containedEntities first. + self assert: self topEntity sourceCode equals: 'yield from x'. + self assert: (self topEntity isOfType: FASTPyYield). + + "Testing value of monovalue relation expression" + self assert: self topEntity expression isNotNil. + + stack push: self topEntity expression. + self assert: self topEntity sourceCode equals: 'x'. + self assert: (self topEntity isOfType: FASTPyIdentifier) +``` + +### Customize the test generation + +We can add hooks to the test generation. + +A first hook is here to define which traits we should verify. Sometimes we want to ensure a class has some traits (for example because they are used by FAST algos). In that case, we can add them to `TSFASTAbstractImporterTest>>#traitsToTest` by overriding the method. If an entity has one of the trait returned by this method, we will assert that the entities are using the trait. + +A second customization possible is about properties. + +By default, the test generator will test all properties that: +- Are not derived (derived properties should be tested in the FAST model, not in the importer) +- Are not defining a parent (Since we are testing children and relations are bidirectional, it is useless) +- Are not nil or returning an empty collection + +It is also possible to reject some properties using `#propertiesToExclude`. By default, we reject `#startPos` and `#endPos`. + +### Regenerate all tests + +It is possible to regenerate all tests if the model had a big change using `#regenerateAllTests`. +BUT BE CAREFUL! We should ALWAYS read each test that got modifier. NEVER commit regenerated code that you do not double check, else tests are useless. + +### Generate multiple tests + +Here is a little snippet I used when developing the FAST-Python importer: + +```Smalltalk +Dictionary new + at: 'identifier' put: 'x'; + at: 'attribute' put: 'x.y'; + at: 'await' put: 'await x'; + at: 'binaryOperator' put: '1 + x'; + at: 'booleanOperator' put: 'x or y'; + at: 'call' put: 'factory()'; + at: 'comparisonOperator' put: 'x > y'; + at: 'conditionalExpression' put: 'x if True else y'; + at: 'dictionary' put: '{ 1:x }'; + at: 'dictionaryComprehension' put: '{ v:x for v in y }'; + at: 'dictionarySplat' put: '**kwargs'; + at: 'ellipsis' put: '...'; + at: 'false' put: 'False'; + at: 'float' put: '0.0'; + at: 'integer' put: '0'; + at: 'complexe' put: '0j'; + at: 'lambda' put: '(lambda x:x)'; + at: 'list' put: '[ 1 ]'; + at: 'listComprehension' put: '[i + x for i in range(3)]'; + at: 'listSplat' put: '*args'; + at: 'none' put: 'None'; + at: 'notOperator' put: '(not old)'; + "at: 'patternList' put: 'a, b';" + at: 'set' put: '{ 1 }'; + at: 'setComprehension' put: '{i + x for i in range(3) }'; + at: 'string' put: '"Hello"'; + at: 'subscript' put: 'x[2]'; + at: 'true' put: 'True'; + at: 'tuple' put: '(x, y)'; + at: 'unaryOperator' put: '-x'; + keysAndValuesDo: [ :name :code | + FASTPythonImporterTest generateTestNamed: 'calWithArgument' , name capitalized fromCode: 'f(', code , ')' protocol: 'tests - calls' ]. +``` + +## Error handling + +### Error node + +In some case it can happen that we parse some code with syntax errors or it is possible to find a bug in tree sitter. + +If this happens, we will generate a `FASTXXErrorNode` (XX been the prefix of your language). + +### Error report + +Since it is common to have errors during the import of a project (importers are hard to build since so many things can happen), we catch the errors happening during the visit of the nodes and we proceed with the visit. + +At the end of the import, the errors are inspected by default. +This behavior can be changed by overriding the method or setting `#errorReportBlock:` to do something else. + +During development you can toggle a debug mode in the Pharo "Debug" menu. + +## Utilities + +Some utilities are really great to help us customize our FAST model and implement the visitor. They are explained in the section [Tree Sitter utilities](ts_utilities.md). + +## Cyril's tips on how to work + +When doing the Python importer I used all the tools presented. + +In general my playground looked like this: + +```Smalltalk +FASTPythonMetamodelGenerator generate. + +code := 'yield from x'. +FASTPythonImporterTest generateTestNamed: 'yieldFrom' fromCode: code protocol: 'tests'. + +FASTPythonImporterTest regenerateAllTests. + +"=======================" + +Dictionary new + at: 'identifier' put: 'x'; + "at: 'attribute' put: 'x.y'; + at: 'await' put: '(await x)'; + at: 'binaryOperator' put: '1 + x'; + at: 'booleanOperator' put: 'x or y';" + at: 'call' put: 'factory()'; + "at: 'comparisonOperator' put: 'x > y';" + at: 'conditionalExpression' put: '(x if True else y)'; + "at: 'dictionary' put: '{ 1:x }'; + at: 'dictionaryComprehension' put: '{ v:x for v in y }'; + at: 'dictionarySplat' put: '**kwargs'; + at: 'ellipsis' put: '...'; + at: 'false' put: 'False'; + at: 'float' put: '0.0'; + at: 'integer' put: '0'; + at: 'complexe' put: '0j'; + at: 'lambda' put: '(lambda x:x)'; + at: 'list' put: '[ 1 ]'; + at: 'listComprehension' put: '[i + x for i in range(3)]'; + at: 'listSplat' put: '*args'; + at: 'none' put: 'None'; + at: 'notOperator' put: '(not old)'; + at: 'patternList' put: 'a, b'; + at: 'set' put: '{ 1 }'; + at: 'setComprehension' put: '{i + x for i in range(3) }';" + at: 'string' put: '"print "Hello""'; + at: 'subscript' put: 'x[2]'; +" at: 'true' put: 'True'; + at: 'tuple' put: '(x, y)'; + at: 'unaryOperator' put: '-x';" + keysAndValuesDo: [ :name :code | + FASTPythonImporterTest generateTestNamed: 'execStatementOf' , name capitalized fromCode: 'exec ', code protocol: 'tests - statements' ]. + +"=======================" + +(TSParser new language: TSLanguage python; parseString: 'yield from x' withPlatformLineEndings) rootNode. + +"=======================" + +folder := '/Users/cyril/marius/10batches' asFileReference. +folder := '/Users/cyril/testPython/cpython-main' asFileReference. + +TSSymbolsBuilderVisitor language: TSLanguage python extensions: #( 'py' ) buildOn: folder. + +"=======================" + +TSNodeFinderVisitor language: TSLanguage python extensions: #( 'py' ) selection: [ :node | + node type = #assignment and: [ node collectFieldNameOfNamedChild at: #right ifPresent: [ : n | false ] ifAbsent: [ true ] ] ] buildOn: folder. + +``` + +The first section allow to generate a test after regenerating the MM or to negenerate all tests. + +The second allow to generate a batch of tests. + +The third one allow to get a `TSTree` from a piece of code. + +The fourth one allow to have informations on all symbols present in a folder of code. + +The last one allow to detect some patterns of TS trees and find a piece of code to produce the pattern. + + +Here is the end of this documentation, hoping this can help you implement your own fast importer! \ No newline at end of file diff --git a/resources/doc/ts_utilities.md b/resources/doc/ts_utilities.md new file mode 100644 index 0000000..2f9bd64 --- /dev/null +++ b/resources/doc/ts_utilities.md @@ -0,0 +1,86 @@ +# Tree Sitter utilities + + + +- [Tree Sitter utilities](#tree-sitter-utilities) + - [Debug](#debug) + - [Inspect the structure of you tree with TSSymbolsBuilderVisitor](#inspect-the-structure-of-you-tree-with-tssymbolsbuildervisitor) + - [Look for a specific pattern in your tree with TSNodeFinderVisitor](#look-for-a-specific-pattern-in-your-tree-with-tsnodefindervisitor) + - [Find the list of possible symbols of your tree](#find-the-list-of-possible-symbols-of-your-tree) + + + +## Debug + +You might want to debug a tree sitter parser from Pharo. +Whereas we did not find _yet_ a way to use the pharo debugger. +You can create a logger attached to the parser. + +```st +callback := (TSLogCallback on: [ :payload :log_type :buffer | Transcript crShow: buffer ]). +logger := TSLogger new log: callback . +parser logger: logger. +``` + +## Inspect the structure of you tree with TSSymbolsBuilderVisitor + +This section will describe `TSSymbolsBuilderVisitor`. It is a little visitor used to understand the structure of TS nodes present in the tree you are managing. + +You can use it like this: + +```smalltalk + folder := '/Users/cyril/testPython/cpython-main' asFileReference. + TSSymbolsBuilderVisitor language: TSLanguage python extensions: #( 'py' ) buildOn: folder +``` + +And you will get a result like this: + +![Inspector](TSSymbolsBuilderVisitor.png) + +This inspector allow you to see multiple information: +- All the nodes types present in the source code parsed +- All the fields present in each node type +- The node types found in each symbols +- The cardinality of the children (for example, if you are in a node and it always have 1 child in a field, it will display `Size: 1`. If it happened there was nothing in this field in some cases and once you got 12 children in the same field, it will display: `Size: 0..12` +- By selecting a child node type, we can see an example of code with the configuration selected +- The list of possible node types in which the symbol was found + +For example, here we see at the left the list of node types found in CPython. One of them is `if_statement` and it can have 4 fields: +- `condition` that always have 1 child +- `consequence` that always have 1 child +- `alternative` that is optional and can have multiple children +- an unnamed field that is optional and can have multiple children + +We can also see a piece of code of an `else_clause` is an `alterative` field. + + +> [!NOTE] +> Be careful, you are not guarantee to have all possible child and parents since it will produce the mapping from what it encounters in the files you will provide. To be more accurate, give it the maximum number of sources possible. + +## Look for a specific pattern in your tree with TSNodeFinderVisitor + +Sometime we can find some possible patterns in the nodes but we do not know what kind of code can produce it. `TSNodeFinderVisitor` is here for this. + +This visitor can be configured with a condition to match on a node and it will inspect the first node matching it. It will also display its source code and the source code of the node highlighted in the full source. + +For example, I got suprised to find that the python `asssignment` node has the `right` field optional and I wanted to find in which case this can happen: + +```smalltalk + TSNodeFinderVisitor + language: TSLanguage python + extensions: #( 'py' ) + selection: [ :node | node type = #assignment and: [ node collectFieldNameOfNamedChild at: #right ifPresent: [ :node | false ] ifAbsent: [ true ] ] ] + buildOn: '/Users/cyril/testPython/cpython-main' asFileReference. +``` + +This will produce this: + +![Inspector](TSNodeFinderVisitor.png) + +## Find the list of possible symbols of your tree + +It is possible to extract the full list of symbols that can appear in a tree for a specific language executing this piece of code: + +```smalltalk +(TSLanguage python symbolsOfType: TSSymbolType tssymboltyperegular) collect: [ :s | TSLanguage python nameOfSymbol: s ] +``` \ No newline at end of file diff --git a/src/TreeSitter-FAST-Utils/MooseEntity.extension.st b/src/TreeSitter-FAST-Utils/MooseEntity.extension.st new file mode 100644 index 0000000..d7fb67b --- /dev/null +++ b/src/TreeSitter-FAST-Utils/MooseEntity.extension.st @@ -0,0 +1,9 @@ +Extension { #name : 'MooseEntity' } + +{ #category : '*TreeSitter-FAST-Utils' } +MooseEntity >> addGenericChild: anObject [ + "This is just for compatibility during the time we regenerate all FAST models based on TreeSitter. To remove later." + + self deprecated: 'The Metamodel needs to be regenerated in order to generate the right #addGenericChild: method on the fast entities.'. + ^ self addGenericChildren: anObject +] diff --git a/src/TreeSitter-FAST-Utils/TSFASTAbstractImporter.class.st b/src/TreeSitter-FAST-Utils/TSFASTAbstractImporter.class.st new file mode 100644 index 0000000..2e4fb85 --- /dev/null +++ b/src/TreeSitter-FAST-Utils/TSFASTAbstractImporter.class.st @@ -0,0 +1,106 @@ +" +I am an abstract importer for FAST based on TreeSitter. + +Projects implementing a FAST importer with TreeSitter should inherit from me. + +In order to work I need two things: +- To define the `tsLanguage` to use with TreeSitter. +- To have a visitor to do the actual import once we parsed the code. + +By default the visitor will be a `TSFASTVisitor` that is fully automatic. But if your goal is to have a ""clean"" FAST model, you will need to create a subclass of `TSFASTCustomizableVisitor`. + +For more information check the documentation of the project in the github repository. + +In the future I should be able to manage incremental imports. +" +Class { + #name : 'TSFASTAbstractImporter', + #superclass : 'Object', + #instVars : [ + 'errorReportBlock', + 'tsParser' + ], + #category : 'TreeSitter-FAST-Utils', + #package : 'TreeSitter-FAST-Utils' +} + +{ #category : 'testing' } +TSFASTAbstractImporter class >> isAbstract [ + + ^ self = TSFASTAbstractImporter +] + +{ #category : 'parsing' } +TSFASTAbstractImporter class >> parse: aString [ + + ^ self new parse: aString +] + +{ #category : 'parsing' } +TSFASTAbstractImporter class >> parseFile: aFileReference [ + "We could optimize by using the binary read stream of the file, but that means we would have to manage the encoding and the importer is not managing that yet. + + This implementation only accept utf8 for now. Next step would be to detect the encoding of the file before reading the contents." + + ^ self parse: aFileReference asFileReference contents +] + +{ #category : 'initialization' } +TSFASTAbstractImporter >> deleteTreeSitterParser [ + + self tsParser delete +] + +{ #category : 'accessing' } +TSFASTAbstractImporter >> errorReportBlock: aBlock [ + + errorReportBlock := aBlock +] + +{ #category : 'initialization' } +TSFASTAbstractImporter >> initialiseTreeSitterParser [ + + self tsParser: (TSParser new + language: self tsLanguage; + yourself) +] + +{ #category : 'parsing' } +TSFASTAbstractImporter >> parse: aString [ + + | tsTree fastTree | + self initialiseTreeSitterParser. + + tsTree := self tsParser parseString: aString. + + fastTree := self visitorClass new + originString: aString; + in: [ :importer | errorReportBlock ifNotNil: [ importer errorReportBlock: errorReportBlock ] ]; + import: tsTree rootNode. + + self deleteTreeSitterParser. + + ^ fastTree +] + +{ #category : 'accessing' } +TSFASTAbstractImporter >> tsLanguage [ + + ^ self subclassResponsibility +] + +{ #category : 'accessing' } +TSFASTAbstractImporter >> tsParser [ + ^ tsParser +] + +{ #category : 'accessing' } +TSFASTAbstractImporter >> tsParser: anObject [ + tsParser := anObject +] + +{ #category : 'accessing' } +TSFASTAbstractImporter >> visitorClass [ + + ^ TSFASTVisitor +] diff --git a/src/TreeSitter-FAST-Utils/TSFASTAbstractImporterTest.class.st b/src/TreeSitter-FAST-Utils/TSFASTAbstractImporterTest.class.st new file mode 100644 index 0000000..eda3da6 --- /dev/null +++ b/src/TreeSitter-FAST-Utils/TSFASTAbstractImporterTest.class.st @@ -0,0 +1,207 @@ +" +I am an abstract test case to help test FAST importers managed with TreeSitter. + +I have a system of stack used by `TSFASTImporterTestGenerator` and I provide a `#parse:` method to easily produce a model. + +## Regression tests generation + +I implement with `TSFASTImporterTestGeneration` a way to generate regression tests. + +In order to work, I need the importer to be able to handle the code to parse. If it works, I can give the piece of code to the generator and generate a test. +The test builds a model and assert a lot of things based on Fame properties (the moose descriptions). + +For example I can do: + +```st +code := 'yield from x'. +FASTPythonImporterTest generateTestNamed: 'yieldFrom' fromCode: code protocol: 'tests'. +``` + +And this will generate: + +```st +testYieldFrom + + ""Generated as regression test by FASTPyImporterTestGenerator>>#generateTestNamed:fromCode:protocol: + Regenerate executing: self regenerateTest: #testYieldFrom "" + + self parse: 'yield from x'. + + self assert: (fast allWithType: FASTPyIdentifier) size equals: 1. + self assert: (fast allWithType: FASTPyModule) size equals: 1. + self assert: (fast allWithType: FASTPyYield) size equals: 1. + + stack push: fast rootEntities anyOne containedEntities first. + self assert: self topEntity sourceCode equals: 'yield from x'. + self assert: (self topEntity isOfType: FASTPyYield). + + ""Testing value of monovalue relation expression"" + self assert: self topEntity expression isNotNil. + + stack push: self topEntity expression. + self assert: self topEntity sourceCode equals: 'x'. + self assert: (self topEntity isOfType: FASTPyIdentifier) +``` + +### Customize the test generation + +We can add hooks to the test generation. + +A first hook is here to define which traits we should verify. Sometimes we want to ensure a class has some traits (for example because they are used by FAST algos). In that case, we can add them to `TSFASTAbstractImporterTest>>#traitsToTest` by overriding the method. If an entity has one of the trait returned by this method, we will assert that the entities are using the trait. + +A second customization possible is about properties. + +By default, the test generator will test all properties that: +- Are not derived (derived properties should be tested in the FAST model, not in the importer) +- Are not defining a parent (Since we are testing children and relations are bidirectional, it is useless) +- Are not nil or returning an empty collection + +It is also possible to reject some properties using `#propertiesToExclude`. By default, we reject `#startPos` and `#endPos`. + +### Regenerate all tests + +It is possible to regenerate all tests if the model had a big change using `#regenerateAllTests`. +BUT BE CAREFUL! We should ALWAYS read each test that got modifier. NEVER commit regenerated code that you do not double check, else tests are useless. + +### Generate multiple tests + +Here is a little snippet I used when developing the FAST-Python importer: + +```st +Dictionary new + at: 'identifier' put: 'x'; + at: 'attribute' put: 'x.y'; + at: 'await' put: 'await x'; + at: 'binaryOperator' put: '1 + x'; + at: 'booleanOperator' put: 'x or y'; + at: 'call' put: 'factory()'; + at: 'comparisonOperator' put: 'x > y'; + at: 'conditionalExpression' put: 'x if True else y'; + at: 'dictionary' put: '{ 1:x }'; + at: 'dictionaryComprehension' put: '{ v:x for v in y }'; + at: 'dictionarySplat' put: '**kwargs'; + at: 'ellipsis' put: '...'; + at: 'false' put: 'False'; + at: 'float' put: '0.0'; + at: 'integer' put: '0'; + at: 'complexe' put: '0j'; + at: 'lambda' put: '(lambda x:x)'; + at: 'list' put: '[ 1 ]'; + at: 'listComprehension' put: '[i + x for i in range(3)]'; + at: 'listSplat' put: '*args'; + at: 'none' put: 'None'; + at: 'notOperator' put: '(not old)'; + ""at: 'patternList' put: 'a, b';"" + at: 'set' put: '{ 1 }'; + at: 'setComprehension' put: '{i + x for i in range(3) }'; + at: 'string' put: '""Hello""'; + at: 'subscript' put: 'x[2]'; + at: 'true' put: 'True'; + at: 'tuple' put: '(x, y)'; + at: 'unaryOperator' put: '-x'; + keysAndValuesDo: [ :name :code | + FASTPythonImporterTest generateTestNamed: 'calWithArgument' , name capitalized fromCode: 'f(', code , ')' protocol: 'tests - calls' ]. +``` +" +Class { + #name : 'TSFASTAbstractImporterTest', + #superclass : 'TestCase', + #instVars : [ + 'fast', + 'importer', + 'stack' + ], + #category : 'TreeSitter-FAST-Utils', + #package : 'TreeSitter-FAST-Utils' +} + +{ #category : 'test generation' } +TSFASTAbstractImporterTest class >> generateTestNamed: aString fromCode: code [ + + ^ self generateTestNamed: aString fromCode: code protocol: 'tests' +] + +{ #category : 'test generation' } +TSFASTAbstractImporterTest class >> generateTestNamed: aString fromCode: code protocol: protocol [ + + (TSFASTImporterTestGenerator testCase: self) generateTestNamed: aString fromCode: code protocol: protocol +] + +{ #category : 'testing' } +TSFASTAbstractImporterTest class >> isAbstract [ + + ^ self = TSFASTAbstractImporterTest +] + +{ #category : 'test generation' } +TSFASTAbstractImporterTest class >> propertiesToExclude [ + "Some properties to exclude from the tests." + + ^ #( startPos endPos ) +] + +{ #category : 'test generation' } +TSFASTAbstractImporterTest class >> regenerateAllTests [ + +