ast trivia.md

Author: basarat

SUMMARY.md

@@ -34,3 +34,4 @@
   * [Program](docs/compiler/program.md)
   * [AST](docs/compiler/ast.md)
     * [TIP: Visit Children](docs/compiler/ast-tip-children.md)
+    * [Trivia](docs/compiler/ast-trivia.md)

docs/compiler/ast-trivia.md

@@ -0,0 +1,48 @@
+### Trivia
+Trivia (called that because its `trivial`) represent the parts of the source text that are largely insignificant for normal understanding of the code, such as whitespace, comments, and even conflict markers. Trivia is *not stored* in the AST (to keep it lightweight). However it can be fetched *on demand* using a few `ts.` APIs. Before we show them you need to understand
+
+#### Trivia Ownership
+In General:
+* A token owns any trivia after it on the *same* line *upto* the next token.
+* Any comment *after that line* is associated with the following token.
+
+For leading and ending comments in a file:
+* The first token in the source file gets all the initial trivia
+* The last sequence of trivia in the file is tacked onto the end-of-file token, which otherwise has zero width.
+
+The first token in the source file gets all the initial trivia, and the last sequence of trivia in the file is tacked onto the end-of-file token, which otherwise has zero width.
+
+#### Trivia APIs
+For most basic uses, comments are the "interesting" trivia. The comments that belong to a Node which can be fetched through the following functions:
+
+Function | Description
+---------|------------
+`ts.getLeadingCommentRanges` | Given the source text and position within that text, returns ranges of comments between the first line break following the given position and the token itself (probably most useful with `ts.Node.getFullStart`).
+`ts.getTrailingCommentRanges` | Given the source text and position within that text, returns ranges of comments until the first line break following the given position (probably most useful with `ts.Node.getEnd`).
+
+As an example, imagine this portion of a source file:
+
+```ts
+debugger;/*hello*/
+    //bye
+  /*hi*/    function
+```
+
+`getLeadingCommentRanges` for the `function` will only return the last 2 comments `//bye` and `/*hi*/`.
+
+Appropriately, calling `getTrailingCommentRanges` on the end of the debugger statement will extract the `/*hello*/` comment.
+
+#### Token Start/Full Start
+Nodes have what is called a "token start" and a "full start".
+
+* Token Start: the more natural version, which is the position in file where the text of a token begins
+* Full Start: the point at which the scanner began scanning since the last significant token
+
+AST nodes have an API for `getStart` and `getFullStart`. In the following example:
+
+```ts
+debugger;/*hello*/
+    //bye
+  /*hi*/    function
+```
+for `function` the token start is at `function` whereas *full* start is at `/*hello*/`. Note that full start even includes the trivia that would otherwise be owned by the previous node.

docs/compiler/overview.md

@@ -29,3 +29,6 @@ An AST node.
 
 ## File: System
 `system.ts`. All interaction of the TypeScript compiler with the operating system goes through a `System` interface. Both the interface and its implementations (`WScript` and `Node`) are defined in `system.ts`. You can think of it as the *Operating Environment* (OE).
+
+
+Note: Thanks to the TypeScript team for providing much of the docs : https://github.com/Microsoft/TypeScript/wiki/Architectural-Overview that are used to write this story.

docs/compiler/scanner.md

@@ -0,0 +1,5 @@
+
+
+
+### Standalone scanner
+You can create a standalone scanner using `createScanner` and use `setText`/`setTextPos` to scan at different points in a file.