Remove the need of NodeJS for generating parsers

[XVilka]

There are no things that JavaScript can do that can't be done with Rust. It makes possible the next logical step - get rid of NodeJS for generating parsers from the description as well.

There is a number of JavaScript parsers in Rust:

• https://github.com/ratel-rust/ratel-core
• https://github.com/dherman/esprit
• https://github.com/jasonwilliams/boa
• probably more

[maxbrunsfeld]

It'd be nice to avoid depending on external programs, but in order to evaluate grammar.js files, we need a full JavaScript runtime, not just a parser. Grammars can also use require to load external files, so the runtime needs to be compatible with Node.js in that way.

We could embed a V8 or Spidermonkey interpreter inside of the Tree-sitter CLI, but it seems like overkill to me. Node.js is very easy to install and many users already have it. In fact, I think most people use node.js (and npm) to install tree-sitter.

[thestr4ng3r]

Duktape can potentially be used as a lightweight runtime.

[turbo]

Throwing this in here, too:

Duktape can potentially be used as a lightweight runtime.

Or bellard's QuickJS, which is almost twice as fast as Duktape even after a recent round of security fixes. Plus the API is nice (subjectively).

[thestr4ng3r]

I think the greatest problem with depending on nodejs is that any project that doesn't ship a pre-generated parser in their source, but generates it on the fly as part of their build system also has to depend on nodejs.
In our case, this is not acceptable. We would have to always keep an already generated parser in our source tree as a workaround, which is bad practice.

[maxbrunsfeld]

Thanks for the explanation. Personally, I think it's fine to check generated code into a source repository, as it is platform-agnostic, and it makes it easier to use the project. But if you're ok with running tree-sitter generate as part of your build, that seems like a reasonable approach.

Just so I understand your use case - why are you ok with your build depending on the tree-sitter binary but not the node binary? Would you prefer to swap out Node for some other executable JavaScript interpreter like QuickJS, DukTape, V7, Rhino, or SpiderMonkey?

Currently, Tree-sitter just looks for node on your PATH and passes JavaScript to it via stdin. We do rely on process.env and require currently, but we could probably remove the dependence on those particular APIs.

[XVilka]

Well, to my taste I don't understand why do you need the full power of JavaScript, why not just use a limited subset of it? So a parser in Rust can be written and no need for JavaScript interpreter whatsoever.

[maxbrunsfeld]

Grammars define helper functions, call methods like Array.map, and .filter, merge objects using the spread operator, etc.

JavaScript is the most widely used programming language in the world. There’s just no reason to create some custom subset with its own interpreter.

[thestr4ng3r]

Well popularity doesn't mean it's good 😜

Just so I understand your use case - why are you ok with your build depending on the tree-sitter binary but not the node binary? Would you prefer to swap out Node for some other executable JavaScript interpreter like QuickJS, DukTape, V7, Rhino, or SpiderMonkey?

The tree-sitter binary will be required in any case, but depending on nodejs just for reading in the grammar seems way too much to me. If you say you do need full javascript, I think such a small and embedded interpreter, which would then be part of the tree-sitter binary, could be a good compromise.

[turbo]

No need to devolve into JS bashing. FWIW I agree that in general, I like programs to be self-contained, but I can also understand that this is the lowest of low priorities.

I guess the better question would be: If someone were to do the work of integrating one of the smaller JS engines into tree-sitter, would that work be accepted?

[Arcanemagus]

I guess the better question would be: If someone were to do the work of integrating one of the smaller JS engines into tree-sitter, would that work be accepted?

The issue I see with this is later on when somebody not familiar with this discussion comes in and starts working on a Tree-sitter grammar, sees that they can use JavaScript files and builds it with that... only to use some JavaScript feature that the tiny custom parser doesn't understand. How are they supposed to know that it's a problem in the miniature parser and not with their code? How are they supposed to fix that?

I don't see the benefit of removing the Node.js dependency here, without entirely removing the ability to use JavaScript.

[maxbrunsfeld]

Yeah, I think Node.js needs to remain the default. I am open to generalizing the grammar-evaluating code path so that you can use an alternative JavaScript interpreter if you want. I don't think I'm interested in embedding the JavaScript interpreter into the tree-sitter binary. The vast majority of users will want to use Node.js, so it seems like unnecessary bloat.

[maxbrunsfeld]

Are folks interested in this option: allowing Tree-sitter to use different (and possibly lighter-weight) executables besides node as the JavaScript engine? If not, I think we should close this out.

[turbo]

use an alternative JavaScript interpreter if you want

Just the choice would be great. I for example use deno for JS scripting (https://github.com/denoland/deno) and not node.

[thestr4ng3r]

Yes, I think that could be a good option.