Port the DSL to Rust

[sunfishcode]

Cretonne's Python-based DSL allows target ISAs to be described in a concise manner, and can generate tables, code, and documentation from them. It's common for compilers to use DSLs for such purposes; see also LLVM's tablegen, GCC's machine description language, libfirm's perl (example) and python (example), and others.

Python compares pretty favorably to obvious alternatives, in particular by being a real programming language, and a fairly friendly one at that. However, using Python in Cretonne has some downsides too:

• There's extra overhead for developers that don't work with Python on a regular basis. They may not have mypy, flake8, or python-aware or even mypy-aware IDE plugins installed, they may not remember how variable scoping works, and so on.
• The current Python code makes use of some "magical" features, such as __getitem__, isinstance, etc. which may be fine for people who work with Python regularly, but which make it harder to follow the code for people who aren't.
• The current Python code also behaves in "magical" ways sometimes, such as PUT_OP in the x86 target. It's used in Rust code, but it's not a Rust construct; it's edited out by replace_put_op. In some cases, it's replaced by put_mp2, but it's not easy to determine when that name is used, as there are no other occurrences of put_mp2 or mp2 in the repository. As another example, the DSL infers instruction formats from instruction fields implicitly, in a way that makes it confusing when one is adding a new format and accidentally colliding with an existing format.
• Cretonne has ambitions of being able to compile Rust code in the future, so it would be cooler if it could compile all of itself.

None of these are show-stoppers, and one option is to just keep the current code and refactor it to help out with some of these issues. But we should also consider where we want to be in the long term; it'd be easier to do a large architectural change now, while the code is (relatively) simple, as it'll grow more complex in the future as we add more features. One option is Rust's procedural macros, except that they're not yet stable. A possible intermediate step could be to use a build.rs.

The instruction descriptions are fairly elaborate, so one way to start experimenting with this would be to start by rewriting some of the simper parts, such as just gen_types.py, to get started setting up the infrastructure. gen_types.py's output file is target/debug/build/cretonne-codegen-*/out/types.rs, which is pretty simple and demonstrates what it does.

If anyone's interested in working on this, I'd be happy to mentor.

[data-pup]

Implementing this in Rust seems like it would be nice, given that the rest of the codebase is in Rust. At least from afar, I can't think of any reasons that would make switching to another language similar to Python beneficial.

Whatever is chosen as the direction forward however, happy to help out with this 😄

[sunfishcode]

Thinking about this more, I think using Rust, with a build.rs approach, is the best one right now.

So as I mentioned above, I think gen_types.py is a good initial goal. And looking at it more closely, gen_types.py depends on srcgen.py. And, srcgen.py is a very simple file that should be very straightforward.

So the very first step here is to make a Rust version of srcgen.py. It's a simple file that just makes it easy to produce Rust source code, managing things like indent levels.

@data-pup Are you interested? :-).

[data-pup]

I'm in, that sounds fun! I'll start looking into this, and I can follow up if I have further questions. I'll keep you posted in IRC too :o)

[sunfishcode]

Woohoo! The first step, #403, is now merged!

The next logical step is to look at the compiler settings, and in particular settings.rs. It's generated by gen_settings.py, which also generates target-specific settings files, but we can split those out into a separate step. That way, we won't need the isas in this step.

https://github.com/CraneStation/cranelift/blob/master/lib/codegen/meta/base/settings.py shows what settings are all about. Note that SettingsGroup's constructor registers the created object in a static _current variable, and all the settings are automatically collected from Python's globals() at the end. This is an example of the "DSL" approach, and it does make the file a little more streamlined. A less magical approach would be to just add settings to a SettingsGroup object explicitly, which would also be more Rust-friendly.

@data-pup How does this sound to you? :-)

[data-pup]

I can start looking into this further this weekend, but that sounds good to me! I'll let you know if I have any further questions about that 😺

[lachlansneff]

Procedural macros are stable now. (Or will be soon).

rust-lang/rust#52081

It looks like you can define a procedural macro, but not import it yet.

[data-pup]

I think the plan is definitely to move towards procedural macros in the future. Currently however, cranelift supports 1.25, which I do not believe would support this? @sunfishcode would know more regarding that, but currently we'll have to work around that :)

[sunfishcode]

That's right. However, if there's a major advantage to switching to procedural macros now, let's talk about it, as we may have options.

[sunfishcode]

Ok, after experimenting with several approaches with cargo publish, it seems we really need to publish the meta crate, so I've now made that change (cranelift-codegen-meta).

The next step here is the settings.rs file, which @data-pup expressed an interest in.

[data-pup]

🎉 Yay! Glad we've figured out the publishing thing. I've started looking into the settings.rs thing, I can keep you updated on progress over IRC or here :)

[lachlansneff]

Another way to do this would be to use a dynamic language like dyon (which is written in rust) in build.rs.

[bnjbvr]

Hey @data-pup, I'd like to help on this issue, but I wouldn't want to step onto your toes. Are you still working on the settings.rs file? Or have you been working on other files? If not, I'll start migrating other stuff, and will make sure to announce it here first.