ctest: Add translation of Rust types. #4501

mbyx · 2025-06-17T17:21:45Z

Description

The ability to translate various Rust types that are translatable into equivalent C types. Additionally some basic unit tests have been added.

Sources

Checklist

Relevant tests in libc-test/semver have been updated
No placeholder or unstable values like *LAST or *MAX are
included (see #3131)
Tested locally (cd libc-test && cargo test --target mytarget);
especially relevant for platforms that may not be checked in CI

mbyx · 2025-06-17T17:24:31Z

There's quite a lot of unreachable or unimplemented because the original ctest2 did the same. If that should be changed to minimize it then please tell. This is mostly a direct port of the original code over to syn, there may be some bugs.

mbyx · 2025-06-19T14:42:36Z

Tests seem to fail at a parse int error, so rustc --version is giving a different result there for some reason. EDIT: I forgot nightly is a thing. That's embarassing.

mbyx · 2025-06-19T15:47:56Z

The error on msvc goes away if I replace const bool ON = true with #define ON true in the hierarchy.h file, but I don't know why it doesn't allow it.
freebsd-15 is spurious but freebsd-13 seems to also error on hierarchy, and most likely something in generate_files is causing it.

mbyx · 2025-06-20T06:34:41Z

From what I understand powerpc64-unknown-linux-gnu is being cross compiled on a normal linux distro, causing a mismatch in the library file and the environment it is being built in.

…running.

ctest-next/templates/test.rs

ctest-next/src/translator.rs

ctest-next/templates/constants/test_constant.c

ctest-next/templates/test.rs

tgross35 · 2025-06-25T18:54:58Z

ctest-next/build.rs

+        .or_else(|_| env::var("CC"))
+        .or_else(|_| env::var(format!("CC_{target_key}")))


For linker, the env is LD. CC is just the compiler (don't you need that too though?)

Doesn't the compiler also act as the linker if given the correct arguments? I didn't think of it that much but passing whatever is set for those three variables as the linker worked (and in the CI it's the compiler not the linker).

Often they are but not necessarily, that's the difference between the CC and LD variables. The CARGO_*_LINKER env doesn't actually need to build anything (it's only used for rustc's final link) so somebody could pass ld rather than gcc. At least, that should work but the flags do need to line up.

So you should just reexport all variables starting with CC for the cc crate (probably just iterate the env and export everything starting with CC) and then use the CARGO_*_LINKER var to the rustc invocation with -Clinker=....

In every Dockerfile, one of those three environment variables is set for the linker and it's always referencing the compiler, which is why it is like that.

Right, in our cases we use the compiler for both. But we don't want it to blow up if this isn't the case of course, so we shouldn't assume that they will always be the same.

That snip actually has a good point though, the AR_* variables also need to be forwarded.

Would they need to be? The ENV vars are set before it runs and they are not unset by cc. TARGET and HOST are only set for the duration of build.rs so they have to be forwarded. The linker and runner are forwarded there for convenience.

From it's docs:
"In addition to the above optional environment variables, cc-rs has some functions with hard requirements on some variables supplied by cargo’s build-script driver that it has the TARGET, OUT_DIR, OPT_LEVEL, and HOST variables."

So those 4 are the only ones that really have to be forwarded, linker and runner are just for convenience. In one of the previous CI runs for the powerpc64 architecture, I remember the CC_target variable was set in the debug output of cc despite ctest not setting it because it was already set by ENV.

Fair enough, you're right that we don't actually need to manually do all of these. Anything CARGO_* does need to be extracted because we need to pass it to rustc (for linking) or the runner (for qemu) since we're not going via Cargo. But the rest we can just let cc figure out.

tgross35 · 2025-06-25T18:56:30Z

ctest-next/src/generator.rs

+    }
+
+    /// Configures the host.
+    pub fn host(&mut self, host: &str) -> &mut Self {


This isn't in the original ctest; why is it needed here? cc should be able to figure out the host.

Before ctest only worked inside of build.rs which set the HOST env var for us, but ctest-next forwards those from the build.rs instead so it has to be manually set for cc inside generate. I thought that it would be better to also have the option to manually set it.

It seems like this isn't used for tests or anything currently, so let's just remove it for now. Easy to add back once/if we need it.

tgross35 · 2025-06-25T19:00:59Z

ctest-next/tests/basic.rs

+    gen.header("simple.h")
+        .generate(crate_path, "simple_out")
        .unwrap();


.header should be an included header and not an output, right? Where is simple.h coming from?

To clarify: we should be building a simple.c file (using the template) and asserting it has the same contents as tests/input/simple.c. Or, if CTEST_BLESS=1 is present, update the file instead. That way we see what the generated output looks like.

But isn't that different from the old ctest? Here it already has a manually written header file for t1.c, it uses t1.rs to generate t1gen.c and t1gen.rs, t1gen.c includes the header file t1.h and uses it to define test functions we call in t1gen.rs via ffi. Then finally it would build bin/t1.rs that includes the original t1.rs as well as the generated one, to run the tests.
So the generated C file just holds tests to be ran and uses the header file to do it.

Ah, you are correct. In that case we should have both t1.h which is handwritten, and t1.c which is the autogenerated file that gets updated if CTEST_BLESS is set.

^ I forgot we also build a Rust file; that should get blessed as well. I guess if the test file is basic.rs, name the output file basic.out.rs. Maybe also basic.out.c rather than basic.c for some consistency?

ctest-next/src/generator.rs

ctest-next/build.rs

tgross35 · 2025-06-25T19:40:41Z

ctest-next/src/tests.rs

We need to ensure that we also reject fat pointer types like &str and &[T]

ctest-next/templates/constants/test_constant.rs

ctest-next/templates/test.rs

tgross35 · 2025-06-25T19:49:24Z

ctest-next/tests/basic.rs

+    gen.header("simple.h")
+        .generate(crate_path, "simple_out")
        .unwrap();


To clarify: we should be building a simple.c file (using the template) and asserting it has the same contents as tests/input/simple.c. Or, if CTEST_BLESS=1 is present, update the file instead. That way we see what the generated output looks like.

ctest-next/templates/test.rs

ctest-next/templates/constants/test_constant.rs

ctest-next/src/runner.rs

ctest-next/src/generator.rs

tgross35 · 2025-06-26T10:35:29Z

Have you been able to run any cross build+test locally by the way? Thinking about things a bit more, I'm not sure things will work with the current setup (building in integration tests). The biggest problem is that on cross compiled targets it will be trying to run host tools from within the runner: e.g. when you're running an integration test in qemu-system-arm, it's going to try to launch rustc and cc which don't exist, or somehow try to use the host's which are for the wrong target. That probably explains the loongarch failure.

Unfortunately I think this might mean we have to use the original ctest's method of using a separate test crate, running ctest in build.rs. You're welcome to try to get things working, but I expect the cross-arch targets might not be possible without quite a bit of hacking.

For this PR: we need to have the "bless" tests that show what the generated .c and .rs files look like (#4501 (comment)) but you don't need to worry about actually compiling the tests here. (do check that the generated files build+run without errors locally, though).

mbyx · 2025-06-26T11:16:30Z

I was able to get powerpc64 to cross compile from x86_64 linux (such that only the CARGO_target_LINKER and CARGO_target_RUNNER variables had to be set), and of course non cross compiled does work. I wanted to try to cross compile loongarch locally but had problems installing the correct linker and such. I'll keep trying on the side.

Running the test doesn't invoke rustc or cc though, so qemu isn't really a problem. The only problems that can occur are during the building step when we need to make sure cargo test has access to the correct linker and runner (via the CARGO_target_LINKER and CARGO_target_RUNNER variables, these are set for all CI jobs, so this is a non issue), that cc has access to the correct linker (via CC and CC_target, this is where most problems occur), and that rustc has access to the correct linker (in theory this should just require passing CARGO_target_LINKER to it, and so far I haven't seen any problems here).

It should be possible, but I do agree it would be somewhat hacky, making sure that specific environment variables are set and forwarded.

tgross35 · 2025-06-26T11:25:55Z

Running the test doesn't invoke rustc or cc though

Doesn't it? The #[test] functions call .generate(..), which definitely seems to call cc.

mbyx · 2025-06-26T14:01:46Z

Oh wait, you're right. It wasn't a problem when I tried cross compiling powerpc64, it probably does use the host provided cc and rustc, but so long as we specify the correct linker and such with the environment variables it won't fail.

mbyx · 2025-06-27T12:46:55Z

I'm not very familiar with how Rust syntax should be translated to C, so the unit tests were made according to my understanding, if the translation is wrong I'll change the parser. Similarly I made a bunch of changes to templates/test.rs that I have no idea if they're correct (they do work for the test case I added in simple.rs).

ctest-next/Cargo.toml

ctest-next/askama.toml

ctest-next/build.rs

ctest-next/src/runner.rs

tgross35 · 2025-06-28T01:40:36Z

ctest-next/src/generator.rs

+    }
+
+    /// Configures the host.
+    pub fn host(&mut self, host: &str) -> &mut Self {


It seems like this isn't used for tests or anything currently, so let's just remove it for now. Easy to add back once/if we need it.

ctest-next/templates/test.rs

tgross35 · 2025-06-28T03:35:17Z

ctest-next/src/generator.rs

+    /// Generate the Rust and C testing files.
+    ///
+    /// Returns the path to t generated file.
+    pub fn generate_files(


For each generated file, you should do a regex to replace \n{3,} with \n\n to get rid of all the extra whitespace that the templates seem to insert.

Hm, it looks like Askama has #[template(whitespace = "suppress")] or #[template(whitespace = "minimize")]. Might be worth playing with before the regex thing.

ctest-next/templates/test.c

ctest-next/tests/input/macro.h

tgross35

Few remaining small comments, then I think we can get this merged!

Also please rebase + clean up history.

tgross35 · 2025-06-28T21:32:23Z

ctest-next/.cargo/config.toml

+[build]
+rustflags = ["--cfg", "procmacro2_semver_exempt"]


Oh, this is kind of crummy isn't it. .cargo/config.toml is kind of unreliable since it doesn't set config for anything that uses this as a dependency, I was thinking this was a Cargo feature.

So for now, could you just comment out the .file() .start() etc parts of the message with a FIXME(ctest):...? These just stabilized so we should be able to use them somewhat soon dtolnay/proc-macro2#503.

tgross35 · 2025-06-28T21:44:48Z

ctest-next/src/tests.rs

-            .collect::<Vec<_>>(),
-        ["malloc"]
+        translator.translate_type(&ty),
+        Ok("uint8_t (*const *) [5]".to_string())


I guess we can just figure this out once we have better testing with C. For now, just comment out the test with a FIXME

tgross35 · 2025-06-28T21:51:38Z

ctest-next/tests/basic.rs

+    let new_content = fs::read_to_string(&new_file)?
+        .replace("\r\n", "\n")
+        .replace("\r", "");
+    let old_content = fs::read_to_string(&old_file)?
+        .replace("\r\n", "\n")
+        .replace("\r", "");
+


You should be able to remove the first replace for each file, since the second replace does the same thing.

tgross35 · 2025-06-28T21:51:49Z

ctest-next/tests/basic.rs

+        .replace("\r", "");
+
+    let equal = new_content == old_content;
+    if env::var("LIBCBLESS").is_ok() && !equal {


Suggested change

if env::var("LIBCBLESS").is_ok() && !equal {

if env::var("LIBC_BLESS").is_ok() && !equal {

tgross35 · 2025-06-28T21:53:17Z

ctest-next/tests/basic.rs

+///
+/// If the contents do not match and LIBCBLESS is set, overwrite the
+/// test file with the content of the generated file.
+fn bless_equal(new_file: impl AsRef<Path>, old_file: impl AsRef<Path>) -> Result<bool> {


No need to have a Result here I think, you can just do the assertions in this function. Use https://docs.rs/pretty_assertions/latest/pretty_assertions/ so there is a nice diff.

tgross35 · 2025-06-28T22:40:09Z

ctest-next/templates/test.rs

+    use super::*;
+
+    use std::mem;
+    use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
+    use std::fmt::{Debug, LowerHex};
+    use std::ptr;
+    use std::slice;
+    use std::ffi::CStr;


Suggested change

use super::*;

use std::mem;

use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};

use std::fmt::{Debug, LowerHex};

use std::ptr;

use std::slice;

use std::ffi::CStr;

use std::ffi::CStr;

use std::fmt::{Debug, LowerHex};

use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};

use std::{mem, ptr, slice};

use super::*;

Making this match rustfmt

tgross35 · 2025-06-28T22:40:59Z

ctest-next/templates/test.rs

+    if FAILED.load( std::sync::atomic::Ordering::Relaxed) {
+        panic!("some tests failed");
+    } else {
+        println!("PASSED {} tests", NTESTS.load( std::sync::atomic::Ordering::Relaxed));
+    }


Suggested change

if FAILED.load( std::sync::atomic::Ordering::Relaxed) {

panic!("some tests failed");

} else {

println!("PASSED {} tests", NTESTS.load( std::sync::atomic::Ordering::Relaxed));

}

if FAILED.load(std::sync::atomic::Ordering::Relaxed) {

panic!("some tests failed");

} else {

println!(

"PASSED {} tests",

NTESTS.load(std::sync::atomic::Ordering::Relaxed)

);

}

Same thing, just making rustfmt happy with the output files

rustbot added ctest Issues relating to the ctest crate S-waiting-on-review labels Jun 17, 2025

mbyx force-pushed the ctest-c-translation branch from fc9641c to 1d766de Compare June 17, 2025 17:30

mbyx force-pushed the ctest-c-translation branch from 43a06e3 to 3c411d1 Compare June 19, 2025 15:17

mbyx force-pushed the ctest-c-translation branch from 6d9706f to 030ce77 Compare June 23, 2025 05:44

ctest: Add translation of types and test generation, compilation and …

4241857

…running.

mbyx force-pushed the ctest-c-translation branch from 030ce77 to 4241857 Compare June 23, 2025 05:54

tgross35 reviewed Jun 25, 2025

View reviewed changes

ctest-next/src/generator.rs Outdated Show resolved Hide resolved

tgross35 reviewed Jun 25, 2025

View reviewed changes

ctest-next/src/runner.rs Outdated Show resolved Hide resolved

tgross35 reviewed Jun 25, 2025

View reviewed changes

ctest-next/src/generator.rs Outdated Show resolved Hide resolved

ctest: Add translation of Rust types.

721e13e

mbyx force-pushed the ctest-c-translation branch from 7a675b9 to 721e13e Compare June 27, 2025 12:40

mbyx requested a review from tgross35 June 27, 2025 12:43

tgross35 reviewed Jun 28, 2025

View reviewed changes

ctest: Add translation of Rust types.

3d87682

mbyx requested a review from tgross35 June 28, 2025 10:42

tgross35 reviewed Jun 28, 2025

View reviewed changes

		.or_else(\|_\| env::var("CC"))
		.or_else(\|_\| env::var(format!("CC_{target_key}")))

	if env::var("LIBCBLESS").is_ok() && !equal {
	if env::var("LIBC_BLESS").is_ok() && !equal {

		[build]
		rustflags = ["--cfg", "procmacro2_semver_exempt"]

ctest: Add translation of Rust types. #4501

Are you sure you want to change the base?

ctest: Add translation of Rust types. #4501

Conversation

mbyx commented Jun 17, 2025

Description

Sources

Checklist

Uh oh!

mbyx commented Jun 17, 2025

Uh oh!

mbyx commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mbyx commented Jun 19, 2025

Uh oh!

mbyx commented Jun 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tgross35 commented Jun 26, 2025

Uh oh!

mbyx commented Jun 26, 2025

Uh oh!

tgross35 commented Jun 26, 2025

Uh oh!

mbyx commented Jun 26, 2025

Uh oh!

mbyx commented Jun 27, 2025

Uh oh!

mbyx commented Jun 19, 2025 •

edited

Loading

tgross35 left a comment •

edited

Loading