Skip to content

Commit 33262d2

Browse files
ManishearthManishearth
committed
eRFC: Post Build Contexts
1 parent 743efe4 commit 33262d2

File tree

1 file changed

+350
-0
lines changed

1 file changed

+350
-0
lines changed

‎text/0000-erfc-post-build-contexts.md

+350
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,350 @@
1+
- Feature Name: post_build_contexts
2+
- Start Date: 2018-01-25
3+
- RFC PR: (leave this empty)
4+
- Rust Issue: (leave this empty)
5+
6+
# Summary
7+
[summary]: #summary
8+
9+
This is an *experimental RFC* for adding the ability to integrate custom test/bench/etc frameworks ("post-build frameworks") in Rust.
10+
11+
# Motivation
12+
[motivation]: #motivation
13+
14+
Currently, Rust lets you write unit tests with a `#[test]` attribute. We also have an unstable `#[bench]` attribute which lets one write benchmarks.
15+
16+
In general it's not easy to use your own testing strategy. Implementing something that can work
17+
within a `#[test]` attribute is fine (`quickcheck` does this with a macro), but changing the overall
18+
strategy is hard. For example, `quickcheck` would work even better if it could be done as:
19+
20+
```rust
21+
#[quickcheck]
22+
fn test(input1: u8, input2: &str) {
23+
// ...
24+
}
25+
```
26+
27+
If you're trying to do something other than testing, you're out of luck -- only tests, benches, and examples
28+
get the integration from `cargo` for building auxiliary binaries the correct way. [cargo-fuzz] has to
29+
work around this by creating a special fuzzing crate that's hooked up the right way, and operating inside
30+
of that. Ideally, one would be able to just write fuzz targets under `fuzz/`.
31+
32+
[Compiletest] (rustc's test framework) would be another kind of thing that would be nice to
33+
implement this way. Currently it compiles the test cases by manually running `rustc`, but it has the
34+
same problem as cargo-fuzz where getting these flags right is hard. This too could be implemented as
35+
a custom test framework.
36+
37+
A profiling framework may want to use this mode to instrument the binary in a certain way. We
38+
can already do this via proc macros, but having it hook through `cargo test` would be neat.
39+
40+
Overall, it would be good to have a generic framework for post-build steps that can support use
41+
cases like `#[test]` (both the built-in one and quickcheck), `#[bench]` (both built in and custom
42+
ones like [criterion]), `examples`, and things like fuzzing. While we may not necessarily rewrite
43+
the built in test/bench/example infra in terms of the new framework, it should be possible to do so.
44+
45+
The main two problems that we need to solve are:
46+
47+
- Having a nice API for generating custom post-build binaries
48+
- Having good `cargo` integration so that custom tests are at the same level of integration as regular tests as far as build processes are concerned
49+
50+
[cargo-fuzz]: https://github.com/rust-fuzz/cargo-fuzz
51+
[criterion]: https://github.com/japaric/criterion.rs
52+
[Compiletest]: http://github.com/laumann/compiletest-rs
53+
54+
# Detailed proposal
55+
[detailed-proposal]: #detailed-proposal
56+
57+
(As an eRFC I'm merging the "guide-level/reference-level" split for now; when we have more concrete
58+
ideas we can figure out how to frame it and then the split will make more sense)
59+
60+
## Procedural macro for a new post-build context
61+
62+
A custom post-build context is essentially a whole-crate procedural
63+
macro that is evaluated after all other macros in the target crate have
64+
been evaluated. It is passed the `TokenStream` for every element in the
65+
target crate that has a set of attributes the post-build context has
66+
registered interest in. Essentially:
67+
68+
```rust
69+
extern crate proc_macro;
70+
use proc_macro::TokenStream;
71+
72+
// attributes() is optional
73+
#[post_build_context(test, attributes(foo, bar))]
74+
pub fn like_todays_test(items: &[AnnotatedItem]) -> TokenStream {
75+
// ...
76+
}
77+
```
78+
79+
where
80+
81+
```rust
82+
struct AnnotatedItem
83+
tokens: TokenStream,
84+
span: Span,
85+
attributes: TokenStream,
86+
path: SomeTypeThatRepresentsPathToItem
87+
}
88+
```
89+
90+
`items` here contains an `AnnotatedItem` for every element in the
91+
target crate that has one of the attributes declared in `attributes`
92+
along with attributes sharing the name of the context (`test`, here).
93+
94+
An post-build context could declare that it reacts to multiple different
95+
attributes, in which case it would get all items with any of the
96+
listed attributes. These items be modules, functions, structs,
97+
statics, or whatever else the post-build context wants to support. Note
98+
that the post-build context function can only see all the annotated
99+
items, not modify them; modification would have to happen with regular
100+
procedural macros The returned `TokenStream` will become the `main()`
101+
when this post-build context is used.
102+
103+
Because this procedural macro is only loaded when it is used as the
104+
post-build context, the `#[test]` annotation should probably be kept
105+
behind `#[cfg(test)]` so that you don't get unknown attribute warnings
106+
whilst loading. (We could change this by asking attributes to be
107+
registered in Cargo.toml, but we don't find this necessary)
108+
109+
## Cargo integration
110+
111+
Alternative post-build contexts need to integrate with cargo.
112+
In particular, when crate `a` uses a crate `b` which provides an
113+
post-build context, `a` needs to be able to specify when `b`'s post-build
114+
context should be used. Furthermore, cargo needs to understand that when
115+
`b`'s post-build context is used, `b`'s dependencies must also be linked.
116+
Note that `b` could potentially provide multiple post-build contexts ---
117+
these are named according to the name of their `#[post_build_context]`
118+
function.
119+
120+
Crates which define an post-build context must have an `post-build-context = true`
121+
key.
122+
123+
For crates that wish to *use* a custom post-build context, they do so by
124+
defining a new post-build context under a new `post-build` section in
125+
their `Cargo.toml`:
126+
127+
```toml
128+
[post-build.context.fuzz]
129+
provider = { rust-fuzz = "1.0" }
130+
folder = "fuzz/"
131+
specify-single-target = true # false by default
132+
```
133+
134+
This defines an post-build context named `fuzz`, which uses the
135+
implementation provided by the `rust-fuzz` crate. When run, it will be
136+
applies to all files in the `fuzz` directory. `specify-single-target`
137+
addresses whether it must be run with a single target. If true, you will
138+
be forced to run `cargo post-build foobar --test foo`. This is useful for cases
139+
like `cargo-fuzz` where running tests on everything isn't possible.
140+
141+
By default, the following contexts are defined:
142+
143+
```toml
144+
[post-build.context.test]
145+
provider = { test = "1.0", context = "test" }
146+
folder = "tests/"
147+
148+
[post-build.context.bench]
149+
provider = { test = "1.0", context = "bench" }
150+
folder = ["benchmarks/", "morebenchmarks/"]
151+
```
152+
153+
These can be overridden by a crate's `Cargo.toml`. The `context`
154+
property is used to disambiguate when a single crate has multiple
155+
functions tagged `#[post_build_context]` (if we were using the example
156+
post-build provider further up, we'd give `like_todays_test` here).
157+
`test` here is `libtest`, though note that it could be maintained
158+
out-of-tree, and shipped with rustup.
159+
160+
To invoke a particular post-build context, a user invokes `cargo post-build
161+
<context>`. `cargo test` and `cargo bench` are aliases for `cargo
162+
post-build test` and `cargo post-build bench` respectively. Any additional
163+
arguments are passed to the post-build context binary. By convention, the
164+
first position argument should allow filtering which
165+
test/benchmarks/etc. are run.
166+
167+
168+
By default, the crate has an implicit "test", "bench", and "example" context that use the default libtest stuff.
169+
(example is a no-op context that just runs stuff). However, declaring a context with the name `test`
170+
will replace the existing `test` context. In case you wish to supplement the context, use a different
171+
name.
172+
173+
By default, `cargo test` will run doctests and the `test` and `examples` context. This can be customized:
174+
175+
```toml
176+
[post-build.set.test]
177+
contexts = [test, quickcheck, examples]
178+
```
179+
180+
This means that `cargo test` will, aside from doctests, run `cargo post-build test`, `cargo test post-build quickcheck`,
181+
and `cargo test post-build examples` (and similar stuff for `cargo bench`). It is not possible to make `cargo test`
182+
_not_ run doctests.
183+
184+
There are currently only two custom post-build sets (test and bench).
185+
186+
Custom test targets can be declared via `[[post-build.target]]`
187+
188+
```toml
189+
[[post-build.target]]
190+
context = fuzz
191+
path = "foo.rs"
192+
name = "foo"
193+
```
194+
195+
`[[test]]` is an alias for `[[post-build.target]] context = test` (same goes for `[[bench]]` and `[[example]]`).
196+
197+
198+
The generated test binary should be able to take one identifier argument, used for narrowing down what tests to run.
199+
I.e. `cargo test --kind quickcheck my_test_fn` will build the test(s) and call them with `./testbinary my_test_fn`.
200+
Typically, this argument is used to filter tests further; test harnesses should try to use it for the same purpose.
201+
202+
203+
## To be designed
204+
205+
This contains things which we should attempt to solve in the course of this experiment, for which this eRFC
206+
does not currently provide a concrete proposal.
207+
208+
### Standardizing the output
209+
210+
We should probably provide a crate with useful output formatters and stuff so that if test harnesses desire, they can
211+
use the same output formatting as a regular test. This also provides a centralized location to standardize things
212+
like json output and whatnot.
213+
214+
@killercup is working on a proposal for this which I will try to work in.
215+
216+
### Configuration
217+
218+
Currently we have `cfg(test)` and `cfg(bench)`. Should `cfg(test)` be applied to all? Should `cfg(nameofharness)`
219+
be used instead? Ideally we'd have a way when declaring a framework to declare what cfgs it should be built with.
220+
221+
# Drawbacks
222+
[drawbacks]: #drawbacks
223+
224+
- This adds more sections to `Cargo.toml`.
225+
- This complicates the execution path for cargo, in that it now needs
226+
to know about post-build contexts and sets.
227+
- Flags and command-line parameters for test and bench will now vary
228+
between post-build contexts, which may confuse users as they move
229+
between crates.
230+
231+
# Rationale and alternatives
232+
[alternatives]: #alternatives
233+
234+
We should either do this or stabilize the existing bencher.
235+
236+
## Alternative procedural macro
237+
238+
An alternative proposal was to expose an extremely general whole-crate proc macro:
239+
240+
```rust
241+
#[post_build_context(test, attributes(foo, bar))]
242+
pub fn context(crate: TokenStream) -> TokenStream {
243+
// ...
244+
}
245+
```
246+
247+
and then we can maintain a helper crate, out of tree, that uses `syn` to provide a nicer
248+
API, perhaps something like:
249+
250+
```rust
251+
fn clean_entry_point(tree: syn::ItemMod) -> syn::ItemMod;
252+
253+
trait TestCollector {
254+
fn fold_function(&mut self, path: syn::Path, func: syn::ItemFn) -> syn::ItemFn;
255+
}
256+
257+
fn collect_tests<T: TestCollector>(collector: &mut T, tree: syn::ItemMod) -> ItemMod;
258+
```
259+
260+
This lets us continue to develop things outside of tree without perma-stabilizing an API;
261+
and it also lets us provide a friendlier API via the helper crate.
262+
263+
It also lets crates like `cargo-fuzz` introduce things like a `#![no_main]` attribute or do
264+
other antics.
265+
266+
Finally, it handles the "profiling framework" case as mentioned in the motivation. On the other hand,
267+
these tools usually operate at a differeny layer of abstraction so it might not be necessary.
268+
269+
A major drawback of this proposal is that it is very general, and perhaps too powerful. We're currently using the
270+
more focused API in the eRFC, and may switch to this during experimentation if a pressing need crops up.
271+
272+
# Unresolved questions
273+
[unresolved]: #unresolved-questions
274+
275+
These are mostly intended to be resolved during the experimental
276+
feature.
277+
278+
## Integration with doctests
279+
280+
Documentation tests are somewhat special, in that they cannot easily be
281+
expressed as `TokenStream` manipulations. In the first instance, the
282+
right thing to do is probably to have an implicitly defined execution
283+
context called `doctest` which is included in the execution context set
284+
`test` by default.
285+
286+
Another argument for punting on doctests is that they are intended to
287+
demonstrate code that the user of a library would write. They're there
288+
to document *how* something should be used, and it then makes somewhat
289+
less sense to have different "ways" of running them.
290+
291+
## Translating existing cargo test flags
292+
293+
Today, `cargo test` takes a number of flags such as `--lib`, `--test
294+
foo`, and `--doc`. As it would be a breaking change to change these,
295+
cargo should recognize them and map to the appropriate execution
296+
contexts.
297+
298+
Currently, `cargo test` lets you pick a single testing target via `--test`,
299+
and `cargo bench` via `--bench`. We'll need to create an agnostic flag
300+
for `cargo post-build` (we cannot use `--target` because it is already used for
301+
the target architecture, and `--test` is too specific for tests). `--post-build-target`
302+
is one rather verbose suggestion.
303+
304+
## Standardizing the output
305+
306+
We should probably provide a crate with useful output formatters and
307+
stuff so that if test harnesses desire, they can use the same output
308+
formatting as a regular test. This also provides a centralized location
309+
to standardize things like json output and whatnot.
310+
311+
## Configuration
312+
313+
Currently we have `cfg(test)` and `cfg(bench)`. Should `cfg(test)` be
314+
applied to all? Should `cfg(post_build_context)` be used instead?
315+
Ideally we'd have a way when declaring an post-build context to declare
316+
what cfgs it should be built with.
317+
318+
## Runtime dependencies and flags
319+
320+
The generated harness itself may have some dependencies. Currently there's
321+
no way for the post-build context to specify this. One proposal is for the crate
322+
to specify _runtime_ dependencies of the post-build context via:
323+
324+
```toml
325+
[post-build.dependencies]
326+
libfuzzer-sys = ...
327+
```
328+
329+
If a crate is currently running this post-build context, its dev-dependencies
330+
will be semver-merged with the post-build-context.dependencies.
331+
332+
However, this may not be strictly necessary. Custom derives have
333+
a similar problem and they solve it by just asking users to import the correct
334+
crate and keep it in their dev-dependencies.
335+
336+
## Other questions
337+
338+
- The general syntax and toml stuff should be approximately settled on
339+
before this eRFC merges, but iterated on later. "execution contexts"
340+
or "post-build contexts" are interesting names but we might want to try for better
341+
- Should an post-build context be able to declare "defaults" for what
342+
folders and post-build sets it should be added to? This might save
343+
users from some boilerplate in a large number of situations.
344+
- Should we be shipping a bencher by default at all (i.e., in libtest)?
345+
Could we instead default `cargo bench` to a `rust-lang-nursery`
346+
crate?
347+
- `specify-single-target = true` probably should be specified by the execution context itself,
348+
not the consumer. It's also questionable if it's necessary -- cargo-fuzz is going to need
349+
a wrapper script anyway, so it's fine if the CLI isn't as ergonomic for that use case.
350+

0 commit comments

Comments
 (0)