Updated based on feedback

Author: nrc

text/0000-proc-macros.md

@@ -15,11 +15,16 @@ This RFC specifies the architecture of the procedural macro system. It relies on
 [RFC 1561](https://github.com/rust-lang/rfcs/pull/1561) which specifies the
 naming and modularisation of macros. It leaves many of the details for further
 RFCs, in particular the details of the APIs available to macro authors
-(tentatively called `libmacro`). See this [blog post](http://ncameron.org/blog/libmacro/)
-for some ideas of how that might look.
+(tentatively called `libproc_macro`, formerly `libmacro`). See this
+[blog post](http://ncameron.org/blog/libmacro/) for some ideas of how that might
+look.
+
+[RFC 1681](https://github.com/rust-lang/rfcs/pull/1681) specified a mechanism
+for custom derive using 'macros 1.1'. That RFC is essentially a subset of this
+one. Changes and differences are noted throughout the text.
 
 At the highest level, macros are defined by implementing functions marked with
-a `#[macro]` attribute. Macros operate on a list of tokens provided by the
+a `#[proc_macro]` attribute. Macros operate on a list of tokens provided by the
 compiler and return a list of tokens that the macro use is replaced by. We
 provide low-level facilities for operating on these tokens. Higher level
 facilities (e.g., for parsing tokens to an AST) should exist as library crates.
@@ -57,13 +62,18 @@ like macro is used with syntax `foo!(...)`, and an attribute-like macro with
 `#[foo(...)] ...`. Macros may be used in the same places as `macro_rules` macros
 and this remains unchanged.
 
+There is also a third kind, custom derive, which are specified in [RFC
+1681](https://github.com/rust-lang/rfcs/pull/1681). This RFC extends the
+facilities open to custom derive macros beyond the string-based system of RFC
+1681.
+
 To define a procedural macro, the programmer must write a function with a
 specific signature and attribute. Where `foo` is the name of a function-like
 macro:
 
 ```
-#[macro]
-pub fn foo(TokenStream, &mut MacroContext) -> TokenStream;
+#[proc_macro]
+pub fn foo(TokenStream) -> TokenStream;
 ```
 
 The first argument is the tokens between the delimiters in the macro use.
@@ -75,8 +85,8 @@ The value returned replaces the macro use.
 Attribute-like:
 
 ```
-#[macro_attribute]
-pub fn foo(Option<TokenStream>, TokenStream, &mut MacroContext) -> TokenStream;
+#[prco_macro_attribute]
+pub fn foo(Option<TokenStream>, TokenStream) -> TokenStream;
 ```
 
 The first argument is a list of the tokens between the delimiters in the macro
@@ -101,54 +111,93 @@ by another macro without parsing, in which case they do not need to parse. The
 distinction is not statically enforced. It could be, but I don't think the
 overhead would be justified.
 
-We also introduce a special configuration option: `#[cfg(macro)]`. Items with
+Custom derive:
+
+```
+#[proc_macro_derive]
+pub fn foo(TokenStream) -> TokenStream;
+```
+
+Similar to attribute-like macros, the item a custom derive applies to must
+parse. Custom derives may on be applied to the items that a built-in derive may
+be applied to (structs and enums).
+
+Currently, macros implementing custom derive only have the option of converting
+the `TokenStream` to a string and converting a result string back to a
+`TokenStream`. This option will remain, but macro authors will also be able to
+operate directly on the `TokenStream` (which should be preferred, since it
+allows for hygiene and span support).
+
+Procedural macros which take an identifier before the argument list (e.g, `foo!
+bar(...)`) will not be supported (at least initially).
+
+My feeling is that this macro form is not used enough to justify its existence.
+From a design perspective, it encourages uses of macros for language extension,
+rather than syntactic abstraction. I feel that such macros are at higher risk of
+making programs incomprehensible and of fragmenting the ecosystem).
+
+Behind the scenes, these functions implement traits for each macro kind. We may
+in the future allow implementing these traits directly, rather than just
+implementing the above functions. By adding methods to these traits, we can
+allow macro implementations to pass data to the compiler, for example,
+specifying hygiene information or allowing for fast re-compilation.
+
+## `proc-macro` crates
+
+[Macros 1.1](https://github.com/rust-lang/rfcs/pull/1681) added a new crate
+type: proc-macro. This both allows procedural macros to be declared within the
+crate, and dictates how the crate is compiled. Procedural macros must use
+this crate type.
+
+We introduce a special configuration option: `#[cfg(proc_macro)]`. Items with
 this configuration are not macros themselves but are compiled only for macro
 uses.
 
-Initially, it will only be legal to apply `#[cfg(macro)]` to a whole crate and
-the `#[macro]` and `#[macro_attribute]` attributes may only appear within a
-`#[cfg(macro)]` crate. This has the effect of partitioning crates into macro-
-defining and non-macro defining crates. Macros may not be used in the crate in
-which they are defined, although they may be called as regular functions. In the
-future, I hope we can relax these restrictions so that macro and non-macro code
-can live in the same crate.
+If a crate is a `proc-macro` crate, then the `proc_macro` cfg variable is true
+for the whole crate. Initially it will be false for all other crates. This has
+the effect of partitioning crates into macro- defining and non-macro defining
+crates. In the future, I hope we can relax these restrictions so that macro and
+non-macro code can live in the same crate.
 
 Importing macros for use means using `extern crate` to make the crate available
 and then using `use` imports or paths to name macros, just like other items.
 Again, see [RFC 1561](https://github.com/rust-lang/rfcs/pull/1561) for more
 details.
 
-When a `#[cfg(macro)]` crate is `extern crate`ed, it's items (even public ones)
-are not available to the importing crate; only macros declared in that crate.
-There should be a lint to warn about public items which will not be visible due
-to `#[cfg(macro)]`. The crate is dynamically linked with the compiler at
-compile-time, rather than with the importing crate at runtime.
+When a `proc-macro` crate is `extern crate`ed, it's items (even public ones) are
+not available to the importing crate; only macros declared in that crate. There
+should be a lint to warn about public items which will not be visible due to
+`proc_macro`. The crate is used by the compiler at compile-time, rather than
+linked with the importing crate at runtime.
+
+[Macros 1.1](https://github.com/rust-lang/rfcs/pull/1681) required `#[macro_use]`
+on `extern crate` which imports procedural macros. This will not be required
+and should be deprecated.
 
 
 ## Writing procedural macros
 
 Procedural macro authors should not use the compiler crates (libsyntax, etc.).
-Using these will remain unstable. We will make available a new crate, libmacro,
-which will follow the usual path to stabilisation, will be part of the Rust
-distribution, and will be required to be used by procedural macros (because, at
-the least, it defines the types used in the required signatures).
+Using these will remain unstable. We will make available a new crate,
+libproc_macro, which will follow the usual path to stabilisation, will be part
+of the Rust distribution, and will be required to be used by procedural macros
+(because, at the least, it defines the types used in the required signatures).
 
-The details of libmacro will be specified in a future RFC. In the meantime, this
-[blog post](http://ncameron.org/blog/libmacro/) gives an idea of what it might
-contain.
+The details of libproc_macro will be specified in a future RFC. In the meantime,
+this [blog post](http://ncameron.org/blog/libmacro/) gives an idea of what it
+might contain.
 
-The philosophy here is that libmacro will contain low-level tools for
+The philosophy here is that libproc_macro will contain low-level tools for
 constructing macros, dealing with tokens, hygiene, pattern matching, quasi-
 quoting, interactions with the compiler, etc. For higher level abstractions
 (such as parsing and an AST), macros should use external libraries (there are no
-restrictions on `#[cfg(macro)]` crates using other crates).
+restrictions on `#[cfg(proc_macro)]` crates using other crates).
 
-The `MacroContext` is an object passed to all procedural macro definitions. It
-is the main entry point to the libmacro API and for interaction with the
-compiler. Via the `MacroContext`, a procedural macro can access information
-about the context in which it is used and defined, and perform operations which
-rely on the state of the compiler. It will be more fully defined in the upcoming
-RFC proposing libmacro.
+A `MacroContext` is an object placed in thread-local storage when a macro is
+expanded. It contains data about how the macro is being used and defined. It is
+expected that for most uses, macro authors will not use the `MacroContext`
+directly, but it will be used by library functions. It will be more fully
+defined in the upcoming RFC proposing libproc_macro.
 
 Rust macros are hygienic by default. Hygiene is a large and complex subject, but
 to summarise: effectively, naming takes place in the context of the macro
@@ -157,8 +206,8 @@ definition, not the expanded macro.
 Procedural macros often want to bend the rules around macro hygiene, for example
 to make items or variables more widely nameable than they would be by default.
 Procedural macros will be able to take part in the application of the hygiene
-algorithm via libmacro. Again, full details must wait for the libmacro RFC and a
-sketch is available in this [blog post](http://ncameron.org/blog/libmacro/).
+algorithm via libproc_macro. Again, full details must wait for the libproc_macro
+RFC and a sketch is available in this [blog post](http://ncameron.org/blog/libmacro/).
 
 
 ## Tokens
@@ -219,12 +268,12 @@ pub enum TokenKind {
     // The content of the comment can be found from the span.
     Comment(CommentKind),
 
-    // Symbol is the string contents, not including delimiters. It would be nice
+    // `text` is the string contents, not including delimiters. It would be nice
     // to avoid an allocation in the common case that the string is in the
-    // source code. We might be able to use `&'Codemap str` or something.
-    // `Option<usize> is for the count of `#`s if the string is a raw string. If
+    // source code. We might be able to use `&'codemap str` or something.
+    // `raw_markers` is for the count of `#`s if the string is a raw string. If
     // the string is not raw, then it will be `None`.
-    String(Symbol, Option<usize>, StringKind),
+    String { text: Symbol, raw_markers: Option<usize>, kind: StringKind },
 
     // char literal, span includes the `'` delimiters.
     Char(char),
@@ -269,6 +318,10 @@ pub enum StringKind {
 pub struct Symbol { ... }
 ```
 
+Note that although tokens exclude whitespace, by examining the spans of tokens,
+a procedural macro can get the string representation of a `TokenStream` and thus
+has access to whitespace information.
+
 ### Open question: `Punctuation(char)` and multi-char operators.
 
 Rust has many compound operators, e.g., `<<`. It's not clear how best to deal
@@ -294,14 +347,14 @@ Some solutions:
 ## Staging
 
 1. Implement [RFC 1561](https://github.com/rust-lang/rfcs/pull/1561).
-2. Implement `#[macro]` and `#[cfg(macro)]` and the function approach to
+2. Implement `#[proc_macro]` and `#[cfg(proc_macro)]` and the function approach to
    defining macros. However, pass the existing data structures to the macros,
    rather than tokens and `MacroContext`.
-3. Implement libmacro and make this available to macros. At this stage both old
+3. Implement libproc_macro and make this available to macros. At this stage both old
    and new macros are available (functions with different signatures). This will
    require an RFC and considerable refactoring of the compiler.
 4. Implement some high-level macro facilities in external crates on top of
-   libmacro. It is hoped that much of this work will be community-led.
+   libproc_macro. It is hoped that much of this work will be community-led.
 5. After some time to allow conversion, deprecate the old-style macros. Later,
    remove old macros completely.
 
@@ -345,7 +398,10 @@ latter can be written today, the former require more work on an interface to the
 compiler to be practical).
 
 We could use the `macro` keyword rather than the `fn` keyword to declare a
-macro. We would then not require a `#[macro]` attribute.
+macro. We would then not require a `#[proc_macro]` attribute.
+
+We could use `#[macro]` instead of `#[proc_macro]` (and similarly for the other
+attributes). This would require making `macro` a contextual keyword.
 
 We could have a dedicated syntax for procedural macros, similar to the
 `macro_rules` syntax for macros by example. Since a procedural macro is really
@@ -365,31 +421,6 @@ would require additional rules on token trees and may not be possible.
 # Unresolved questions
 [unresolved]: #unresolved-questions
 
-### macros with an extra identifier
-
-We currently allow procedural macros to take an extra ident after the macro name
-and before the arguments, e.g., `foo! bar(...)` where `foo` is the macro name
-and `bar` is the extra identifier. This is used for `macro_rules` and is useful
-for macros which define classes of items, rather than instances of items. E.g.,
-a `struct!` macro might be used similarly to the `struct` keyword.
-
-My feeling is that this macro form is not used enough to justify its existence.
-From a design perspective, it encourages uses of macros for language extension,
-rather than syntactic abstraction. I feel that such macros are at higher risk of
-making programs incomprehensible and of fragmenting the ecosystem).
-
-Therefore, I would like to remove them from the language. Alternatively, they
-could be incorporated into the new design by having another kind of macro
-function:
-
-```
-#[macro_with_ident]
-pub fn foo(&Token, TokenStream, &mut MacroContext) -> TokenStream;
-```
-
-where the first argument is the extra identifier.
-
-
 ### Linking model
 
 Currently, procedural macros are dynamically linked with the compiler. This