dtolnay/proc-macro2

proc-macro2

A wrapper around the procedural macro API of the compiler's proc_macro crate. This library serves two purposes:

• Bring proc-macro-like functionality to other contexts like build.rs and main.rs. Types from proc_macro are entirely specific to procedural macros and cannot ever exist in code outside of a procedural macro. Meanwhile proc_macro2 types may exist anywhere including non-macro code. By developing foundational libraries like syn and quote against proc_macro2 rather than proc_macro, the procedural macro ecosystem becomes easily applicable to many other use cases and we avoid reimplementing non-macro equivalents of those libraries.

• Make procedural macros unit testable. As a consequence of being specific to procedural macros, nothing that uses proc_macro can be executed from a unit test. In order for helper libraries or components of a macro to be testable in isolation, they must be implemented using proc_macro2.

Usage

[dependencies]
proc-macro2 = "1.0"

The skeleton of a typical procedural macro typically looks like this:

extern crate proc_macro;

#[proc_macro_derive(MyDerive)]
pub fn my_derive(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
    let input = proc_macro2::TokenStream::from(input);

    let output: proc_macro2::TokenStream = {
        /* transform input */
    };

    proc_macro::TokenStream::from(output)
}

If parsing with Syn, you'll use parse_macro_input! instead to propagate parse errors correctly back to the compiler when parsing fails.

Unstable features

The default feature set of proc-macro2 tracks the most recent stable compiler API. Functionality in proc_macro that is not yet stable is not exposed by proc-macro2 by default.

To opt into the additional APIs available in the most recent nightly compiler, the procmacro2_semver_exempt config flag must be passed to rustc. We will polyfill those nightly-only APIs back to Rust 1.56.0. As these are unstable APIs that track the nightly compiler, minor versions of proc-macro2 may make breaking changes to them at any time.

RUSTFLAGS='--cfg procmacro2_semver_exempt' cargo build

Note that this must not only be done for your crate, but for any crate that depends on your crate. This infectious nature is intentional, as it serves as a reminder that you are outside of the normal semver guarantees.

Semver exempt methods are marked as such in the proc-macro2 documentation.

License

^{Licensed under either of Apache License, Version 2.0 or MIT license at your option.}
_{Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.}

proc-macro2's Issues

Derive traits for LineColumn

proc_macro::LineColumn has #[derive(Copy, Clone, Debug, PartialEq, Eq)] but we provide none of those.

More useful Debug for Symbol

Currently:

Symbol(Symbol { intern: 1, not_send_sync: PhantomData })

It should be something like:

Symbol("proc_macro2")

Iterating over parsed tokens causes char boundary index panic

extern crate proc_macro2;

fn main() {
    let s = std::str::from_utf8(b"b\'\xc2\x86  \x00\x00\x00^\"").unwrap();
    if let Ok(token_stream) = s.parse::<proc_macro2::TokenStream>() {
        for _ in token_stream { }
    }
}

> cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/proc-macro2-foo`
thread 'main' panicked at 'byte index 1 is not a char boundary; it is inside '\u{86}' (bytes 0..2) of `�  ^"`', src/libcore/str/mod.rs:2234:4
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic
   6: std::panicking::begin_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::str::slice_error_fail
  10: core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::RangeFrom<usize>>::index::{{closure}}
             at /Users/travis/build/rust-lang/rust/src/libcore/str/mod.rs:1987
  11: <core::option::Option<T>>::unwrap_or_else
             at /Users/travis/build/rust-lang/rust/src/libcore/option.rs:376
  12: core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::RangeFrom<usize>>::index
             at /Users/travis/build/rust-lang/rust/src/libcore/str/mod.rs:1987
  13: core::str::traits::<impl core::ops::index::Index<core::ops::range::RangeFrom<usize>> for str>::index
             at /Users/travis/build/rust-lang/rust/src/libcore/str/mod.rs:1734
  14: proc_macro2::strnom::Cursor::advance
             at /Users/corey/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.1/src/strnom.rs:20
  15: proc_macro2::imp::cooked_byte
             at /Users/corey/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.1/src/stable.rs:907
  16: proc_macro2::imp::byte
             at /Users/corey/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.1/src/strnom.rs:160
  17: proc_macro2::imp::literal_nocapture
             at /Users/corey/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.1/src/strnom.rs:160
  18: proc_macro2::imp::literal
             at /Users/corey/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.1/src/stable.rs:697
  19: proc_macro2::imp::token_kind
             at /Users/corey/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.1/src/strnom.rs:160
  20: proc_macro2::imp::token_tree
             at /Users/corey/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.1/src/stable.rs:596
  21: proc_macro2::imp::token_stream
             at /Users/corey/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.1/src/strnom.rs:160
  22: proc_macro2::imp::get_cursor
             at /Users/corey/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.1/src/stable.rs:66
  23: <str as core::str::StrExt>::parse
             at /Users/travis/build/rust-lang/rust/src/libcore/str/mod.rs:2511
  24: alloc::str::<impl str>::parse
             at /Users/travis/build/rust-lang/rust/src/liballoc/str.rs:1793
  25: alloc::slice::<impl [T]>::as_mut_ptr
             at /Users/corey/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.1/src/lib.rs:59
  26: <str as core::str::StrExt>::parse
             at /Users/travis/build/rust-lang/rust/src/libcore/str/mod.rs:2511
  27: alloc::str::<impl str>::parse
             at /Users/travis/build/rust-lang/rust/src/liballoc/str.rs:1793
  28: proc_macro2_foo::main
             at src/main.rs:5
  29: __rust_maybe_catch_panic
  30: std::rt::lang_start
  31: proc_macro2_foo::main

found via afl.rs

Invert the "stable" feature into an "unstable" feature

The current cfg situation is:

#[path = "stable.rs"]
#[cfg(feature = "stable")]
mod imp;
#[path = "unstable.rs"]
#[cfg(not(feature = "stable"))]
mod imp;

This is annoying because it requires every library that transitively depends on proc-macro2 to provide its own "stable" cfg that mirrors proc-macro2's "stable" cfg. If I want to use nightly with nice errors in my app, I need to track down the "stable" cfg of every single library in my dependency graph that uses proc-macro2, and disable the "stable" cfg of each one individually.

Inverting this to an "unstable" feature means that libraries using proc-macro2 do not need a cfg at all. The end user can opt into nightly by adding their own dependency on proc-macro2 that sets the unstable cfg.

Distributing binary built with `procmacro2_semver_exempt`

I've got a project that's using syn 0.3.8 with procmacro2_semver_exempt. I tried doing cargo install from the git repo/branch and it attempted to build without the rustflags in .cargo/config so the build failed. Obviously, this behaviour isn't desired as a user shouldn't have to specify RUSTFLAGS before running cargo install so I'm wondering if this is still the case when installing from crates.io and if there is any advice for more easily distributing my binary?

Convert Span to proc_macro::Span

Hi.
It would be useful to be able to convert a proc_macro2::Span to a proc_macro::Span.
For instance, this will allow to create Diagnostics with Span.
Thank you to add a From implementation for unstable only or something along these lines.

Rust nightly fails

At least 2017-11-11:

error[E0599]: no function or associated item named `def_site` found for type `proc_macro::Span` in the current scope
   --> /Users/daniel/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.1.5/src/unstable.rs:171:14
    |
171 |         Span(proc_macro::Span::def_site())
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^

Build fails on `-musl` targets because of `proc_macro`

Hi there,
I'm cross-building (OpenWrt with musl, nightly from 2018-01-09) a package that indirectly depends on this crate. Unfortunately the build fails due to the crate proc_macro missing. AFAIK this is due to a Rust compiler gimmick but I'm not sure how to approach this problem. I've got a couple questions:

• Do you happen to know the root cause of this issue?
• Is there a solution out there? If so, what is it?
• If not, what can I do to help resolve it?

I'd really appreciate your thoughts on this.

Thank you,
Stanisław

Should fail to parse space before lifetime name

' static is successfully parsed as a lifetime.

/**/ should be insignificant whitespace

Input	Expected	proc-macro2
/* */	whitespace	correct
/** */	attribute	correct
/***/	whitespace	correct
/**/	whitespace	INCORRECT

extern crate proc_macro2;
use proc_macro2::TokenStream;

fn main() {
    for tt in "/**/".parse::<TokenStream>().unwrap() {
        println!("{:?}", tt.kind);
    }
}

Literal(Literal("/**/"))

Figure out how to automatically use proc_macro once stable

When the new proc_macro API is eventually stabilized, we may want proc-macro2 to automatically use it if building with a new enough compiler. This saves compile time of compiling our own lexer where possible, while continuing to support older compiler versions as well. In particular, Serde will want a proc-macro2 version that works on Rust 1.15 but forwards to the stable libproc_macro on sufficiently new compilers.

Numeric suffixes need to check for word boundary

This is not correct:

"1u80".parse::<TokenStream>().unwrap()

TokenStream(
    TokenStream {
        inner: [
            TokenTree {
                span: Span(
                    Span
                ),
                kind: Literal(
                    Literal(
                        Literal(
                            "1u8"
                        )
                    )
                )
            },
            TokenTree {
                span: Span(
                    Span
                ),
                kind: Literal(
                    Literal(
                        Literal(
                            "0"
                        )
                    )
                )
            }
        ]
    }
)

Implement `Eq` for `Span`

The real proc_macro::Span has Eq and PartialEq implemented for it, while our Span does not.

Link to proc_macro documentation from the crate-level docs

The documentation mentions proc_macro a few times but never links to it. The API in proc_macro has some useful things that are not reflected in proc-macro2, e.g. diagnostics.

https://doc.rust-lang.org/proc_macro/

Remove impl Default for Span

This was removed upstream in rust-lang/rust#45999. As discussed in rust-lang/rust#45934 (comment), there isn't really anything more "default" about def_site over call_site.

Worth changing before the syn 0.12 release before more people start banging on this stuff?

Should not be unit structs

The current stable imp contains:

pub struct LexError;
pub struct Span;

These should be structs with a private field to match the ones in proc_macro. That way users can't write code that works with stable but not unstable.

Don't check `digits` for overflow when parsing in `stable`

We don't ever need the actual u64 value, and this will allow us to parse u128 on stable rust.

Trim down and reduce compile times

Right now proc-macro2 stable makes up a decent amount of the build time for syn-master.

It shouldn't be too hard to strip down its dependencies and make the parsing code compile a bit faster, especially once #4 lands.

Switch from `IResult<I, O>` to `Result<(I, O), ...>`

Just like dtolnay/syn#151.

Wishlist

Things we want in proc-macro that are not in rust-lang/rust#40939:

• Debug impls for the public types. Because debugging this stuff is hard enough even when you understand what the input contains.

Unstable feature broken - pm::Span no longer Default

rust-lang/rust@74cc1fd

error[E0277]: the trait bound `proc_macro::Span: std::default::Default` is not satisfied
   --> C:\Users\Arnavion\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro2-0.1.3\src\unstable.rs:163:17
    |
163 | pub struct Span(proc_macro::Span);
    |                 ^^^^^^^^^^^^^^^^^ the trait `std::default::Default` is not implemented for `proc_macro::Span`
    |
    = note: required by `std::default::Default::default`

--- src/unstable.rs.orig        2017-11-19 01:22:18.446739600 -0800
+++ src/unstable.rs     2017-11-19 01:16:34.641581800 -0800
@@ -159,9 +159,15 @@ impl fmt::Debug for TokenTreeIter {
     }
 }

-#[derive(Copy, Clone, Default)]
+#[derive(Copy, Clone)]
 pub struct Span(proc_macro::Span);

+impl Default for Span {
+    fn default() -> Self {
+        Span(proc_macro::Span::def_site())
+    }
+}
+
 impl Span {
     pub fn call_site() -> Span {
         Span(proc_macro::Span::call_site())

Span::def_site is missing :(

@dtolnay

https://docs.rs/proc-macro2/0.3.6/proc_macro2/struct.Span.html

the futures await crate used it in version 0.2, and in 0.3 it's gone 😞

Way to get the type of literal

Hi.
Currently, the structure of Literal is private.
Could you please add a way to inspect the literals?
Thank you.

malformed literal returned: `Literal("///!{")`

extern crate proc_macro2;
use proc_macro2::TokenStream;

fn main() {
    for tt in "dc0<///!{\x0a(continue|||||self||||)".parse::<TokenStream>().unwrap() {
        println!("{:?}", tt.kind);
    }
}

$ rustc -vV
rustc 1.23.0 (766bd11c8 2018-01-01)
binary: rustc
commit-hash: 766bd11c8a3c019ca53febdcd77b2215379dd67d
commit-date: 2018-01-01
host: x86_64-unknown-linux-gnu
release: 1.23.0
LLVM version: 4.0
$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/test-proc-macro`
Term(Term("dc0"))
Op('<', Joint)
Literal(Literal("///!{"))
Group(Parenthesis, TokenStream { inner: [TokenTree { span: Span, kind: Term(Term("continue")) }, TokenTree { span: Span, kind: Op('|', Joint) }, TokenTree { span: Span, kind: Op('|', Joint) }, TokenTree { span: Span, kind: Op('|', Joint) }, TokenTree { span: Span, kind: Op('|', Joint) }, TokenTree { span: Span, kind: Op('|', Alone) }, TokenTree { span: Span, kind: Term(Term("self")) }, TokenTree { span: Span, kind: Op('|', Joint) }, TokenTree { span: Span, kind: Op('|', Joint) }, TokenTree { span: Span, kind: Op('|', Joint) }, TokenTree { span: Span, kind: Op('|', Alone) }] })

syn expects this to be impossible, and panics: https://github.com/dtolnay/syn/blob/139efc6f42299d16cae11c5789a692f868e19024/src/lit.rs#L550

Symbol should implement neither Send nor Sync

It is an index into a thread local vector.

Upstream issue: rust-lang/rust#42407

Doc comment without newline does not match libproc_macro

for tt in "/// doc".parse::<TokenStream>().unwrap() {
    println!("{:?}", tt.kind);
}

Proc-macro sees this as a doc comment despite not ending with a newline.

Literal(Literal(DocComment(/// doc)))

Proc-macro2 sees it as a / joint op followed by insignificant whitespace // doc.

Op('/', Joint)

Consider moving all APIs not in proc_macro under the `procmacro2_semver_exempt` cfg option

We have some APIs which are exposed from proc-macro2, but aren't exposed from proc_macro itself. It might be good to move these APIs behind the flag which is being added in #36

IntoIterator for &TokenStream

I found when using TokenStream I couldn't iterate over a &TokenStream because IntoIterator wasn't implemented for it. Is there a reason this isn't included? If not I can submit a PR for this and any similar things in the same area with the same issue.

Enabling the `nightly` feature can be a breaking change

I've seen a number of crates now where when the nightly feature is enabled the crate's compilation breaks due to procedural macros. This typically happens around resolution in macro expansion and has to do primarily with quote! I believe.

If the nightly feature is disabled then everything is unhygienic and I think works with the equivalent of Span::call_site(), meaning that all the tokens produced by quote! ended up morally being used with Span::call_site(). When nightly is enabled, however, the quote! macro is actually under the hood using Span::def_site() everywhere (it was basically just ignored without the nightly feature).

tl;dr; tokens produced by quote! use Span::call_site() when nightly is not enabled, and Span::def_site() when nightly is enabled

@dtolnay do you have thoughts on this? Do you think we should, while Span is unstable, move the quote crate to using Span::call_site() by default?

Enabling "nightly" feature breaks processing of mod in another file

Situation

my_mod is a module with body in file my_mod.rs. To include it in the project it must be declared in another file, let's say lib.rs. I'm trying to push whole my_mod into proc macro my_macro, so I write in lib.rs:

#[my_macro]
mod my_mod;

Without "nightly" feature

Whole content of my_mod is passed into macro.

With "nightly" feature

Only mod my_mod; is passed into macro.

Question

Is it regression or is it the way proc macros 2.0 are supposed to work?

Any breaking changes lined up?

I am getting ready to cut a Syn major version later this week with proc-macro2 in its public API. If we are planning to do any breaking changes in the near term, this would be the best time.

Rustdoc needed

We either need to copy over documentation from libproc_macro or provide a link from each of our types to the corresponding type in https://doc.rust-lang.org/nightly/proc_macro so that people find this documentation.

Parse underscore as Term

Nightly currently treats _ as:

TokenStream [Term { sym: _, span: #0 bytes(75..76) }]

whereas in proc-macro2 it is:

TokenStream [Op { op: \'_\', spacing: Alone }]

Be smarter about string escaping

Something like

proc_macro2::Literal::string(&format!("Couldn't find {}", class_name));

will end up as "Couldn\'t find SomeProtocol" when used in quote!
Since we're in a string surrounded by double quotes, the escaping there is useless.

Show semver exempt APIs on docs.rs

I believe something not entirely unlike the following may do the trick but let's keep this issue open until we confirm everything we expect is visible in http://docs.rs/proc-macro2.

[package.metadata.docs.rs]
rustdoc-args = ["--cfg procmacro2_semver_exempt"]

Requires #81.

Consider providing meaningful Spans in stable implementation

It looks like cpp_syn uses low and high indices into the input string like so:

pub struct Span {
    pub lo: usize,
    pub hi: usize,
}

@mystor if we are able to provide the same thing here, would that eliminate the need for cpp_syn?

Fails to lex underscore in unicode escapes

This test case from the compiler test suite results in a lex error:

https://github.com/rust-lang/rust/blob/1.22.1/src/test/run-pass/issue-43692.rs

This was implemented upstream in rust-lang/rust#43716 and stabilized in Rust 1.22.

Version 0.2.3 breaks the futures await crate

proc_macro2 version v0.2.3 as used in the futures-await crate
the errors are coming from this crate on the most recent nightly compiler. (rustc 1.27.0-nightly (4b9b70c39 2018-04-09))

currently alexcrichton/futures-await#86 is blocked because of this.

   Compiling proc-macro2 v0.2.3
error[E0433]: failed to resolve. Could not find `TokenNode` in `proc_macro`
  --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:63:33
   |
63 |                     proc_macro::TokenNode::Group(delim, (s.0).0)
   |                                 ^^^^^^^^^ Could not find `TokenNode` in `proc_macro`

error[E0433]: failed to resolve. Could not find `TokenNode` in `proc_macro`
  --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:70:33
   |
70 |                     proc_macro::TokenNode::Op(ch, kind)
   |                                 ^^^^^^^^^ Could not find `TokenNode` in `proc_macro`

error[E0433]: failed to resolve. Could not find `TokenNode` in `proc_macro`
  --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:73:33
   |
73 |                     proc_macro::TokenNode::Term((s.0).0)
   |                                 ^^^^^^^^^ Could not find `TokenNode` in `proc_macro`

error[E0433]: failed to resolve. Could not find `TokenNode` in `proc_macro`
  --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:76:33
   |
76 |                     proc_macro::TokenNode::Literal((l.0).0)
   |                                 ^^^^^^^^^ Could not find `TokenNode` in `proc_macro`

error[E0433]: failed to resolve. Could not find `TokenNode` in `proc_macro`
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:124:29
    |
124 |                 proc_macro::TokenNode::Group(delim, s) => {
    |                             ^^^^^^^^^ Could not find `TokenNode` in `proc_macro`

error[E0433]: failed to resolve. Could not find `TokenNode` in `proc_macro`
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:133:29
    |
133 |                 proc_macro::TokenNode::Op(ch, kind) => {
    |                             ^^^^^^^^^ Could not find `TokenNode` in `proc_macro`

error[E0433]: failed to resolve. Could not find `TokenNode` in `proc_macro`
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:140:29
    |
140 |                 proc_macro::TokenNode::Term(s) => {
    |                             ^^^^^^^^^ Could not find `TokenNode` in `proc_macro`

error[E0433]: failed to resolve. Could not find `TokenNode` in `proc_macro`
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:143:29
    |
143 |                 proc_macro::TokenNode::Literal(l) => {
    |                             ^^^^^^^^^ Could not find `TokenNode` in `proc_macro`

error[E0433]: failed to resolve. Could not find `TokenNode` in `proc_macro`
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:362:21
    |
362 |         proc_macro::TokenNode::Literal(l) => l,
    |                     ^^^^^^^^^ Could not find `TokenNode` in `proc_macro`

error[E0574]: expected struct, variant or union type, found enum `proc_macro::TokenTree`
  --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:53:21
   |
53 |         TokenStream(proc_macro::TokenTree {
   |                     ^^^^^^^^^^^^^^^^^^^^^ not a struct, variant or union type
help: possible better candidate is found in another module, you can import it into scope
   |
1  | use TokenTree;
   |

error[E0412]: cannot find type `TokenTreeIter` in module `proc_macro`
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:102:38
    |
102 | pub struct TokenTreeIter(proc_macro::TokenTreeIter);
    |                                      ^^^^^^^^^^^^^ did you mean `TokenTree`?
help: possible candidates are found in other modules, you can import them into scope
    |
1   | use TokenTreeIter;
    |
1   | use imp::TokenTreeIter;
    |

error[E0277]: the trait bound `proc_macro::TokenStream: std::iter::FromIterator<proc_macro::TokenStream>` is not satisfied
  --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:86:29
   |
86 |         TokenStream(streams.collect::<proc_macro::TokenStream>())
   |                             ^^^^^^^ a collection of type `proc_macro::TokenStream` cannot be built from an iterator over elements of type `proc_macro::TokenStream`
   |
   = help: the trait `std::iter::FromIterator<proc_macro::TokenStream>` is not implemented for `proc_macro::TokenStream`

error[E0599]: no function or associated item named `intern` found for type `proc_macro::Term` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:279:14
    |
279 |         Term(proc_macro::Term::intern(string))
    |              ^^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Term`

error[E0599]: no function or associated item named `float` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:319:17
    |
319 |         Literal(proc_macro::Literal::float(s))
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`

error[E0599]: no function or associated item named `integer` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:323:17
    |
323 |         Literal(proc_macro::Literal::integer(s.into()))
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`

error[E0609]: no field `kind` on type `proc_macro::TokenTree`
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:361:46
    |
361 |     match stream.into_iter().next().unwrap().kind {
    |                                              ^^^^

error[E0599]: no function or associated item named `u8` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:371:25
    |
371 |                   Literal(proc_macro::Literal::$t(t))
    |                           ^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`
...
377 | / ints! {
378 | |     u8, u16, u32, u64, usize,
379 | |     i8, i16, i32, i64, isize,
380 | | }
    | |_- in this macro invocation

error[E0599]: no function or associated item named `u16` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:371:25
    |
371 |                   Literal(proc_macro::Literal::$t(t))
    |                           ^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`
...
377 | / ints! {
378 | |     u8, u16, u32, u64, usize,
379 | |     i8, i16, i32, i64, isize,
380 | | }
    | |_- in this macro invocation

error[E0599]: no function or associated item named `u32` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:371:25
    |
371 |                   Literal(proc_macro::Literal::$t(t))
    |                           ^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`
...
377 | / ints! {
378 | |     u8, u16, u32, u64, usize,
379 | |     i8, i16, i32, i64, isize,
380 | | }
    | |_- in this macro invocation

error[E0599]: no function or associated item named `u64` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:371:25
    |
371 |                   Literal(proc_macro::Literal::$t(t))
    |                           ^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`
...
377 | / ints! {
378 | |     u8, u16, u32, u64, usize,
379 | |     i8, i16, i32, i64, isize,
380 | | }
    | |_- in this macro invocation

error[E0599]: no function or associated item named `usize` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:371:25
    |
371 |                   Literal(proc_macro::Literal::$t(t))
    |                           ^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`
...
377 | / ints! {
378 | |     u8, u16, u32, u64, usize,
379 | |     i8, i16, i32, i64, isize,
380 | | }
    | |_- in this macro invocation

error[E0599]: no function or associated item named `i8` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:371:25
    |
371 |                   Literal(proc_macro::Literal::$t(t))
    |                           ^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`
...
377 | / ints! {
378 | |     u8, u16, u32, u64, usize,
379 | |     i8, i16, i32, i64, isize,
380 | | }
    | |_- in this macro invocation

error[E0599]: no function or associated item named `i16` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:371:25
    |
371 |                   Literal(proc_macro::Literal::$t(t))
    |                           ^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`
...
377 | / ints! {
378 | |     u8, u16, u32, u64, usize,
379 | |     i8, i16, i32, i64, isize,
380 | | }
    | |_- in this macro invocation

error[E0599]: no function or associated item named `i32` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:371:25
    |
371 |                   Literal(proc_macro::Literal::$t(t))
    |                           ^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`
...
377 | / ints! {
378 | |     u8, u16, u32, u64, usize,
379 | |     i8, i16, i32, i64, isize,
380 | | }
    | |_- in this macro invocation

error[E0599]: no function or associated item named `i64` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:371:25
    |
371 |                   Literal(proc_macro::Literal::$t(t))
    |                           ^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`
...
377 | / ints! {
378 | |     u8, u16, u32, u64, usize,
379 | |     i8, i16, i32, i64, isize,
380 | | }
    | |_- in this macro invocation

error[E0599]: no function or associated item named `isize` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:371:25
    |
371 |                   Literal(proc_macro::Literal::$t(t))
    |                           ^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`
...
377 | / ints! {
378 | |     u8, u16, u32, u64, usize,
379 | |     i8, i16, i32, i64, isize,
380 | | }
    | |_- in this macro invocation

error[E0599]: no function or associated item named `f32` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:386:25
    |
386 |                   Literal(proc_macro::Literal::$t(t))
    |                           ^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`
...
392 | / floats! {
393 | |     f32, f64,
394 | | }
    | |_- in this macro invocation

error[E0599]: no function or associated item named `f64` found for type `proc_macro::Literal` in the current scope
   --> /home/seun/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.3/src/unstable.rs:386:25
    |
386 |                   Literal(proc_macro::Literal::$t(t))
    |                           ^^^^^^^^^^^^^^^^^^^^^^^ function or associated item not found in `proc_macro::Literal`
...
392 | / floats! {
393 | |     f32, f64,
394 | | }
    | |_- in this macro invocation

error: aborting due to 28 previous errors

Some errors occurred: E0277, E0412, E0433, E0574, E0599, E0609.
For more information about an error, try `rustc --explain E0277`.
error: Could not compile `proc-macro2`.
warning: build failed, waiting for other jobs to finish...
error: build failed

Custom derive could not find trait when build with nightly feature

I have the following custom derive implemented for Component trait. When I used it without nightly feature it works. But with it, it errors out with cannot find trait Component in this scope. I would like to use nightly feature for handling and showing errors for #[proc_macro]. Is this bug or intended? Or am I doing something wrong?

pub fn derive_component(input: TokenStream) -> TokenStream {
    let ast: syn::DeriveInput = syn::parse(input).unwrap();
    let tokens = impl_component(&ast);
    tokens.into()
}

fn impl_component(ast: &syn::DeriveInput) -> quote::Tokens {
    let name = ast.ident;
    quote! {
        impl Component for #name {
            fn component(&self) -> &str {
                "Hello World"
            }
        }
    }
}

Crate almost compiles... (OSX box, )

Seems to have stopped compiling with the last couple of weeks nighties on OSX:

error[E0277]: the trait bound proc_macro::Span: std::default::Default is not satisfied
--> /Users/username/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.1.3/src/unstable.rs:163:17
|
163 | pub struct Span(proc_macro::Span);
| ^^^^^^^^^^^^^^^^^ the trait std::default::Default is not implemented for proc_macro::Span
|
= note: required by std::default::Default::default

Any suggestions to get it working again?

Should reject keyword as lifetime name

Currently accepts 'mut as a lifetime.

Do not parse doc comment as Literal

As of 0.3.2:

extern crate proc_macro2;
use proc_macro2::TokenStream;

fn main() {
    println!("{:?}", "/// doc\n".parse::<TokenStream>().unwrap());
}

TokenStream { inner: [Literal(Literal { text: "/// doc", span: Span })] }

This should be the tokens # [ doc = " doc" ].

attempting to tokenize lots of open parens results in a stack overflow

https://gist.github.com/frewsxcv/3b427948da01685c95c26ab4a005c337

> cargo run
   Compiling proc-macro2-foo v0.1.0 (file:///private/tmp/proc-macro2-foo)
    Finished dev [unoptimized + debuginfo] target(s) in 2.4 secs
     Running `target/debug/proc-macro2-foo`

thread 'main' has overflowed its stack
fatal runtime error: stack overflow
fish: 'cargo run' terminated by signal SIGABRT (Abort)

found via afl.rs

quote panics when proc-macro2 uses nightly feature

I have encountered this problem after I enabled the nightly feature of proc-macro2. There is no issue when the nightly feature is disabled.

I am testing it with the recent nightly toolchain, but the problem goes back to nightly-2018-04-06 at the very least.

See also: rust-lang/rust#50184

A simple usage of quote!() causes:

thread 'main' panicked at 'proc_macro::__internal::with_sess() called before set_parse_sess()!', libproc_macro/lib.rs:1319:9
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:205
   3: std::panicking::default_hook
             at libstd/panicking.rs:221
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:457
   5: std::panicking::begin_panic
   6: proc_macro::__internal::with_sess
   7: proc_macro2::imp::Span::call_site
             at /home/bigos/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.3.7/src/unstable.rs:214
   8: proc_macro2::Span::call_site
             at /home/bigos/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.3.7/src/lib.rs:186
   9: quote_test::main
             at src/main.rs:5
  10: std::rt::lang_start::{{closure}}
             at /checkout/src/libstd/rt.rs:74
  11: std::panicking::try::do_call
             at libstd/rt.rs:59
             at libstd/panicking.rs:304
  12: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:105
  13: std::rt::lang_start_internal
             at libstd/panicking.rs:283
             at libstd/panic.rs:361
             at libstd/rt.rs:58
  14: std::rt::lang_start
             at /checkout/src/libstd/rt.rs:74
  15: main
  16: __libc_start_main
  17: _start

You can reproduce it with:

Cargo.toml

[package]
name = "quote-test"
version = "0.1.0"

[dependencies]
proc-macro2 = { version = "0.3.1", features = ["nightly"] }
quote = "0.5.1"

src/main.rs

#[macro_use]
extern crate quote;

fn main() {
    quote!();
}

Op followed by comment should not be considered joint

for tt in "~// comment".parse::<TokenStream>().unwrap() {
    println!("{:?}", tt);
}

Proc-macro:

Op(Op { op: '~', spacing: Alone, span: Span(...) })

Proc-macro2:

Op(Op { op: '~', spacing: Joint, span: Span })

The comment should be considered whitespace, making the preceding op alone rather than joint.

#[proc_macro] with proc-macro2

I'm trying to make one library to compile with proc-macro2 in order to use it without nightly rust, but I can't find replacement for #[proc_macro] attribute. Is there any? Or how am I supposed to do it?

Thanks a lot in advance!

Rename either --cfg procmacro2_unstable or --features proc-macro2/unstable

These are basically orthogonal flags so it is confusing to give them such similar names. Someone may want to use procmacro2_unstable without proc-macro2/unstable to get the emulation of line and column from #36 on the stable compiler, or may want to use proc-macro2/unstable without procmacro2_unstable to get real spans without the semver-exempt APIs.

Punctuation not always recognized as joint

It looks like punctuation is only recognized as joint if it begins a sequence of punctuation to which Rust assigns meaning.

This is not my interpretation of rust-lang/rust#40939 and if it is intentional, I would object to that behavior. A punctuation should be joint if it is followed immediately by other punctuation.

"++".parse::<TokenStream>().unwrap()

TokenStream(
    TokenStream {
        inner: [
            TokenTree {
                span: Span(
                    Span
                ),
                kind: Op(
                    '+',
                    Alone
                )
            },
            TokenTree {
                span: Span(
                    Span
                ),
                kind: Op(
                    '+',
                    Alone
                )
            }
        ]
    }
)

Get the span of an Expr

Hi.
Is there an easy way to get the Span of an Expr?
That is without having to use pattern matching on node and getting the span manually.
Thank you.

Document which APIs are semver exempt in rustdoc

The output of cargo rustdoc -- --cfg procmacro2_semver_exempt currently gives no indication which types and functions are semver exempt. This would be a prerequisite for displaying semver exempt APIs on docs.rs.