eupn/macrotest

Test harness for declarative and procedural macros expansion via `cargo-expand`

macrotest's Issues

Use the same edition as the crate under test

Copy the edition field of package section of the crate's Cargo.toml into the "temporary crate"'s manifest. It should be reasonable to assume that the test cases are using the same edition as the crate which macros they're testing.

Typo in test failed message

I noticed this in part of the output when a test fails:

thread 'pass' panicked at '1 or 4 tests failed', /git/tmp/macrotest/src/expand.rs:58:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

It seems like macrotest should know whether it was 1 or 4 tests that failed!

Probably this should be "1 of 4 tests failed".

Wrong lines stripped if an input contains module-level attribute

Macrotest assumes that the output of cargo expand always begins with the following exactly 5 lines.

macrotest/src/expand.rs

Lines 390 to 391 in 779cfa5

	// `cargo expand` does always produce some fixed amount of lines that should be ignored
	const CARGO_EXPAND_SKIP_LINES_COUNT: usize = 5;

#![feature(prelude_import)]
#[prelude_import]
use std::prelude::$edition::*;
#[macro_use]
extern crate std;

However, that is not the case if there is an inner attribute in the input. For example the following crate:

#![feature(proc_macro_span)]

fn foo() {}

expands as:

#![feature(prelude_import)]
#![feature(proc_macro_span)]
#[prelude_import]
use std::prelude::rust_2018::*;
#[macro_use]
extern crate std;
fn foo() {}

and after stripping exactly 5 lines from the front, macrotest's .expanded.rs will result in:

extern crate std;
fn foo() {}

which is not what is expected. The correct expansion would be:

#![feature(proc_macro_span)]
fn foo() {}

Allow the passing of flags to expand

I would like the possibility of passing flags to cargo in macrotest::expand_without_refresh.

My use case is that I would like to test my expansion with different crate features. It seems to me that macrotest::expand_without_refresh simply calls cargo expand, which means the default features of the crate under test are used.
I would like to be able to disable the default features and enable other features in a granular fashion:

// Test expansion without any features
macrotest::expand_without_refresh("some_test_file.rs", "--no-default-features"); 

// Test expansion with only feature "some_feature" enabled
macrotest::expand_without_refresh("some_test_file.rs", "--features some_feature"); 

Of course the exact API doesn't have to look like the above.

Wrong report when macro expansion fails

For all macrotest::expand variants, if the macro being expanded panics, the report gives wrong or confusing errors.

First, if you have no .expanded file and not using expand_without_refresh, the call will succeed (even though the expanded macro panicked) and create an empty .expanded file. If you are using expand_without_refresh the report will correctly show an error however will give no indication about what happened.

If you do have an .expanded file the report will give a <file-path> - different! error, and show the contents of your expanded file but nothing else.

For all of these cases, I would expect the error message to tell me that the macro expansion failed and maybe show me the error message it failed with.

Note: I found these cases while testing #66, however, these problems are also present in v1.0.8.

git tag missing for 1.0.9

I wanted to investigate an issue, but found that I although was using 1.0.9 from crates.io but there wasn't a corresponding git tag.

Normally, cargo puts a .cargo_vcs_info.json in the generated .crate file, if the release is made from git. Investigating the .crate file I think the release was made roughly from 3751785. The only difference (apart from changes made to Cargo.toml by cargo publish) is a one-line change to .gitignore.

Was 1.0.9 made from a dirty git working tree? I recommend the use of automated tooling for releasing which checks things like "is the git tree clean" and also makes the tags.

To remedy this, I suggest: make a temporary git branch containing this delta

diff --git a/.gitignore b/.gitignore
index ff23b2d..9345978 100644
--- a/.gitignore
+++ b/.gitignore
@@ -4,3 +4,4 @@ Cargo.lock
 *.iml
 *.ipr
 *.iws
+.idea

and tag it 1.0.9

Common, error

What is this stuff? Are users supposed to use it? It wasn't clear from the docs what role they play, since the examples only use TestCases.

Test passes when there are errors in cargo-expand output

I am not sure if this is a cargo-expand issue or intentional

If a syn::Type has generics and is used in an expression like #ty::#ident, cargo-expand outputs "(/ERROR/)"

use quote::quote;
let ty = syn::parse_quote! { Box<Foo> };
// put this somewhere in the expand output
quote! { #ty::some_thing }

The output will be:

(/*ERROR*/)

The catch is the syntax should be quote! { <#ty>::some_thing }. However, macrotest will pass.

Ideally, there are some places in other tests/examples that use the macro and will catch the issue, but would be nice if macrotest fails.

Requirement to precisely pin the version of cargo-expand

macrotest depends fundamentally on the stability of the output of cargo expand. But this is not something that cargo expand's maintainers promise: dtolnay/cargo-expand#179.

If one is using macrotest in CI, as one should, the CI will occasionally need to rebuild cargo expand. If it gets a new version, the tests break.

The solution is to use something like

cargo install --locked --version 1.0.44 --features=prettyplease cargo-expand

I think this should be documented.

v1.0.9 violates MSRV

Version 1.0.9 of macrotest introduced the use of prettyplease crate. See #78.

Problem is, all versions of prettyplease use edition 2021, meaning the earliest rustc version that will work now is 1.56 (the version introducing edition 2021).
Since this crate officially has an MSRV of 1.34, version 1.0.9 is violating it. In my opinion v1.0.9 should be yanked from crates.io.
There should probably also be some testing happening for whether the MSRV is upheld so this doesn't happen again.

macrotest and trybuild together keep doing rebuilds

Steps to reproduce:

(install nightly Rust and cargo-expand)
git clone https://github.com/taiki-e/pin-project
cd pin-project
cargo +nightly --locked --offline test --workspace -- ui # 1
cargo +nightly --locked --offline test --workspace -- ui # 2
cargo +nightly --locked --offline test --workspace -- expandtest # 3
cargo +nightly --locked --offline test --workspace -- expandtest # 4
cargo +nightly --locked --offline test --workspace -- ui # 5
cargo +nightly --locked --offline test --workspace -- expandtest #6

Expected behaviour: In steps 1 and 3, some actual compilation work takes place. In steps 2,4,5,6, nothing much is compiled and the tests run quickly.

Actual behaviour: In steps 5 and 6, many things are rebuilt including proc-macro2, syn, ...

I don't know what is going on here, and the root cause is almost certainly a bug in cargo. But I think it would be good if both macrotest and trybuild could work around it. Probably, the best workaround is for them each to use a separate target directory. That would involve rebuilding extra things on the first iteration, but thereeafter the previously built things would be reused.

This happened to me with an unpublished version of a package of mine. When investigating I thought this seemed to be a bug elsewhere, so I looked through the rdependencies of macrotest for a package which used both. pin-project was the obvious candidate, and indeed it is affected. I looked at the test driver code in pin-project and there doesn't seem to be anything unexpected there that might cause this problem.

Since github doesn't support a single issue affecting multiple projects, and anyway I think probably it would be nice if both packages were changed to incorporate a workaround (even though I think one package doing so would suffice), I am filing an identical ticket against trybuild too.

Mention that rustfmt is required for proper work

At least for examples in test-*-projects.

MSRV violation

macrotest no longer compiles on rust v1.34.2 because serde_json v1.0.73 depends on itoa v1.0.1 which doesn't compile on that version. See (duplicate build)[https://github.com/Emoun/duplicate/runs/4636793495?check_suite_focus=true].

As far as I can tell serde_json has no MSRV, which means macrotest should depend on a specific version to avoid violating MSRV.

workspaces `dev-dependencies` don't work

I have a workspace where one of the crates uses macrotest. In that crate, I also put a workspace dev-dependency on another crate for other tests via test-sys.workspace = true. This causes macrotest to fail to pass. Other non-macrotest tests pass fine, and removing the workspace dev-dependency causes macrotest tests to pass

output:

---- tests::macro_expansion stdout ----
Running 1 macro expansion tests
Expansion error:
error: failed to parse manifest at `D:\test-ws\target\tests\test-package\macrotest002\Cargo.toml`
Caused by:
  error inheriting `test-sys` from workspace root manifest's `workspace.dependencies.test-sys`
Caused by:
  `workspace.dependencies` was not defined

macrotest version: latest on main c4151a5

Cut dependencies

My first reaction when trying out this crate was wow that's a lot of dependencies for a simple crate.

Would it be possible to cut this down to the minimum required to implement the functionality? Failure, derive_more, and tempdir would be good candidates to cut.

macrotest v0.1.1 (/git/tmp/macrotest)
├── derive_more v0.15.0
│   ├── lazy_static v1.4.0
│   ├── proc-macro2 v0.4.30
│   │   └── unicode-xid v0.1.0
│   ├── quote v0.6.13
│   │   └── proc-macro2 v0.4.30 (*)
│   ├── regex v1.3.1
│   │   ├── aho-corasick v0.7.6
│   │   │   └── memchr v2.2.1
│   │   ├── memchr v2.2.1 (*)
│   │   ├── regex-syntax v0.6.12
│   │   └── thread_local v0.3.6
│   │       └── lazy_static v1.4.0 (*)
│   └── syn v0.15.44
│       ├── proc-macro2 v0.4.30 (*)
│       ├── quote v0.6.13 (*)
│       └── unicode-xid v0.1.0 (*)
│   [build-dependencies]
│   └── rustc_version v0.2.3
│       └── semver v0.9.0
│           └── semver-parser v0.7.0
├── difference v2.0.0
├── failure v0.1.6
│   ├── backtrace v0.3.40
│   │   ├── backtrace-sys v0.1.32
│   │   │   └── libc v0.2.65
│   │   │   [build-dependencies]
│   │   │   └── cc v1.0.46
│   │   ├── cfg-if v0.1.10
│   │   ├── libc v0.2.65 (*)
│   │   └── rustc-demangle v0.1.16
│   └── failure_derive v0.1.6
│       ├── proc-macro2 v1.0.6
│       │   └── unicode-xid v0.2.0
│       ├── quote v1.0.2
│       │   └── proc-macro2 v1.0.6 (*)
│       ├── syn v1.0.7
│       │   ├── proc-macro2 v1.0.6 (*)
│       │   ├── quote v1.0.2 (*)
│       │   └── unicode-xid v0.2.0 (*)
│       └── synstructure v0.12.1
│           ├── proc-macro2 v1.0.6 (*)
│           ├── quote v1.0.2 (*)
│           ├── syn v1.0.7 (*)
│           └── unicode-xid v0.2.0 (*)
├── glob v0.3.0
├── tempdir v0.3.7
│   ├── rand v0.4.6
│   │   └── libc v0.2.65 (*)
│   └── remove_dir_all v0.5.2
└── toml v0.5.4
    └── serde v1.0.102

Quote comment

I noticed this line in the example project:

macrotest/test-procmacro-project/src/lib.rs

Lines 1 to 2 in 0beeb43

	// You might need to use this in future since quote! macro can be highly-recursive
	//#![recursion_limit="1024"]

I don't think this is accurate anymore since quote isn't recursive.

Feature Request: Expand fail

Hi,

I want to ask whether you would consider implementing what trybuild essentially does, I.e. allowing us to check that a compilation fails with a given error message?

The reason I ask is that I'm in need of the functionality that trybuild provides, however it is missing essential features that macrotest has.
Most essential is the fact that it doesn't seem to panic upon wrong test results, which is a deal-breaker for me.
Also, features that have been implemented in macrotest are also missing (#41,#43).
Additionally, it seems silly to me that I would need to use two different crates for two functions that are so similar.

Essentially what I would need is expand_fail versions of expand and expand_without_refresh, that expects the expansion to fail and then checks the error message is as expected. If the expansion doesn't fail, or the error message is wrong, the test is a failure.

test-project: cargo test shows failures

Context: Investigating the cause of issue #71, in the course of making PR #72.

Using the current master:

cd test-project
cargo test -- --nocapture

Shows some tests failing:

    Finished test [unoptimized + debuginfo] target(s) in 0.02s
     Running unittests (target/debug/deps/test_project-90e6064d5d3cb538)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/par_tests_regression.rs (target/debug/deps/par_tests_regression-ed3d7a792e4170a2)

running 2 tests
Running 1 macro expansion tests
Running 1 macro expansion tests
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest000)
    Blocking waiting for file lock on build directory
    Finished dev [unoptimized + debuginfo] target(s) in 0.18s
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest001)
    Finished dev [unoptimized + debuginfo] target(s) in 0.48s

use std::prelude::rust_2018::*;

use std::prelude::rust_2018::*;
tests/expand/second.rs - ok
test parallel_2 ... ok
tests/expand/first.rs - ok
test parallel_1 ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.78s

     Running tests/tests.rs (target/debug/deps/tests-78123928aa400a33)

running 7 tests
Running 4 macro expansion tests
Running 1 macro expansion tests
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest000)
    Blocking waiting for file lock on build directory
    Finished dev [unoptimized + debuginfo] target(s) in 0.24s
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest001)

use std::prelude::rust_2018::*;
    Finished dev [unoptimized + debuginfo] target(s) in 0.99s

use std::prelude::rust_2018::*;
tests/no_expanded/first.expanded.rs is expected but not found
tests/no_expanded_args/with_args.expanded.rs is expected but not found



thread 'fail_expect_expanded_args' panicked at '1 of 1 tests failed', /home/user/src/macrotest/src/expand.rs:171:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test fail_expect_expanded_args - should panic ... ok
Running 4 macro expansion tests
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest002)
tests/no_expanded/fourth.expanded.rs is expected but not found
    Finished dev [unoptimized + debuginfo] target(s) in 0.55s

use std::prelude::rust_2018::*;
tests/no_expanded/second.expanded.rs is expected but not found
tests/expand/first.rs - ok
tests/no_expanded/third.expanded.rs is expected but not found



thread 'fail_expect_expanded' panicked at '4 of 4 tests failed', /home/user/src/macrotest/src/expand.rs:171:9
test fail_expect_expanded - should panic ... ok
Running 1 macro expansion tests
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest003)
tests/expand/fourth.rs - ok
    Finished dev [unoptimized + debuginfo] target(s) in 1.00s

use std::prelude::rust_2018::*;
tests/expand/second.rs - ok
tests/expand_args/with_args.rs - ok
test pass_args ... ok
Running 4 macro expansion tests
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest004)
tests/expand/third.rs - ok
test pass ... ok
Running 1 macro expansion tests
    Finished dev [unoptimized + debuginfo] target(s) in 0.57s
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest005)

use std::prelude::rust_2018::*;
    Finished dev [unoptimized + debuginfo] target(s) in 0.47s

use std::prelude::rust_2018::*;
tests/expand/first.rs - ok
tests/expand_args/with_args.rs - ok
test pass_expect_expanded_args ... ok
Running 2 macro expansion tests
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest006)
tests/expand/fourth.rs - ok
    Finished dev [unoptimized + debuginfo] target(s) in 0.47s

use std::prelude::rust_2018::*;
tests/expand/second.rs - ok
tests/pr61/a/test.rs - ok
tests/expand/third.rs - ok
test pass_expect_expanded ... ok
tests/pr61/b/test.rs - ok
test pr61 ... ok

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 17.27s

   Doc-tests test-project

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Consider cargo_toml crate for Cargo.toml manipulations

It might be better and less error-prone to use cargo_toml crate for Cargo.toml manipulations. Currently it's done by using definitions from manifest.rs.

Configurable refresh

I would be useful if we were able to configure whether the "refresh" outcome of expand could be turned off or result in a failure.

I specifically have a setup where I'd rather have a missing .expanded.rs file trigger a test failure and not create the expanded file for me. I like that the file could be created for me, but I'd prefer if it was only when I specifically ask for it.

A possible solution is simply adding another method that doesn't create expanded files when they are missing, simply failing instead:

#[test]
pub fn pass() {
    macrotest::expand_without_refresh("tests/expand/*.rs");
}

Support workspace inheritance

I've encountered this issue while implementing a infinyon/fluvio plugin-style integration test harness for Minitest, PR tikv/minitrace-rust/pull/127.

I have now reproduced the issue in a test-virtual project that will, hopefully, land here via PR #72.

The test-virtual example shows how to setup macrotest with a generic (non-libtest) test harness, and allows macrotest users to compare this plugin-style integration test suite with the default Rust libtest test harness.

At the moment the test-virtual project returns this error:

cargo test integration-tests 

<snip>

Expansion error:
error: Package `wrkspc-test-tests v0.0.0 (/home/user/src/macrotest/test-virtual/target/tests/wrkspc-test/macrotest000)` does not have the feature `test-feature`

I've banged my head on both projects but can't seem to figure out where the mismatch is occuring.

Appreciate any help or insights you can share.

Emit newline character at the end of .expanded.rs files

Proper text files should contain a newline character at the end.

Parallel tests fail in workspace project

Thank you for the effort you have put into making macrotest, and for making it open source.

I ran into this issue in the course of trying to isolate what appears to be unexpected behavior.

PR #72 proposes to add a workspace project example, and this error can be observed by checking out the first commit in the PR.

cd workspace-project
cargo test -- --nocapture

Should show these two results for parallel_1 and parallel_2 test cases:

tests/expand/second.rs - different!
Diff [lines: 6 added, 1 removed]:
--------------------------
 #[macro_use]
 extern crate test_project;
 pub fn main() {
+    {
+        let mut temp_vec = Vec::new();
+        temp_vec.push(1);
+        temp_vec
+    };
-    ();
 }
+
--------------------------



thread 'parallel_2' panicked at '1 of 1 tests failed', /home/user/src/macrotest/src/expand.rs:171:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test parallel_2 ... FAILED
tests/expand/first.rs - different!
Diff [lines: 2 added, 1 removed]:
--------------------------
 #[macro_use]
 extern crate test_project;
 pub fn main() {
+    Vec::new();
-    ();
 }
+
--------------------------



thread 'parallel_1' panicked at '1 of 1 tests failed', /home/user/src/macrotest/src/expand.rs:171:9
test parallel_1 ... FAILED

failures:

failures:
    parallel_1
    parallel_2

Place multiple test cases under a single crate

Otherwise, in the case of proc-macros, we have to recompile the entire crate for each test case. This is really hurting test performance.

We should place each test case as a [[bin]] section in temporary crate's Cargo.toml and pass the --bin argument to the cargo expand for each corresponding test case.

Wrong diff

To test what a test failure looks like, I tried this simple test case:

fn main() {
    println!();
}

and an .expanded.rs file with an extra newline at the top to trigger a mismatch:

​
fn main() {
    ::std::io::_print(::core::fmt::Arguments::new_v1(
        &["\n"],
        &match () {
            () => [],
        },
    ));
}

The diff printed by macrotest is:

Running 1 macro expansion tests
x - different!
Diff [lines: 5 added, 0 removed]:
--------------------------
+#![feature(prelude_import)]
+#[prelude_import]
+use std::prelude::v1::*;
+#[macro_use]
+extern crate std;
 fn main() {
     ::std::io::_print(::core::fmt::Arguments::new_v1(
         &["\n"],
         &match () {
             () => [],
         },
     ));
 }
--------------------------

It claims 5 added/0 removed. The correct diff would have 0 added/1 removed.

Please publish 1.0.9

I am specifically interested in using #78 in serde_derive's test suite.

Thanks!

Allow RegEx or wild cards

Is it possible to test an expansion and compare against a regular expression, or to allow wild cards?

Support uncolored terminals

The current output depends on red/green lines to highlight the diff, but when looking at this without colors the diff doesn't make sense. Is there a different representation that could be more accessible when colors don't exist?

I hit this when running:

$ cargo test 2>&1 | gvim -

Release latest version to fix `expand_arg`

I was trying to setup macrotest and found that expand_arg was not working to pass my feature flags to the tests. This seems to be an issue with version "1.0.9", because pointing macrotest to the latest commit on main c4151a5 fixed to issue.

The output i got when on "1.0.9" was:

Running 1 macro expansion tests
Expansion error:
error: Package `package v0.0.0 (path/to/package)` does not have the feature `nightly`

Fails under cargo nextest

Cargo Nextest has a different test execution model where each test is compiled into a separate binary. This causes issues with macrotest- it seems that it assumes that each crate will contain at most one call to macrotest from a single binary.

This is reproducible from the tests within this repo:

cargo install cargo-nextest
git clone https://github.com/eupn/macrotest.git
cd macrotest/test-project
cargo nextest run

Output:

cargo nextest run
    Finished test [unoptimized + debuginfo] target(s) in 0.04s
    Starting 9 tests across 3 binaries (run ID: 0c2815c1-f154-41a9-bf6f-819b59413968, nextest profile: default)
        FAIL [   0.194s] test-project::par_tests_regression parallel_2

--- STDOUT:              test-project::par_tests_regression parallel_2 ---

running 1 test
Running 1 macro expansion tests
test parallel_2 ... FAILED

failures:

failures:
    parallel_2

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.07s


--- STDERR:              test-project::par_tests_regression parallel_2 ---
thread 'parallel_2' panicked at D:\git-repos\github\macrotest\src\expand.rs:128:9:
prepare failed: IoError(
    Os {
        code: 32,
        kind: Uncategorized,
        message: "The process cannot access the file because it is being used by another process.",
    },
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

   Canceling due to test failure: 8 tests still running
        PASS [   0.174s] test-project::tests fail_expect_expanded
        PASS [   0.153s] test-project::tests fail_expect_expanded_args
        FAIL [   0.134s] test-project::tests pass

--- STDOUT:              test-project::tests pass ---

running 1 test
Running 4 macro expansion tests
test pass ... FAILED

failures:

failures:
    pass

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 6 filtered out; finished in 0.08s


--- STDERR:              test-project::tests pass ---
thread 'pass' panicked at D:\git-repos\github\macrotest\src\expand.rs:128:9:
prepare failed: IoError(
    Os {
        code: 32,
        kind: Uncategorized,
        message: "The process cannot access the file because it is being used by another process.",
    },
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.140s] test-project::tests pass_args

--- STDOUT:              test-project::tests pass_args ---

running 1 test
Running 1 macro expansion tests
test pass_args ... FAILED

failures:

failures:
    pass_args

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 6 filtered out; finished in 0.08s


--- STDERR:              test-project::tests pass_args ---
thread 'pass_args' panicked at D:\git-repos\github\macrotest\src\expand.rs:128:9:
prepare failed: IoError(
    Os {
        code: 32,
        kind: Uncategorized,
        message: "The process cannot access the file because it is being used by another process.",
    },
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.133s] test-project::tests pass_expect_expanded

--- STDOUT:              test-project::tests pass_expect_expanded ---

running 1 test
Running 4 macro expansion tests
test pass_expect_expanded ... FAILED

failures:

failures:
    pass_expect_expanded

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 6 filtered out; finished in 0.08s


--- STDERR:              test-project::tests pass_expect_expanded ---
thread 'pass_expect_expanded' panicked at D:\git-repos\github\macrotest\src\expand.rs:128:9:
prepare failed: IoError(
    Os {
        code: 32,
        kind: Uncategorized,
        message: "The process cannot access the file because it is being used by another process.",
    },
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.144s] test-project::tests pass_expect_expanded_args

--- STDOUT:              test-project::tests pass_expect_expanded_args ---

running 1 test
Running 1 macro expansion tests
test pass_expect_expanded_args ... FAILED

failures:

failures:
    pass_expect_expanded_args

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 6 filtered out; finished in 0.09s


--- STDERR:              test-project::tests pass_expect_expanded_args ---
thread 'pass_expect_expanded_args' panicked at D:\git-repos\github\macrotest\src\expand.rs:128:9:
prepare failed: IoError(
    Os {
        code: 32,
        kind: Uncategorized,
        message: "The process cannot access the file because it is being used by another process.",
    },
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.131s] test-project::tests pr61

--- STDOUT:              test-project::tests pr61 ---

running 1 test
Running 2 macro expansion tests
test pr61 ... FAILED

failures:

failures:
    pr61

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 6 filtered out; finished in 0.08s


--- STDERR:              test-project::tests pr61 ---
thread 'pr61' panicked at D:\git-repos\github\macrotest\src\expand.rs:128:9:
prepare failed: IoError(
    Os {
        code: 32,
        kind: Uncategorized,
        message: "The process cannot access the file because it is being used by another process.",
    },
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        PASS [   1.853s] test-project::par_tests_regression parallel_1
------------
     Summary [   1.856s] 9 tests run: 3 passed, 6 failed, 0 skipped
        FAIL [   0.194s] test-project::par_tests_regression parallel_2
        FAIL [   0.134s] test-project::tests pass
        FAIL [   0.140s] test-project::tests pass_args
        FAIL [   0.133s] test-project::tests pass_expect_expanded
        FAIL [   0.144s] test-project::tests pass_expect_expanded_args
        FAIL [   0.131s] test-project::tests pr61
error: test run failed

Reduce public API

This crate does exactly one thing from a user's perspective so I would expect exactly one function as the entry point:

#[test]
fn expand() {
    macrotest::expand("tests/expand/*.rs");
}

Preserve compiled dependencies across test runs

Consecutive invocations of cargo test should not recompile all the dependencies each time. By placing the work under Cargo's targets directory rather than temporary directory, the already compiled dependencies persist across invocations.

Migrate from Travis CI to Github Actions

See title.

Request: MSRV

I would like request that macrotest decides on a Minimum Supported Rust Version (MSRV).

The latest release 1.0.6 doesn't build on Rust v1.34.
This is a problem for duplicate, as we have an MSRV of 1.34. Therefore, it cannot update to the latest version: see this build result.
As long as macrotest doesn't commit to an MSRV (that is compatible with duplicate's), I must assume each release (even patch releases) may break our setup.

Of course, I would prefer that the eventual MSRV chosen for macrotest be compatible with duplicate's MSRV.
It can be found on crates.io in the "MSRV Policy" section.
You can also see how I decided on an MSRV policy on this issue.
You can also look at the Github Actions and Travis CI setups for how duplicate enforces the MSRV.

Even if macrotest doesn't get and MSRV of 1.34, I would prefer to know what the policy is. I can then use that knowledge to guide any MSRV policy updates.

Show cargo output while running

In trybuild, if the crate you are testing has dependencies, you can see Cargo compiling those during the test run. In macrotest it doesn't show any output, it just stalls for a long time which can be unfriendly.

crate ownership - please appoint a co-maintainer

Hi.

I notice that this crate hasn't had a release in a while, and also that there's just one maintaner.

It's normally a good idea to have multiple owners of a crate on crates.io. That way if for any reason you can't work on the project, your co-maintainer can do so (or, further delegate to someone who actually wants to do the work).

Can I respectfully request that you consider appointing a co-maintainer? They don't have to be someone who's willing to do a lot of technical work. They ideally would be someone you'd trust to steward the crate into new ownership, and/or to do the release work after others have prepared the codebase. So ideally they'd be someone you feel aligned with on philosophical/ideological questions as well as just technical taste.

(Having a co-maintainer on crates.io is much more important than co-ownership of the git repository, because switching to a different crate name is very disruptive for downstreams.)

Thanks for your attention.

Bug: Running multiple expansion tests may interfere with each other

Using macrotest::expand_without_refresh in multiple tests may cause them to interfere with each other destructively.

Say we have 2 expansion tests we want to run: set_1.rs and set_2.rs with each their .expanded.rs file to test against.
We then want to run these expansion tests in different tests:

#[test]
fn test_set_1(){
	macrotest::expand_without_refresh("tests/expand/set_1.rs");
}
	
#[test]
fn test_set_2(){
	macrotest::expand_without_refresh("tests/expand/set_2.rs");
}

When we run this with cargo test we get the following error:

---- test_set_1 stdout ----
Running 1 macro expansion tests
Expansion error:
error: no bin target named `set_1`
        Did you mean `set_2`?

However if we run these test one at a time with cargo test -- --test-threads=1 we do not get this error.
I have create a repository that exhibits this behavior.
Note that this is regardless of the contents of the files under test. In this example the set_2.rs test should actually fail because the expanded.rs file is not correct. But even if we change this, the above message will still be thrown.

I realize that we could run these two file in one test (macrotest::expand_without_refresh("tests/expand/*.rs");), however, my original use case requires me to have expansion tests split because of different features I want to test.

Also, I have no idea whether this is a problem with macrotest itself or with cargo expand.

Consider normalizing the expanded code

macrotest/test-procmacro-project/tests/expand/attribute.expanded.rs

Lines 1 to 6 in 0beeb43

	#![feature(prelude_import)]
	#![no_std]
	#[prelude_import]
	use ::std::prelude::v1::*;
	#[macro_use]
	extern crate std;

I think the above lines should be stripped from the expanded code prior to comparing it against expanded.rs.

That way any time rustc makes insignificant changes to this header, we only need one patch in macrotest to accommodate the new header rather than updating every expanded.rs in every project downstream.

proc-macro for generating separate test cases for each generated output sample

It would be awesome to have a macro for generating separate test cases for each generated sample in the expanded/ directory.

Motivation for that is having more granular test results, which would show a test failure for a specific macro or specific macro invocation.

The ideal API, IMHO, would be something like

#[macrotest::tests("expanded/*")]
mod macro_tests;

Print notice when updating .expanded.rs files

Currently a cargo test with deleted .expanded.rs files will silently pass with no indication that new files have been written. It would be better to print some sort of notice, since the tests didn't actually test anything; a human would need to look at the new files to confirm that they are as intended.

panic in syn 1 on "(/*ERROR*/)" token

To reproduce:

rustup toolchain add nightly-2023-09-04
git clone https://gitlab.torproject.org/Diziet/rust-derive-deftly
cd rust-derive-deftly
git  checkout edition-macrotest-broken
RUST_BACKTRACE=1 RUSTFLAGS="--cfg derive_deftly_dprint" TRYBUILD=overwrite MACROTEST=overwrite STDERRTEST=overwrite cargo +nightly-2023-09-04 test --locked --offline --workspace --all-features -- macrotest

Expected behaviour: test failures due to mismatched output, and overwrites of various files (sorry).

Actual behaviour: panic in syn.

Notes: seems to be fixed on macrotest master. I think updating to syn 2 fixes it.

test macrotest_expand ... FAILED

failures:

---- macrotest_expand stdout ----
Running 16 macro expansion tests
thread 'macrotest_expand' panicked at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/lit.rs:1020:13:
Unrecognized literal: `(/*ERROR*/)`
stack backtrace:
   0: rust_begin_unwind
             at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/std/src/panicking.rs:619:5
   1: core::panicking::panic_fmt
             at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/core/src/panicking.rs:72:14
   2: syn::lit::value::<impl syn::lit::Lit>::new
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/lit.rs:1020:13
   3: syn::lit::parsing::<impl syn::parse::Parse for syn::lit::Lit>::parse::{{closure}}
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/lit.rs:749:32
   4: syn::parse::ParseBuffer::step
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/parse.rs:1027:28
   5: syn::lit::parsing::<impl syn::parse::Parse for syn::lit::Lit>::parse
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/lit.rs:747:13
   6: <syn::lit::Lit as syn::token::Token>::peek::peek
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/token.rs:171:21
   7: syn::token::peek_impl
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/token.rs:162:5
   8: <syn::lit::Lit as syn::token::Token>::peek
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/token.rs:173:17
   9: syn::parse::ParseBuffer::peek
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/parse.rs:585:9
  10: syn::expr::parsing::atom_expr
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/expr.rs:1722:19
  11: syn::expr::parsing::trailer_expr
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/expr.rs:1566:20
  12: syn::expr::parsing::unary_expr
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/expr.rs:1536:13
  13: syn::expr::parsing::expr_early
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/expr.rs:2040:28
  14: syn::stmt::parsing::stmt_expr
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/stmt.rs:280:21
  15: syn::stmt::parsing::parse_stmt
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/stmt.rs:207:13
  16: syn::stmt::parsing::<impl syn::stmt::Block>::parse_within
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/stmt.rs:119:25
  17: syn::parse::ParseBuffer::call
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/parse.rs:508:9
  18: syn::item::parsing::parse_rest_of_fn
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/item.rs:1554:21
  19: syn::item::parsing::<impl syn::parse::Parse for syn::item::Item>::parse
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/item.rs:992:21
  20: syn::parse::ParseBuffer::parse
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/parse.rs:468:9
  21: syn::file::parsing::<impl syn::parse::Parse for syn::file::File>::parse
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/file.rs:102:36
  22: core::ops::function::FnOnce::call_once
             at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/core/src/ops/function.rs:250:5
  23: <F as syn::parse::Parser>::parse2
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/parse.rs:1212:20
  24: syn::parse::Parser::parse_str
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/parse.rs:1177:9
  25: syn::parse_str
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/lib.rs:918:5
  26: syn::parse_file
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/syn-1.0.109/src/lib.rs:981:26
  27: macrotest::expand::normalize_expansion
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/macrotest-1.0.9/src/expand.rs:393:33
  28: macrotest::expand::ExpandedTest::run
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/macrotest-1.0.9/src/expand.rs:353:22
  29: macrotest::expand::run_tests
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/macrotest-1.0.9/src/expand.rs:134:15
  30: macrotest::expand::expand_args
             at /home/rustcargo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/macrotest-1.0.9/src/expand.rs:71:5
  31: macrotest::macrotest_expand
             at ./macrotest.rs:17:9
  32: macrotest::macrotest_expand::{{closure}}
             at ./macrotest.rs:15:27
  33: core::ops::function::FnOnce::call_once
             at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/core/src/ops/function.rs:250:5
  34: core::ops::function::FnOnce::call_once
             at /rustc/58e967a9cc3bd39122e8cb728e8cec6e3a4eeef2/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.


failures:
    macrotest_expand

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.77s

error: test failed, to rerun pass `-p derive-deftly-tests --test macrotest`