eupn/macrotest

Test harness for declarative and procedural macros expansion via `cargo-expand`

🕊 NO WAR 🕊

Hi there 👋

• 🦀 Rust
• Embedded Rust
• Cryptography
• Bitcoin

macrotest's Issues

workspaces `dev-dependencies` don't work

I have a workspace where one of the crates uses macrotest. In that crate, I also put a workspace dev-dependency on another crate for other tests via test-sys.workspace = true. This causes macrotest to fail to pass. Other non-macrotest tests pass fine, and removing the workspace dev-dependency causes macrotest tests to pass

output:

---- tests::macro_expansion stdout ----
Running 1 macro expansion tests
Expansion error:
error: failed to parse manifest at `D:\test-ws\target\tests\test-package\macrotest002\Cargo.toml`
Caused by:
  error inheriting `test-sys` from workspace root manifest's `workspace.dependencies.test-sys`
Caused by:
  `workspace.dependencies` was not defined

macrotest version: latest on main c4151a5

Wrong diff

To test what a test failure looks like, I tried this simple test case:

fn main() {
    println!();
}

and an .expanded.rs file with an extra newline at the top to trigger a mismatch:

​
fn main() {
    ::std::io::_print(::core::fmt::Arguments::new_v1(
        &["\n"],
        &match () {
            () => [],
        },
    ));
}

The diff printed by macrotest is:

Running 1 macro expansion tests
x - different!
Diff [lines: 5 added, 0 removed]:
--------------------------
+#![feature(prelude_import)]
+#[prelude_import]
+use std::prelude::v1::*;
+#[macro_use]
+extern crate std;
 fn main() {
     ::std::io::_print(::core::fmt::Arguments::new_v1(
         &["\n"],
         &match () {
             () => [],
         },
     ));
 }
--------------------------

It claims 5 added/0 removed. The correct diff would have 0 added/1 removed.

Consider cargo_toml crate for Cargo.toml manipulations

It might be better and less error-prone to use cargo_toml crate for Cargo.toml manipulations. Currently it's done by using definitions from manifest.rs.

Print notice when updating .expanded.rs files

Currently a cargo test with deleted .expanded.rs files will silently pass with no indication that new files have been written. It would be better to print some sort of notice, since the tests didn't actually test anything; a human would need to look at the new files to confirm that they are as intended.

MSRV violation

macrotest no longer compiles on rust v1.34.2 because serde_json v1.0.73 depends on itoa v1.0.1 which doesn't compile on that version. See (duplicate build)[https://github.com/Emoun/duplicate/runs/4636793495?check_suite_focus=true].

As far as I can tell serde_json has no MSRV, which means macrotest should depend on a specific version to avoid violating MSRV.

Wrong lines stripped if an input contains module-level attribute

Macrotest assumes that the output of cargo expand always begins with the following exactly 5 lines.

macrotest/src/expand.rs

Lines 390 to 391 in 779cfa5

	// `cargo expand` does always produce some fixed amount of lines that should be ignored
	const CARGO_EXPAND_SKIP_LINES_COUNT: usize = 5;

#![feature(prelude_import)]
#[prelude_import]
use std::prelude::$edition::*;
#[macro_use]
extern crate std;

However, that is not the case if there is an inner attribute in the input. For example the following crate:

#![feature(proc_macro_span)]

fn foo() {}

expands as:

#![feature(prelude_import)]
#![feature(proc_macro_span)]
#[prelude_import]
use std::prelude::rust_2018::*;
#[macro_use]
extern crate std;
fn foo() {}

and after stripping exactly 5 lines from the front, macrotest's .expanded.rs will result in:

extern crate std;
fn foo() {}

which is not what is expected. The correct expansion would be:

#![feature(proc_macro_span)]
fn foo() {}

Cut dependencies

My first reaction when trying out this crate was wow that's a lot of dependencies for a simple crate.

Would it be possible to cut this down to the minimum required to implement the functionality? Failure, derive_more, and tempdir would be good candidates to cut.

macrotest v0.1.1 (/git/tmp/macrotest)
├── derive_more v0.15.0
│   ├── lazy_static v1.4.0
│   ├── proc-macro2 v0.4.30
│   │   └── unicode-xid v0.1.0
│   ├── quote v0.6.13
│   │   └── proc-macro2 v0.4.30 (*)
│   ├── regex v1.3.1
│   │   ├── aho-corasick v0.7.6
│   │   │   └── memchr v2.2.1
│   │   ├── memchr v2.2.1 (*)
│   │   ├── regex-syntax v0.6.12
│   │   └── thread_local v0.3.6
│   │       └── lazy_static v1.4.0 (*)
│   └── syn v0.15.44
│       ├── proc-macro2 v0.4.30 (*)
│       ├── quote v0.6.13 (*)
│       └── unicode-xid v0.1.0 (*)
│   [build-dependencies]
│   └── rustc_version v0.2.3
│       └── semver v0.9.0
│           └── semver-parser v0.7.0
├── difference v2.0.0
├── failure v0.1.6
│   ├── backtrace v0.3.40
│   │   ├── backtrace-sys v0.1.32
│   │   │   └── libc v0.2.65
│   │   │   [build-dependencies]
│   │   │   └── cc v1.0.46
│   │   ├── cfg-if v0.1.10
│   │   ├── libc v0.2.65 (*)
│   │   └── rustc-demangle v0.1.16
│   └── failure_derive v0.1.6
│       ├── proc-macro2 v1.0.6
│       │   └── unicode-xid v0.2.0
│       ├── quote v1.0.2
│       │   └── proc-macro2 v1.0.6 (*)
│       ├── syn v1.0.7
│       │   ├── proc-macro2 v1.0.6 (*)
│       │   ├── quote v1.0.2 (*)
│       │   └── unicode-xid v0.2.0 (*)
│       └── synstructure v0.12.1
│           ├── proc-macro2 v1.0.6 (*)
│           ├── quote v1.0.2 (*)
│           ├── syn v1.0.7 (*)
│           └── unicode-xid v0.2.0 (*)
├── glob v0.3.0
├── tempdir v0.3.7
│   ├── rand v0.4.6
│   │   └── libc v0.2.65 (*)
│   └── remove_dir_all v0.5.2
└── toml v0.5.4
    └── serde v1.0.102

Requirement to precisely pin the version of cargo-expand

macrotest depends fundamentally on the stability of the output of cargo expand. But this is not something that cargo expand's maintainers promise: dtolnay/cargo-expand#179.

If one is using macrotest in CI, as one should, the CI will occasionally need to rebuild cargo expand. If it gets a new version, the tests break.

The solution is to use something like

cargo install --locked --version 1.0.44 --features=prettyplease cargo-expand

I think this should be documented.

Release latest version to fix `expand_arg`

I was trying to setup macrotest and found that expand_arg was not working to pass my feature flags to the tests. This seems to be an issue with version "1.0.9", because pointing macrotest to the latest commit on main c4151a5 fixed to issue.

The output i got when on "1.0.9" was:

Running 1 macro expansion tests
Expansion error:
error: Package `package v0.0.0 (path/to/package)` does not have the feature `nightly`

Migrate from Travis CI to Github Actions

See title.

Reduce public API

This crate does exactly one thing from a user's perspective so I would expect exactly one function as the entry point:

#[test]
fn expand() {
    macrotest::expand("tests/expand/*.rs");
}

Request: MSRV

I would like request that macrotest decides on a Minimum Supported Rust Version (MSRV).

The latest release 1.0.6 doesn't build on Rust v1.34.
This is a problem for duplicate, as we have an MSRV of 1.34. Therefore, it cannot update to the latest version: see this build result.
As long as macrotest doesn't commit to an MSRV (that is compatible with duplicate's), I must assume each release (even patch releases) may break our setup.

Of course, I would prefer that the eventual MSRV chosen for macrotest be compatible with duplicate's MSRV.
It can be found on crates.io in the "MSRV Policy" section.
You can also see how I decided on an MSRV policy on this issue.
You can also look at the Github Actions and Travis CI setups for how duplicate enforces the MSRV.

Even if macrotest doesn't get and MSRV of 1.34, I would prefer to know what the policy is. I can then use that knowledge to guide any MSRV policy updates.

macrotest and trybuild together keep doing rebuilds

Steps to reproduce:

(install nightly Rust and cargo-expand)
git clone https://github.com/taiki-e/pin-project
cd pin-project
cargo +nightly --locked --offline test --workspace -- ui # 1
cargo +nightly --locked --offline test --workspace -- ui # 2
cargo +nightly --locked --offline test --workspace -- expandtest # 3
cargo +nightly --locked --offline test --workspace -- expandtest # 4
cargo +nightly --locked --offline test --workspace -- ui # 5
cargo +nightly --locked --offline test --workspace -- expandtest #6

Expected behaviour: In steps 1 and 3, some actual compilation work takes place. In steps 2,4,5,6, nothing much is compiled and the tests run quickly.

Actual behaviour: In steps 5 and 6, many things are rebuilt including proc-macro2, syn, ...

I don't know what is going on here, and the root cause is almost certainly a bug in cargo. But I think it would be good if both macrotest and trybuild could work around it. Probably, the best workaround is for them each to use a separate target directory. That would involve rebuilding extra things on the first iteration, but thereeafter the previously built things would be reused.

This happened to me with an unpublished version of a package of mine. When investigating I thought this seemed to be a bug elsewhere, so I looked through the rdependencies of macrotest for a package which used both. pin-project was the obvious candidate, and indeed it is affected. I looked at the test driver code in pin-project and there doesn't seem to be anything unexpected there that might cause this problem.

Since github doesn't support a single issue affecting multiple projects, and anyway I think probably it would be nice if both packages were changed to incorporate a workaround (even though I think one package doing so would suffice), I am filing an identical ticket against trybuild too.

Please publish 1.0.9

I am specifically interested in using #78 in serde_derive's test suite.

Thanks!

Allow the passing of flags to expand

I would like the possibility of passing flags to cargo in macrotest::expand_without_refresh.

My use case is that I would like to test my expansion with different crate features. It seems to me that macrotest::expand_without_refresh simply calls cargo expand, which means the default features of the crate under test are used.
I would like to be able to disable the default features and enable other features in a granular fashion:

// Test expansion without any features
macrotest::expand_without_refresh("some_test_file.rs", "--no-default-features"); 

// Test expansion with only feature "some_feature" enabled
macrotest::expand_without_refresh("some_test_file.rs", "--features some_feature"); 

Of course the exact API doesn't have to look like the above.

Support uncolored terminals

The current output depends on red/green lines to highlight the diff, but when looking at this without colors the diff doesn't make sense. Is there a different representation that could be more accessible when colors don't exist?

I hit this when running:

$ cargo test 2>&1 | gvim -

Parallel tests fail in workspace project

Thank you for the effort you have put into making macrotest, and for making it open source.

I ran into this issue in the course of trying to isolate what appears to be unexpected behavior.

PR #72 proposes to add a workspace project example, and this error can be observed by checking out the first commit in the PR.

cd workspace-project
cargo test -- --nocapture

Should show these two results for parallel_1 and parallel_2 test cases:

tests/expand/second.rs - different!
Diff [lines: 6 added, 1 removed]:
--------------------------
 #[macro_use]
 extern crate test_project;
 pub fn main() {
+    {
+        let mut temp_vec = Vec::new();
+        temp_vec.push(1);
+        temp_vec
+    };
-    ();
 }
+
--------------------------



thread 'parallel_2' panicked at '1 of 1 tests failed', /home/user/src/macrotest/src/expand.rs:171:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test parallel_2 ... FAILED
tests/expand/first.rs - different!
Diff [lines: 2 added, 1 removed]:
--------------------------
 #[macro_use]
 extern crate test_project;
 pub fn main() {
+    Vec::new();
-    ();
 }
+
--------------------------



thread 'parallel_1' panicked at '1 of 1 tests failed', /home/user/src/macrotest/src/expand.rs:171:9
test parallel_1 ... FAILED

failures:

failures:
    parallel_1
    parallel_2

Wrong report when macro expansion fails

For all macrotest::expand variants, if the macro being expanded panics, the report gives wrong or confusing errors.

First, if you have no .expanded file and not using expand_without_refresh, the call will succeed (even though the expanded macro panicked) and create an empty .expanded file. If you are using expand_without_refresh the report will correctly show an error however will give no indication about what happened.

If you do have an .expanded file the report will give a <file-path> - different! error, and show the contents of your expanded file but nothing else.

For all of these cases, I would expect the error message to tell me that the macro expansion failed and maybe show me the error message it failed with.

Note: I found these cases while testing #66, however, these problems are also present in v1.0.8.

Place multiple test cases under a single crate

Otherwise, in the case of proc-macros, we have to recompile the entire crate for each test case. This is really hurting test performance.

We should place each test case as a [[bin]] section in temporary crate's Cargo.toml and pass the --bin argument to the cargo expand for each corresponding test case.

Emit newline character at the end of .expanded.rs files

Proper text files should contain a newline character at the end.

Mention that rustfmt is required for proper work

At least for examples in test-*-projects.

Consider normalizing the expanded code

macrotest/test-procmacro-project/tests/expand/attribute.expanded.rs

Lines 1 to 6 in 0beeb43

	#![feature(prelude_import)]
	#![no_std]
	#[prelude_import]
	use ::std::prelude::v1::*;
	#[macro_use]
	extern crate std;

I think the above lines should be stripped from the expanded code prior to comparing it against expanded.rs.

That way any time rustc makes insignificant changes to this header, we only need one patch in macrotest to accommodate the new header rather than updating every expanded.rs in every project downstream.

Package macrotest000 does not have the feature `test-feature`

I've encountered this issue while implementing a infinyon/fluvio plugin-style integration test harness for Minitest, PR tikv/minitrace-rust/pull/127.

I have now reproduced the issue in a test-virtual project that will, hopefully, land here via PR #72.

The test-virtual example shows how to setup macrotest with a generic (non-libtest) test harness, and allows macrotest users to compare this plugin-style integration test suite with the default Rust libtest test harness.

At the moment the test-virtual project returns this error:

cargo test integration-tests 

<snip>

Expansion error:
error: Package `wrkspc-test-tests v0.0.0 (/home/user/src/macrotest/test-virtual/target/tests/wrkspc-test/macrotest000)` does not have the feature `test-feature`

I've banged my head on both projects but can't seem to figure out where the mismatch is occuring.

Appreciate any help or insights you can share.

Feature Request: Expand fail

Hi,

I want to ask whether you would consider implementing what trybuild essentially does, I.e. allowing us to check that a compilation fails with a given error message?

The reason I ask is that I'm in need of the functionality that trybuild provides, however it is missing essential features that macrotest has.
Most essential is the fact that it doesn't seem to panic upon wrong test results, which is a deal-breaker for me.
Also, features that have been implemented in macrotest are also missing (#41,#43).
Additionally, it seems silly to me that I would need to use two different crates for two functions that are so similar.

Essentially what I would need is expand_fail versions of expand and expand_without_refresh, that expects the expansion to fail and then checks the error message is as expected. If the expansion doesn't fail, or the error message is wrong, the test is a failure.

proc-macro for generating separate test cases for each generated output sample

It would be awesome to have a macro for generating separate test cases for each generated sample in the expanded/ directory.

Motivation for that is having more granular test results, which would show a test failure for a specific macro or specific macro invocation.

The ideal API, IMHO, would be something like

#[macrotest::tests("expanded/*")]
mod macro_tests;

Typo in test failed message

I noticed this in part of the output when a test fails:

thread 'pass' panicked at '1 or 4 tests failed', /git/tmp/macrotest/src/expand.rs:58:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

It seems like macrotest should know whether it was 1 or 4 tests that failed!

Probably this should be "1 of 4 tests failed".

test-project: cargo test shows failures

Context: Investigating the cause of issue #71, in the course of making PR #72.

Using the current master:

cd test-project
cargo test -- --nocapture

Shows some tests failing:

    Finished test [unoptimized + debuginfo] target(s) in 0.02s
     Running unittests (target/debug/deps/test_project-90e6064d5d3cb538)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/par_tests_regression.rs (target/debug/deps/par_tests_regression-ed3d7a792e4170a2)

running 2 tests
Running 1 macro expansion tests
Running 1 macro expansion tests
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest000)
    Blocking waiting for file lock on build directory
    Finished dev [unoptimized + debuginfo] target(s) in 0.18s
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest001)
    Finished dev [unoptimized + debuginfo] target(s) in 0.48s

use std::prelude::rust_2018::*;

use std::prelude::rust_2018::*;
tests/expand/second.rs - ok
test parallel_2 ... ok
tests/expand/first.rs - ok
test parallel_1 ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.78s

     Running tests/tests.rs (target/debug/deps/tests-78123928aa400a33)

running 7 tests
Running 4 macro expansion tests
Running 1 macro expansion tests
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest000)
    Blocking waiting for file lock on build directory
    Finished dev [unoptimized + debuginfo] target(s) in 0.24s
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest001)

use std::prelude::rust_2018::*;
    Finished dev [unoptimized + debuginfo] target(s) in 0.99s

use std::prelude::rust_2018::*;
tests/no_expanded/first.expanded.rs is expected but not found
tests/no_expanded_args/with_args.expanded.rs is expected but not found



thread 'fail_expect_expanded_args' panicked at '1 of 1 tests failed', /home/user/src/macrotest/src/expand.rs:171:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test fail_expect_expanded_args - should panic ... ok
Running 4 macro expansion tests
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest002)
tests/no_expanded/fourth.expanded.rs is expected but not found
    Finished dev [unoptimized + debuginfo] target(s) in 0.55s

use std::prelude::rust_2018::*;
tests/no_expanded/second.expanded.rs is expected but not found
tests/expand/first.rs - ok
tests/no_expanded/third.expanded.rs is expected but not found



thread 'fail_expect_expanded' panicked at '4 of 4 tests failed', /home/user/src/macrotest/src/expand.rs:171:9
test fail_expect_expanded - should panic ... ok
Running 1 macro expansion tests
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest003)
tests/expand/fourth.rs - ok
    Finished dev [unoptimized + debuginfo] target(s) in 1.00s

use std::prelude::rust_2018::*;
tests/expand/second.rs - ok
tests/expand_args/with_args.rs - ok
test pass_args ... ok
Running 4 macro expansion tests
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest004)
tests/expand/third.rs - ok
test pass ... ok
Running 1 macro expansion tests
    Finished dev [unoptimized + debuginfo] target(s) in 0.57s
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest005)

use std::prelude::rust_2018::*;
    Finished dev [unoptimized + debuginfo] target(s) in 0.47s

use std::prelude::rust_2018::*;
tests/expand/first.rs - ok
tests/expand_args/with_args.rs - ok
test pass_expect_expanded_args ... ok
Running 2 macro expansion tests
    Checking test-project-tests v0.0.0 (/home/user/src/macrotest/test-project/target/tests/test-project/macrotest006)
tests/expand/fourth.rs - ok
    Finished dev [unoptimized + debuginfo] target(s) in 0.47s

use std::prelude::rust_2018::*;
tests/expand/second.rs - ok
tests/pr61/a/test.rs - ok
tests/expand/third.rs - ok
test pass_expect_expanded ... ok
tests/pr61/b/test.rs - ok
test pr61 ... ok

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 17.27s

   Doc-tests test-project

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Preserve compiled dependencies across test runs

Consecutive invocations of cargo test should not recompile all the dependencies each time. By placing the work under Cargo's targets directory rather than temporary directory, the already compiled dependencies persist across invocations.

v1.0.9 violates MSRV

Version 1.0.9 of macrotest introduced the use of prettyplease crate. See #78.

Problem is, all versions of prettyplease use edition 2021, meaning the earliest rustc version that will work now is 1.56 (the version introducing edition 2021).
Since this crate officially has an MSRV of 1.34, version 1.0.9 is violating it. In my opinion v1.0.9 should be yanked from crates.io.
There should probably also be some testing happening for whether the MSRV is upheld so this doesn't happen again.

Use the same edition as the crate under test

Copy the edition field of package section of the crate's Cargo.toml into the "temporary crate"'s manifest. It should be reasonable to assume that the test cases are using the same edition as the crate which macros they're testing.

Quote comment

I noticed this line in the example project:

macrotest/test-procmacro-project/src/lib.rs

Lines 1 to 2 in 0beeb43

	// You might need to use this in future since quote! macro can be highly-recursive
	//#![recursion_limit="1024"]

I don't think this is accurate anymore since quote isn't recursive.

Common, error

What is this stuff? Are users supposed to use it? It wasn't clear from the docs what role they play, since the examples only use TestCases.

Bug: Running multiple expansion tests may interfere with each other

Using macrotest::expand_without_refresh in multiple tests may cause them to interfere with each other destructively.

Say we have 2 expansion tests we want to run: set_1.rs and set_2.rs with each their .expanded.rs file to test against.
We then want to run these expansion tests in different tests:

#[test]
fn test_set_1(){
	macrotest::expand_without_refresh("tests/expand/set_1.rs");
}
	
#[test]
fn test_set_2(){
	macrotest::expand_without_refresh("tests/expand/set_2.rs");
}

When we run this with cargo test we get the following error:

---- test_set_1 stdout ----
Running 1 macro expansion tests
Expansion error:
error: no bin target named `set_1`
        Did you mean `set_2`?

However if we run these test one at a time with cargo test -- --test-threads=1 we do not get this error.
I have create a repository that exhibits this behavior.
Note that this is regardless of the contents of the files under test. In this example the set_2.rs test should actually fail because the expanded.rs file is not correct. But even if we change this, the above message will still be thrown.

I realize that we could run these two file in one test (macrotest::expand_without_refresh("tests/expand/*.rs");), however, my original use case requires me to have expansion tests split because of different features I want to test.

Also, I have no idea whether this is a problem with macrotest itself or with cargo expand.

Configurable refresh

I would be useful if we were able to configure whether the "refresh" outcome of expand could be turned off or result in a failure.

I specifically have a setup where I'd rather have a missing .expanded.rs file trigger a test failure and not create the expanded file for me. I like that the file could be created for me, but I'd prefer if it was only when I specifically ask for it.

A possible solution is simply adding another method that doesn't create expanded files when they are missing, simply failing instead:

#[test]
pub fn pass() {
    macrotest::expand_without_refresh("tests/expand/*.rs");
}

Allow RegEx or wild cards

Is it possible to test an expansion and compare against a regular expression, or to allow wild cards?

Show cargo output while running

In trybuild, if the crate you are testing has dependencies, you can see Cargo compiling those during the test run. In macrotest it doesn't show any output, it just stalls for a long time which can be unfriendly.