la10736/rstest

Fixture-based test framework for Rust

rstest's Issues

rstest_parametrize should accept comma after last case

#[rstest_parametrize(u, s, t,
    case(42, "str", Unwrap(r#"("ss", -12)"#)),
    case(24, "trs", Unwrap(r#"("tt", -24)"#)),
)]
fn should_fail(u: u32, s: &str, t: (&str, i32)) {
    assert!(false);
}

Should work.

Pay attention: should work also if some other attributes exists at the end.

EX:

#[rstest_parametrize(u, s, t,
    case(42, "str", Unwrap(r#"("ss", -12)"#)),
    case(24, "trs", Unwrap(r#"("tt", -24)"#)),
    ::trace
)]
fn should_fail(u: u32, s: &str, t: (&str, i32)) {
    assert!(false);
}

Remove r Unwrap alias and deprecate Unwrap

By introducing #19 Unwrap() have no sense. Before release next version remove r() alias introduced in #12 and deprecate Unwrap() by add a warning message

Suppress unused variable warning

If you have a fixture that declare but not use an input fixture and use an #[allow(unused_variables)] attribute. Also the rendered fixture should suppress the same warning.

This pattern is common when you just need the fixture's data ownership and nothing else.

type SyncGuard = std::sync::MutexGuard<'static, ()>;
#[fixture]
fn synchronize() -> SyncGuard {
    use std::sync::Mutex;
    lazy_static!{
        static ref MUTEX : Mutex<()> = Mutex::new(());
    }
    MUTEX.lock().unwrap()
}

#[fixture]
#[allow(unused_variables)]
fn empty(synchronize: SyncGuard) -> TempFile {
   // Some code that need to be syncronized
}

Note take care to write an end to end test for both fixture, rstest and rstest_parametrize

Move render stuff in a new module

Create a new module for render stuff and move tests (parse and render) to separate files

Dump arguments values before start test

Give a syntax to produce a dump of all fixture value for both rstest and rstest_parametrize.

Use specialization to implement it for non debug value (print just address). Specialization is opted in by feature that require nightly (till specialization become stable).

Standard behavior:

• should indicate that you want dump test input values
• can exclude some values by options
• or enable nightly feature to dump value address

Introduce a new crate rstest_core to relay impl.

Investigate to use cargo-vendor in tests

We can configure integration tests projects to use cargo vendor by create a repository from source.

I want multiple tests to share a single set of cases

Problem

I want multiple tests to share a single set of cases

Suggestion: Use a const vector/array as cases

let cases = [
  (0, "zero"),
  (1, "one"),
];

#[rstest_from(cases)]
fn mytest (case: (i32, &str)) {
  // ...
}

case with wrong arguments number should not compile

Example:

#[rstest_parametrize(a, case(1,2))]
fn example(a: u32) {
    assert!(true)
}

should not compile and give a meaningful error message.

Also

#[rstest_parametrize(a, case())]
fn example2(a: u32) {
    assert!(true)
}

Should indicate that case() don't match with given arguments.

Introduce r() as Unwrap() alias

Unwrap("<arbitrary Rust code>") is to verbose. Introduce r("<arbitrary Rust code>") as alias. r stants for raw code.

Remove `parse_str()` from production code

Use it just in test otherwise use quote!{} and eventually parse2().

fixture that return unsized type

test it

fixture that return impl types

test

rstest_parametrize should relay on module name to group cases

Instead to repeat the test function name for each case is better to create a module with function name and create function with a generic case_xxx_<name> in the module.

So you will have a good tree view in Intellj and also a better list by cargo.

fixture that take dyn in input

test it

Matrix version of rstest_parametrize

this code:

#[rstest_matrix(
  foo => [0, 1, 2],
  bar => ['a', 'b'],
)]
fn mytest(foo: i32, bar: char) {
  // ...
}

should be equivalent to this code:

#[rstest_parametrize(
  foo, bar,
  case(0, 'a'),
  case(1, 'a'),
  case(2, 'a'),
  case(0, 'b'),
  case(1, 'b'),
  case(2, 'b'),
)]
fn mytest(foo: i32, bar: char) {
  // ...
}

Use dbg!() macro to trace arguments

We should use dbg!() macro to trace input arguments instead to use custom println! messages.

Also use eprintln! for banners.

Combine rstest with proptest?

Is there any recommended/possible way of using rstest in combination with the amazing proptest? If there were, this would truly make this a killer combination for integration testing.

Replace `Deindent` by `Unindent` via `unindent` crate

I already use unindent crate. When dtolnay/indoc#30 is merged and release replace all deindent() use by unindent().

Also replace the the use of unindent function in should_add_a_test_cases_from_all_combinations.

Integrate uncover crate

Integrate uncover crate with rstest means to have a possibility to define the tags that a test/case cover or use implicity the test description as cover tag.

Missed argument should raise an error

#[rstest_parametrize(f, case(42))]
fn error_param_not_exist() {}

Should not compile and give and indication that f argument is missed

Fixture injection

Define a fixture procedural macro that define a fixture.

Input fixture can be simple functions or should be fixture functions. Fixture function will return Fixture wrapper that can be nested and control value life cycle.

We'll associate to fixture two static methods:

• get(args) that return exactly what the original function returns
• default() that return get()'s result by resolve all arguments as fixtures

Fixture will execute tear down at the end of the test and not when fixture value will be dropped.

We leave teardown implementation for next release and a new ticket. Maybe get and default will return a tuple instead a wrapper and Fixture will be a trait implemented on that tuples.

Note: We not use a wrapper for fixture return type to not go deep in the impl resolution hell. By now struct cannot wrap any function return type.

Can you do an intermittent release?

I currently can't publish my crate because I really want to use ? inside #[rstest] tests but that only works with the git version. Could you publish a new version that only has those new compatibility changes?

Rewrite `ParametrizeData::parse()`

Write unit tests for ParametrizeData::parse() and rewrite it by peek test case instead serch for ( or ::.

TestCase::peek() should check for case ident, an optional description and a parenthesis. Take a look to UnwrapRustCode::peek() as example.

Meaningful error message in case formatting

Now if we write something like

#[rstest_parametrize(a, case(incorrect(some)))]
fn example(a: u32) {
    assert!(true)
}

We receive

error: custom attribute panicked
 --> src/main.rs:7:1
  |
7 | #[rstest_parametrize(a, case(incorrect(some)))]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = help: message: Unexpected case attribute: List(MetaList { ident: Ident { ident: "incorrect", span: #0 bytes(176..185) }, paren_token: Paren, nested: [Meta(Word(Ident { ident: "some", span: #0 bytes(186..190) }))] })

Should be a meaningful error message that point to incorrect(some) span.

Documentation

Write some examples to document how to use the library.

Use just `[rstest]` to generate test.

My macro knowledge is pretty poor so please excuse my ignorance, but I think it'd be conceptually and syntactically to use have [rstest] which may also be optionally parametrized. What do you think?

fixture function visibility

Fixture should mantain the original visibility

Better error message while cannot trace a fixture

Investigate if it's possible to point out that a type should implement Debug if you want trace it

Take a look to quote_spanned!

Rewrite test by assert_cmd

Adopt https://docs.rs/assert_cmd/0.11.0/assert_cmd/ and rewrite projetct test by use it

Dynamic set of tests

In general the pattern would be:

lazy_static! {
    static ref TEST_CASES = ...; // Some computed set of cases.
}

#[rtest_from(TEST_CASES)]
fn test_case(...) {
    ...
}

This isn't just an alternative syntax, it allows computing a dynamic set of cases. In my use case, each test case is derived from a disk directory containing input files and expected output files. Right now, every time I create a new such test directory, I have to also write a silly:

#[test]
fn test_foo() {
    run_test_using_directory("foo");
}

Admittedly using rtest would reduce this overhead to just adding a single line case("foo") in a list, but it would still be best if I could just use glob to generate this list at run time, so just creating the directory would suffice.

However, this means that rtest_from can't work in the way described above; as a compile macro it has no way to access the dynamic values in TEST_CASES and therefore it can't "simply" generate a static test case function for each one.

Google search did not locate any way to generate a dynamic list of tests... is it even possible?

Originally posted by @orenbenkiki in #37 (comment)

Add CI

Implement name attribute for parametrize case

#[rstest_parametrize(a,
    case(42::name(universal))
)]
fn some(a: u32) {
}

And generate a case like some_case_1_universal

Inject fixture args from test

Now there isn't any possibility to pass argument to fixture from test: fixtures use just fixtures to resolve its arguments.

I would like introduce a new syntax applicable to every test render or fixture to pass some or all arguments to the fixture.

By now it is sufficient to implement the possibility to call the fixture by the first n arguments for every n less equal to fixture arguments. In a future I can develop a way to call named args but it will be a new issue.

Syntax

From calling side every time there is an argument like name(expr_1,...,expr_n) it will try to resolve name arguments of test by call fixture with the first n arguments from (expr_1,...,expr_n).

Will generate an error if

• name is already used (other fixture, matrix or parametrized)
• name is not a function argument

Cavelets

We cannot use case as fixture name. Emit a warning on fixture that use case as fixture name with some arguments that will not possible to use that feature.

The partial functions return type can be hard to guess. Maybe I'll need a syntax to indicate them like I did for default

rstest_parametrize does not work in src/bin files

I have tried #[rstest] and #[rstest_parametrize] in a mod test{…} submodule in src and it works fine. However if I use them in a src/bin file, the #[rstest] works fine but the #[rstest_parametrize] fails to work. It seems that the rstest_parametrize stuff fails to work in a test submodule of a subsidiary binary.

Test does not run when passed a negative number

I've been puzzled by tests that wouldn't run. I finally traced it down to giving a negative argument.

The following works:

    #[rstest_parametrize(
        input, expected,
        case(0.0, 0.0),
        case(1.0, 1.0),
        )]
    fn identity(input: f64, expected: f64) {
        assert_eq!(input, expected);
    }

and produces the following output:

running 2 tests
test point::tests::identity_case_0 ... ok
test point::tests::identity_case_1 ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Adding a case with a negative value, the test doesn't run anymore:

    #[rstest_parametrize(
        input, expected,
        case(0.0, 0.0),
        case(1.0, 1.0),
        case(-1.0, -1.0),
        )]
    fn identity(input: f64, expected: f64) {
        assert_eq!(input, expected);
    }

it produces the following output:

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

The issue happens with or without nightly.
cargo 1.33.0 (f099fe94b 2019-02-12)
rustc 1.33.0 (2aa4c46cf 2019-02-28)

cargo 1.35.0-nightly (6f3e9c367 2019-04-04)
rustc 1.35.0-nightly (acd8dd6a5 2019-04-05)

I'm using rstest 0.2.2. From Cargo.lock:

"checksum rstest 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "17060b44b74f0aed4e7ee6c970e57b5e51adbd3aecd814e1ab38a27e00901d67"

Create a new crate for procedural macro

Investigate if rstest can be a non procedural-macro's crate that reexport the macros. So move macros in a new crate, reexport them and use rstest just for container crate (core logic if need) and reexport some base fixtures.

Support for error return type

By Rust 2018 Edition test functions can return a Result.

Wrong error message when missing function argument

Code like this:

#[rstest_parametrize(
init, cmd, expected,
case(Unwrap("RegValue::B(0xaf)"), Unwrap("Xra(Reg::B)")),
)]
fn should_always_reset_aux_carry<I: Apply>(mut cpu: Cpu, init: I, cmd: Instruction) {
    init.apply(&mut cpu);
    cpu.set_aux_carry_state(true);

    cpu.exec(cmd).unwrap();

    assert!(!cpu.state.flags.get(AuxCarry));
}

Generate follow error:

error: macros that expand to items must be delimited with braces or followed by a semicolon
    --> src/cpu/test.rs:1555:12
     |
1555 | init, cmd, expected,
     |            ^^^^^^^^
help: change the delimiters to curly braces
     |
1555 | init, cmd,  {xpecte},
     |             ^      ^
help: add a semicolon
     |
1555 | init, cmd, expected;,
     |                    ^

error: Missed argument: 'expected' should be a test function argument.
    --> src/cpu/test.rs:1555:12
     |
1555 | init, cmd, expected,
     |            ^^^^^^^^

error: aborting due to 2 previous errors

The last part is correct but the first one is confusing and should be removed.

Show a clear error if parametrize indicate a variable more than once

Follow code should raise a clear error

#[rstest_parametrize(a, a, case(42, 43))]
fn error_more_than_once(a: u32) {}

(expand case_with_wrong_args.rs)

Investigate if is possible to point at the first occurrence too.

Split rstest crate

Split monolithic crate in a crate to provide procedural macro and one for fixtures. The original crate will just reexport procedural macro in root and fixtures in fixtures module

Sub crate in workspace

• rstest_core procedural macros
• rstest_fixtures fixtures

fixture that return trait object

test it

Fixture as set of values

Investigate the possibility to define fixture that produce just a set of possible values. When we use these kind of fixture in rstest or rstest_parametrize we should produce a test case for each value.

Take care of

• Cartesian product of all input fixtures that produce a set of values
• Should wait depend fixture before compile (take a look to enum_dispatch crate )
• Circular reference can create dead locks

Tests errors fail on 1.33

Error message change in beta

rustc 1.33.0-beta.5 (1045131c1 2019-01-31)

did you mean fixture? become help: a function with a similar name exists: fixture`

Should report error for rstest_parametrize's invalid syntax

#[rstest_parametrize(datum, expected,
case("2018-12-29 18:15:33", NaiveDate::from_ymd(2018, 12, 29).and_hms(18, 15, 33)),
)]
fn t(datum: &str, expected:NaiveDate) {
    assert_eq!(1, 2, "Should not compile")
}

should report a some compile error

Custom errors are shown more than once

Maybe just the ones out of generated functions (like argument checking in rstest). That's because procedural macro output is compiled both in standard and in test and the early errors are not surrounded by #[cfg(test)].

Should be sufficient to annotate compiler errors (error_statement()) by #[cfg(test)].

Arbitrary rust code errors

Should span eventually error message at the code source.

Now for

#[rstest_parametrize(
condition,
case(r(r#"vec![1,5,7].contains(23)"#))
)]
fn arbitrary(condition: bool) {
    assert!(condition);
}

We receive

error[E0308]: mismatched types
  --> src/main.rs:7:1
   |
7  | / #[rstest_parametrize(
8  | | condition,
9  | | case(r(r#"vec![1,5,7].contains(23)"#))
10 | | )]
   | |__^ expected &{integer}, found integral variable
   |
   = note: expected type `&{integer}`
              found type `{integer}`
help: consider borrowing here
   |
7  | &#[rstest_parametrize(
8  | condition,
9  | case(r(r#"vec![1,5,7].contains(23)"#))
10 | )]
   |

Here is quite complicated to understand where the error is (should be &23) but in a more complicated example (more cases and wired code) should be real impossible.

The error should point at least to arbitrary code.

Question: why CaseArg::from(t).respan(a.span()) that we already use doesn't do the job?
Should we think to use quote_span! to redirect eventually errors at least to case line?

Accept arbitrary rust code as case argument

As #18 pointed out is too cleaver use Unwrap("some rust code"). We can do better and parse arbitrary rust code in procedural macro attributes

Case like follow should be parsed:

case(42, -42, pippo("pluto"), Vec::new(), String::from(r#"prrr"#), 
            {
                let mut sum=0;
                for i in 1..3 {
                    sum += i;
                }
                sum
            }, vec![1,2,3])

Clean generics in `fixture::default()`

In #5 we just cover the trivial cases where generics input are directly cited in output type.

Here we should cover all cases.

• in impl (maybe just test it)
• in dyn (maybe just test it)
• in trait associated type (maybe just test it)
• transitive annotate for default return type

The transitive case is the harder one. We should write the type's graph and check what input type is connected to output. Implement an annotation to define a default return type if need.

Better error message for missing fixture

Now we issue an error[E0433]: failed to resolve: use of undeclared type or module ...

Maybe we can use quote_span!{} in combination to check some fake field to report that the fixture function or annotation is missed.

Otherwise we can provide a good error by errors API at least in nightly

original fixture function should use `get()`

Replace the given function fixture(args) by call fixture::get(args).

Take care to remove mut in args definition (it will pass just the ownership) or , simpler, shut down the warning.

When we'll implement teardown we should take get() result and discard teardown