johnterickson/cargo2junit

Converts cargo's json output (from stdin) to JUnit XML (to stdout).

cargo2junit's Issues

Error on tracing::info! event in unit tests

Hi, I'm sorry if I got something wrong, but in my case I have an fn that sets up tracing subscriber and writes a message to log minimum example:

#[tokio::main]
async fn main() {
    println!("Hello, world!");
}

#[cfg(test)]
mod test {
    use std::str::FromStr;
    use tracing::info;
    use tracing_subscriber::fmt::layer;
    use tracing_subscriber::layer::SubscriberExt;
    use tracing_subscriber::util::SubscriberInitExt;
    use tracing_subscriber::{registry, EnvFilter, Layer};

    #[tokio::test]
    async fn set_up_subscriber() {
        let layer = layer()
            .json()
            .with_filter(EnvFilter::from_str("info").unwrap_or_default());
        registry().with(layer).try_init().unwrap();

        info!("Subscriber Initialised");
    }
}

unit testing it with:

cargo test | cargo2junit > junit-report.xml

make tests fail:

Error: Custom { kind: Other, error: "Error parsing '{\"timestamp\":\"2022-10-19T06:51:42.193267Z\",\"level\":\"INFO\",\"fields\":{\"message\":\"Subscriber Initialised\"},\"target\":\"test_cargo2junit::test\"}': data did not match any variant of untagged enum Event" }

Any suggestions on how to deal with it ? Thanks in advance.

JSON output format seems to have changed with Rust/Cargo 1.50

With Rust/Cargo 1.48 cargo2junit was working fine, but with version 1.50 we are getting this error:

Error: Custom { kind: Other, error: "Error parsing \'{ \"type\": \"test\", \"name\": \"our_crate::tests::test_something\", \"event\": \"ok\", \"exec_time\": 0.000252123 }\': invalid type: floating point `0.000252123`, expected a string" }

It seems that the JSON format has changed between Rust/Cargo 1.48 and 1.50. Previously it looked like this with exec_time being a string including a unit:

{ "type": "test", "name": "test_something", "event": "ok", "exec_time": "0.005s" }

But with 1.50 it is just a float:

{ "type": "test", "name": "test_something", "event": "ok", "exec_time": 0.001401918 }

Please tag releases

Hey, this git repo currently doesn't reflect your releases on crates.io but I think it'd make sense to publish git tags for every crates.io release you make. Could you publish git tags with new crates.io releases from now on?

Capability to sort by time?

It would be nice to have the output junit.xml sorted by (pass/fail, time). I typically want to know which ones failed, then which ones were the slowest.

cargo2junit panics because of a failed assertion

I'm seeing this error when trying to run my test output through cargo2junit

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `None`,
 right: `Some("domains::production_domain_update")`', /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cargo2junit-0.1.13/src/main.rs:211:21

Looks like this is the assertion is the one that is failing.

Breaking unit tests in test cases based on module.

I was wondering if it would be possible to break the unit tests into testsuite based on the module the tests reside in.

Given the following example of a current file:

 <?xml version="1.0" encoding="utf-8"?>
<testsuites>
  <testsuite id="0" name="darwin" package="darwin" tests="5" errors="0" failures="0" hostname="localhost" timestamp="2022-12-03T21:23:25.627281Z" time="0">
    <testcase name="test_one" time="0" classname="cli::test" />
    <testcase name="test_four" time="0" classname="data::tests" />
    <testcase name="test_two" time="0" classname="data::tests" />
    <testcase name="test_three" time="0" classname="cli::tests" />
  </testsuite>
</testsuites>

The following would be created

 <?xml version="1.0" encoding="utf-8"?>
<testsuites>
  <testsuite id="0" name="cli" package="testsuite/cli" tests="5" errors="0" failures="0" hostname="localhost" timestamp="2022-12-03T21:23:25.627281Z" time="0">
    <testcase name="test_one" time="0" classname="cli::test" />
    <testcase name="test_three" time="0" classname="cli::tests" />
  </testsuite>
  <testsuite id="1" name="data" package="testsuite/data" tests="5" errors="0" failures="0" hostname="localhost" timestamp="2022-12-03T21:23:25.627281Z" time="0">
    <testcase name="test_four" time="0" classname="data::tests" />
    <testcase name="test_two" time="0" classname="data::tests" />
  </testsuite>
</testsuites>

This would help break things in some CI systems, like Jenkins, into more fragmented chunks; imho.

No newline at the end of the XML output

I know that the CI tools couldn't care less, but it's somewhat inconvenient when printing to stdout, and git will complain if someone wants to check in such a report.

Breaking change in v0.1.12

v0.1.12 seems to have made a breaking change:

$ cargo install cargo2junit
    Updating crates.io index
 Downloading crates ...
  Downloaded cargo2junit v0.1.12
  Installing cargo2junit v0.1.12
 Downloading crates ...
  Downloaded vte_generate_state_changes v0.1.1
  Downloaded time-macros v0.2.4
error: failed to compile `cargo2junit v0.1.12`, intermediate artifacts can be found at `/tmp/cargo-install3Oi3Lw`
Caused by:
  failed to parse manifest at `/root/.cargo/registry/src/github.com-1ecc6299db9ec823/time-macros-0.2.4/Cargo.toml`
Caused by:
  feature `resolver` is required
  this Cargo does not support nightly features, but if you
  switch to nightly channel you can add
  `cargo-features = ["resolver"]` to enable this feature

There already is a cargo2junit...

Hadn't seen it on crates.io, but just found https://github.com/mgr-inz-rafal/cargo2junit while searching for my own project. 🤦‍♂️

Add execution time support

Recently, Rust has merged a --report-time flag (rust-lang/rust#64663) into the test library so that the execution time is printed. I think maybe cargo2unit can support this feature.

Can't navigate test cases in Jenkins

#30 (comment)

Could you set the classname attribute only when it is available or maybe to a default value like default?
Having only src/main.rs and src/lib.rs without further modules there is no path:: prefix in the JSON output.
This results in classname="" in the XML output which prevents navigating the test cases in Jenkins.

cargo2junit will stop working with Rust 1.70

The Rust libs team quietly decided to start blocking -Zunstable-options in Rust 1.70 outside of nightly builds. Rust 1.70 is now in beta.

rust-lang/rust#109044

$ cargo +beta test -- -Zunstable-options --format json
    Finished test [unoptimized + debuginfo] target(s) in 0.22s
     Running unittests src/lib.rs ([removed])
error: the option `Z` is only accepted on the nightly compiler
error: test failed, to rerun pass `--lib`

Arguably cargo2unit users were playing with fire by taking a dependency on this... but nonetheless this will strand users on 1.69 who depend on JSON output in their CI builds.

Luckily, it seems that setting RUSTC_BOOTSTRAP=1 in the environment before launching the test will work around this. This seems strictly worse than the status quo since it allows other Rust unstable features to be used as well. Oh well.

Perhaps the README should be updated to reflect this.

Use Pipeline Caching in the build

I'm trying to use my little tool as a testbed for using Pipeline Caching for building Rust projects. I hope to standardize a recommendation.

My general approach is:

• Cache is per config (e.g. x86_64-pc-windows-msvc+vs2017-win2016)
• Cache contains cargo+rustc toolchain
• Cache contains sccache local folder
• Cache contains other "cargo install"ed tools that the build itself uses (e.g. cargo2junit)
• Cache contains tools to make caching work (sccache and cargo-cache)
• Cache contains mechanism to update tools (cargo-update)

Some questions?

• Is sccache ready for general recommendation? Any gotchas?
• Will new versions of sccache/cargo-cache handle a cache folder from an older version well?
• What are the right parameters for cargo-cache so the cache doesn't grow indefinitely?
• Will perf be better even though we're doing all this? 😁

Here's an example build - just look at VS2019-tar and OSX-tar:
https://dev.azure.com/codesharing-su0/cachesandbox/_build/results?buildId=15929&view=results
and it's YAML:
https://github.com/johnterickson/cargo2junit/blob/cacheTar/azure-pipelines.yml

Output goes into the wrong tag

To me it looks like the output (standard output/error) goes into the wrong attribute/tag (message).

Taking a look at some documentation: https://llg.cubic.org/docs/junit/ … there seems to be a system-out, which seems like a better fit.

Writing the output to this tag would improve compatibility with other tooling: https://github.com/inorton/junit2html/blob/248e636278635254d97abcd5d153b08a4254cfbe/junit2htmlreport/templates/report.html#L108-L110

Parse error on windows

I got the following error on windows:

Error: Custom { kind: Other, error: StringError("Error parsing \'{ \"type\": \"test\", \"event\": \"started\", \"name\": \"src\\lib.rs -  (line 17)\" }\': invalid escape at line 1 column 52") }

This might be a serde/json error (seems like it), but it works on the other OSes, so I'm guessing it's OSString or newline related. Maybe it would be possible to normalize the input before JSON parsing it?

Handle ignores

Beyond parsing, check contents of report in tests

Problem with install

OS: Ubuntu 22.04

$ cargo install --locked cargo2junit
    Updating crates.io index
 Downloading crates ...
  Downloaded cargo2junit v0.1.13
  Installing cargo2junit v0.1.13
error: failed to select a version for the requirement `time = "=0.3.20"`
candidate versions found which didn't match: 0.3.15, 0.3.14, 0.3.13, ...
location searched: crates.io index
required by package `junit-report v0.7.1`
    ... which is depended on by `cargo2junit v0.1.13`

Any solutions?

Precompiled binaries

It'd be nice if there were pre-compiled binaries for use in CI pipelines (instead of building it manually).

Propagate exit code

Hi! Thanks for publishing, this package is really useful.
I have one more use case in mind. Is it possible to return non-zero exit code if any test fails?

Enhancement: Add support for benches

Example

[user@localhost my-package]$ cargo bench  -- -Z unstable-options --format json | cargo2junit > test-results.xml
    Finished release [optimized] target(s) in 0.07s
     Running target/release/deps/mylib-0841b1c2beb997da
     Running target/release/deps/mybin-51a51b3413cebaff
     Running target/release/deps/01_bench-0a604b4c894de629
Error: Custom { kind: Other, error: StringError("Error parsing \'{ \"type\": \"bench\", \"name\": \"test_status_code_ok\", \"median\": 1604, \"deviation\": 459 }\': unknown variant `bench`, expected `suite` or `test` at line 1 column 17") }

Include timings

Missing tags?

Hello!

I saw that there are a couple of commits that bump the version but there isn't any corresponding tags created. For example:

• 6a80c50 (0.1.12)
• f57a153 (0.1.9)

I was wondering if you are going to create and push tags so that the latest revision of the source code can be downloaded from GitHub?

cargo2junit returns error if no tests are found

This isn't how cargo test behaves. Having no tests is considered a success and it returns 0.

This is an issue for us because we have shared Rust CI code which runs tests, and a small number of our repos have no tests. That works fine for the stage that runs cargo test today, but if we change that to add ... | cargo2junit (so test results appear in ADO) then all those repositories have their test stages start failing.

Also report file names

It would be neat if this tool could also extract the filenames of the thing that's being tested (in the case of integration tests). For example:

     Running unittests (target/debug/deps/generate_openapi_spec-e3bc6e08330e8181)
{ "type": "suite", "event": "started", "test_count": 0 }
{ "type": "suite", "event": "ok", "passed": 0, "failed": 0, "allowed_fail": 0, "ignored": 0, "measured": 0, "filtered_out": 0, "exec_time": 0.000019477 }
     Running tests/test_api_openapi.rs (target/debug/deps/test_api_openapi-53c722415f8c53df)
{ "type": "suite", "event": "started", "test_count": 1 }
{ "type": "test", "event": "started", "name": "read::can_read" }
{ "type": "test", "name": "read::can_read", "event": "ok" }
{ "type": "suite", "event": "ok", "passed": 1, "failed": 0, "allowed_fail": 0, "ignored": 0, "measured": 0, "filtered_out": 0, "exec_time": 0.06092743 }

As we can see, we can extract the filenames from that text. These would then end up in <testcase file=tests/test_api_openapi.rs/> which I think would be quite neat. GitLab can actually display those: https://gitlab.com/gitlab-org/gitlab/-/issues/119032