unickq/allure-nunit

Archived - Allure adapter for NUnit framework.

Hi, I'm Nick 👋

Software Engineer in Test, who makes automated tests, scrapers, bots, and tunes CI and digs into logs to find the truth 🙌

allure-nunit's Issues

Test name is created as a step and steps inside the tests are substeps

Test name is created as a step and steps inside the tests are substeps .

Ideally , test name should not be a test step .

Awaiting your reply

"System.MissingMethodException" error appears after update NUnit from 3.13.0 to 3.13.1 version

"System.MissingMethodException : Method not found: 'System.Reflection.MethodInfo[] NUnit.Framework.Internal.TestSuite.get_OneTimeTearDownMethods()'." error appears after update of NUnit nuget package from 3.13.0 to 3.13.1 version.

Stack Trace:
AllureNUnitHelper.GetNUnitHelpMethods(NUnitHelpMethodType type, TestFixture fixture)
AllureNUnitHelper.BuildFixtureResults(NUnitHelpMethodType type, TestFixture testFixture)
AllureNUnitHelper.StopAll(Boolean isWrappedIntoStep)
AllureNUnitAttribute.AfterTest(ITest test)
<.ctor>b__1(TestExecutionContext context)
<>c__DisplayClass1_0.b__1()
BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)

The error appears after the execution of a test case. The error disappears after rollback to the 3.13.0 version.

Answer from NUnit team (nunit/nunit#3766)
I guess the problem is that the properties have changed type from MethodInfo to IMethodInfo in #3701, but allure-nunit is compiled again version 3.12.0 (thus matching the types in 3.12.0) - https://github.com/unickq/allure-nunit/blob/15b13970770405c6996e7cb781846e169443386b/src/allure-nunit/allure-nunit.csproj

I think the best solution for now will be to use the same version that allure-nunit references - meaning 3.12.0. You could ask them to bump their dependencies, but I guess this could give problems to users using allure-nunit and older versions of nunit e.g. 3.12.0

Basic usage - How to generate Allure results

I've tried running tests with NUnit.ConsoleRunner but allure-results directory is missing and the TestResults.xml it generates doesn't contain Allure specific attributes.

What are the steps required to generate Allure results?

Thanks!

Could not install package 'NUnit.Allure 1.0.14'

I am using Visual Studio 2017 .net framework 4.8.

< TargetFrameworkVersion >v4.5.2< /TargetFrameworkVersion>

When I install the nuget package Nunit.Allure Then I get error

Severity Code Description Project File Line Suppression State
Error Could not install package 'NUnit.Allure 1.0.14'. You are trying to install this package into a project that targets '.NETFramework,Version=v4.5.2', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author.

I am trying to get a report out of automated api tests using - nunit , c# , restsharp

When I read about Allure it is exactly what am looking for. But am struck on the installation step. Can anyone please help ?

Attribute AllureDisplayIgnored doesn't work with ignoring testcases

Hello!

If I use ignore attribute with testcases like this:
[TestCase("03/03/20166"), Ignore("PRX-1000")]
test doesn't exist in allure report.

If test contains await, then [AllureStep] failed with System.ArgumentNullException : Value cannot be null. (Parameter 'key')

This code given exception:
[Test]
public async Task Test()
{
await Task.Delay(1000);
Hello();
}

[AllureStep]
public void Hello()
{
}

System.ArgumentNullException : Value cannot be null. (Parameter 'key')
Stack Trace:
ConcurrentDictionary2.ThrowKeyNullException() ConcurrentDictionary2.TryGetValue(TKey key, TValue& value)
ConcurrentDictionary2.get_Item(TKey key) AllureStorage.Get[T](String uuid) AllureStorage.AddStep(String parentUuid, String uuid, StepResult stepResult) AllureLifecycle.StartStep(String parentUuid, String uuid, StepResult stepResult) AllureLifecycle.StartStep(String uuid, StepResult result) AllureStepAspect.WrapStep(String name, MethodBase methodBase, Object[] arguments, Func2 method)
Tests.__a$_around_Hello_100663386_w_0(Object[] )
Tests.Hello()

But this code - successful:
[Test]
public async Task Test()
{
Task.Delay(1000).GetAwaiter().GetResult();
Hello();
}

How to add a screenshot to each test step

I wanted to add a screenshot to each test step in allure report, is it possible ?

[AllureStep] not working in NUnit 3 [OneTimeSetUp] method

I have OneTimeSetUp method in test suite:

[OneTimeSetUp]
public void OneTimeSetUp()
{
    var page = new Page(Driver);
    page.GetPage();
}

And following page method:

[AllureStep("Get page")]
public void GetPage()
{
    Driver.Navigate().GoToUrl("https://www.google.com/");
}

When I run this suite with nunit, I receive "System.ArgumentNullException" at page.GetPage();
It seems like a bug.

Project dependencies:
NUnit (3.12.0)
NUnit.Allure (1.0.6.2)
Allure.Commons (>= 2.3.64.1)
NUnit (>= 3.10.1)
NUnit.Console (3.10.0)
NUnit.ConsoleRunner (>= 3.10)
NUnit.Extension.NUnitProjectLoader (>= 3.6)
NUnit.Extension.NUnitV2Driver (>= 3.7)
NUnit.Extension.NUnitV2ResultWriter (>= 3.6)
NUnit.Extension.TeamCityEventListener (>= 1.0.6)
NUnit.Extension.VSProjectLoader (>= 3.8)
NUnit.Extension.NUnitProjectLoader (3.6.0)
NUnit.Extension.NUnitV2ResultWriter (3.6.0)
NUnit.Allure.Steps (1.0.7.1)
AspectInjector (>= 2.2.1)
NUnit.Allure (>= 1.0.6)
NUnit.ConsoleRunner (3.10)
NUnit.Extension.NUnitV2Driver (3.8)
NUnit.Extension.TeamCityEventListener (1.0.7)
NUnit.Extension.VSProjectLoader (3.8)

Edit and continue does not work due to usage of AspectInjector 2.6

The AllureStepAttribute is using AspectInjector 2.6 which does not allow edit and continue while debugging in VisualStudio, this mak developing code much more difficult.

See gif of issue here:
Recording

Test yourself with this code: EditAndContinueIssue.zip

Currently I fixed this issue in my code by copying the code of the AllureStepAttribute into my project and using AspectInjector 2.7 library.

This issue would be fixed by updating allure-nunit to use the latest AspectInjector.

TestFixture without correct signature (params) throws an exception

v1.0.1

[TestFixture("Wwwww")]

throws

No suitable constructor was found. Exception doesn't have a stacktrace

Not getting any results output to allure-results folder

Hi, I've copied your sample code exactly into a new project and run the tests with both the Nunit console runner and from VisualStudio with Resharper, but I am not getting any allure results. Can you me? Thanks

StepWrapper is not working after update

After the update to 1.0.0, my entire implementation of allure is broken.

The very first thing is the way the instance of allure lifecycle is working. We have a custom method for the start and stop steps:

public string StartStep(string name) { var x = TestContext.CurrentContext.Result; string uuid = Guid.NewGuid().ToString(); StepResult step = new StepResult() { name = name, stage = Stage.finished, status = Status.passed }; AllureLifecycle.Instance.StartStep(uuid, step); return uuid; }

This is throwing exceptions. We try to use the new stepWrapper AllureNUnitHelper.WrapInStep(() => action) but is throwing a System.ArgumentNullException.

Is there a way to fix this without refactoring the whole solution? (I have currently more than 600 test cases and we really want to avoid a mayor refactor in our sln)

Thanks in advance

Cannot find "AllureNUnit" attribute after update of from

Error CS0246 The type or namespace name 'AllureNUnitAttribute' could not be found (are you missing a using directive or an assembly reference?)

TestCase attribute aint working without "Category"

Hello!
After i've updated allure-nunit to 1.0.0 - i found some strange thing.
Then im running my tests with resharper - everything went good, but then im using console runner - i got

Error : APITests.StorageAndCacheTests.Tests.test
System.ArgumentOutOfRangeException : Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
at System.Collections.Generic.List1.System.Collections.IList.get_Item(Int32 index) at NUnit.Allure.Core.AllureNUnitHelper.<>c__DisplayClass21_2.<UpdateTestDataFromAttributes>b__2(TestResult x) at Allure.Commons.AllureLifecycle.UpdateTestCase(String uuid, Action1 update)
at Allure.Commons.AllureLifecycle.UpdateTestCase(Action`1 update)
at NUnit.Allure.Core.AllureNUnitHelper.UpdateTestDataFromAttributes()
at NUnit.Allure.Core.AllureNUnitHelper.StopTestCase()
at NUnit.Allure.Core.AllureNUnitHelper.StopAll(Boolean isWrapedIntoStep)
at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.Execute(TestExecutionContext context)
at NUnit.Framework.Internal.Execution.SimpleWorkItem.PerformWork()

.

Found a way to reproduce this -

[Category("Debug")]
[Test(Description = "test")]
[TestCase("1", TestName = "test")]
public void Test(string num)
  {
            Assert.AreEqual("1", num);
        
   }
   <- this test wont work

[Category("Debug")]
        [Test(Description = "test")]
        [TestCase("1", TestName = "test", Category = "Debug")]
        public void Test(string num)
        {
            Assert.AreEqual("1", num);
        }

 <- this will be ok.

Can you fix this? Or i should add this param for each one of my testcases?

i think error somethere around

if (_test.Properties.ContainsKey("Category"))
{
var list = _test.Properties["Category"];
AllureLifecycle.UpdateTestCase(x => x.labels.Add(Label.Tag(list[0].ToString())));
}
in private void UpdateTestDataFromAttributes()

Best regards,
Dmitry

How to enable adding Screen Diff

Thanks for the quick implementation, but I have a question.
I created a test, marked it with [AllureLabel("testType", "screenshotDiff")], but I don't have access to compare? It won't work?
I expect to see it

But I se

Steps do not show in Setup

Noticed that when using steps on the setup they are not showing up on the setup part of the logs. see image bellow.

we can have allure steps running on the nunit setup, i was just expecting them to show on the setup part of the logs. Am i doing something wrong or is this the way it works?

code:

`using NUnit.Allure.Attributes;
using NUnit.Allure.Core;
using NUnit.Framework;

namespace SeleniumTests.tests
{

[TestFixture]
[AllureNUnit]
[AllureSuite("experiment")]
internal class TestExperiment
{
    
    [SetUp]
    public void SetUp()
    {
        login();
    }

    [AllureStep("login")]
    private void login()
    {
        TestContext.WriteLine("login");
    }

    [AllureStep("test action")]
    private void action()
    {
        TestContext.WriteLine("action");
    }

    [Test]
    [AllureName("test1")]
    public void Test1()
    {
        action();
    }
}

}`

nuget:
<ItemGroup> <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.11.0" /> <PackageReference Include="NUnit" Version="3.13.2" /> <PackageReference Include="NUnit.Allure" Version="1.2.1.1" /> <PackageReference Include="NUnit3TestAdapter" Version="4.0.0" /> <PackageReference Include="coverlet.collector" Version="3.1.0" /> </ItemGroup>

Cannot find "AllureNUnit" attribute after update of "NUnite.Allure" nuget package from 1.0.14 to 1.2.0

Cannot find "AllureNUnit" attribute after update of "NUnite.Allure" nuget package from 1.0.14 to 1.2.0
Error CS0234 The type or namespace name 'Core' does not exist in the namespace 'NUnit.Allure' (are you missing an assembly reference?)
Error CS0246 The type or namespace name 'AllureNUnitAttribute' could not be found (are you missing a using directive or an assembly reference?)

NullReferenceException in AllureStepAspect.WrapStep when some of the arguments are null

Hi, seems like there is a problem in WrapStep method in line 23
stepName = stepName?.Replace("{" + i + "}", arguments[i].ToString());
Possible solution is to check argument for null and replace it with something like "{null}"

Thanks in advance

Update for NET4.5

• Merge allure-framework/allure-csharp#11
• Update plugin to the latest Allure.Commons with 4.5 support

Add ability to add attachment

It's not currently possible to add an attachment. It happens because Allure.WriteTestCase event happens before TearDown method

[SetUp]
  ITestAction.BeforeTest
     [Test]
  ITestAction.AfterTest
[TearDown]

Changing the version of an assembly reference is not allowed during debugging

Hi!
My project: Net.Core, NUnit.
Adding this link to the project makes debugging impossible. That is, literally adding the line
var qwe = true;
anywhere results in the error:
Changing the version of an assembly reference is not allowed during debugging: 'netstandard, Version = 2.0.0.0, Culture = neutral, PublicKeyToken = cc7b13ffcd2ddd51' changed version to '2.1.0.0.

Maybe NUnit.Allure.Steps internally links to https://www.nuget.org/packages/AspectInjector/
and it's trying to generate and rebuild some assemblies in memory at compile time?

Archive repo

I guess it's time to archive this repo and add redirect to https://github.com/allure-framework/allure-csharp in the readme?

MaxLimit attribute doesn't logged in allure

When test failed by MaxLimit attribute, allure does not logging that, it is marked like passed

No report if OneTimeSetUp is failed

Great job!

But is any ideas, how to bypass error in OneTimeSetUp method for fixture?
This test will not be included in report, because ITestAction attribute will be applied after OneTimeSetUp

    public class Class3
    {
        [OneTimeSetUp]
        public void ThisWillFail()
        {
            var x = 0;
            var y = 1 / x;
        }

        [Test]
        [AllureTest]
        public void Xxx4()
        {
            Assert.True(1 == 2);
        }
    }

When the allure result is generated, the "AllureLifecycle.Instance.AllureConfiguration.Directory" property is ignored

I'm trying to configure different allure result directories depending on the configuration being run. I have the following code:

private void AllureConfig()
        {
            string resultPath = "C:\\ProgramData\\BaseTrade\\TestsMetaData\\OtcMain_stable";
            AllureLifecycle.Instance.AllureConfiguration.Directory = resultPath;
        }

and standart json config:

{
  "allure": {
    "directory": "allure-results",
    "links": [
      "https://example.org/{issue}",
      "https://example.org/{tms}"
    ]
  }
}

But when generating allure result, the output directory is always allure-results.

Allure.Commons 2.3.67.1
NUnit.Allure 1.0.0

AspectInjector.targets error in Rider

Hello!
Can't build solution in rider using DotNet SDK Builder 6.0.1, due to this error

AspectInjector.targets(93, 5): [MSB4062] The "Microsoft.Build.Tasks.GetFrameworkSdkPath" task could not be loaded from the assembly Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a. Confirm that the declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask.

.NET SDK 6.0.101

Rider version Build #RD-213.6461.51, built on December 22, 2021

Value cannot be null

hi,

when i ran my tests, they pass but the console writes down the following:
System.ArgumentNullException : Value cannot be null.
Parameter name: key
at System.Collections.Concurrent.ConcurrentDictionary2.TryGetValue(TKey key, TValue& value) at System.Collections.Concurrent.ConcurrentDictionary2.get_Item(TKey key)
at Allure.Commons.Storage.AllureStorage.Get[T](String uuid)
at Allure.Commons.AllureLifecycle.UpdateStep(Action1 update) at Allure.Commons.AllureLifecycle.StopStep(Action1 beforeStop)
at NUnit.Allure.Core.AllureNUnitHelper.StopAll(Boolean isWrappedIntoStep)
at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action) in C:\src\nunit\nunit\src\NUnitFramework\framework\Internal\Commands\BeforeAndAfterTestCommand.cs:line 73

Did i miss anything with the setup? It's spitting our the allure-results (json files) and the html in allure-reports, but weird the nunit console is giving me that error???

Get test result id

Hey! I have a need for the AddScreenDiff method, but it requires the uuid property of the TestResult class. Can you implement an open method or open the _testResultGuid property for reading?

Add support to Include test class name in the report

Currently the report just shows the test method name, I think it would be usefull to have also de testclass name.

Fix Retry()

Retry generates many tests.
Use historyId

Cyrrilic text in attached console output in wrong encoding

Nuget packages used:
NUnit 3.10.1
NUnit.Allure 1.0.5
Allure.Commons 2.3.64.1

Example code:

using System;
using NUnit.Framework;
using NUnit.Allure.Core;

namespace AllureTest
{
    [AllureNUnit]
    [TestFixture()]
    public static class Test
    {
        [Test()]
        public static void TestCase()
        {
            Console.WriteLine("ПРОВЕРКА");
            Assert.Pass();
        }
    }
}

In console output attachment (inside allure-reports directory) writes "????????" instead of cyrillic text.

Dependency conflict

Allure adapter requires .NET framework 4.6.1 and Allure.Commons which requires Microsoft.Extensions.Configuration (>= 2.0.0) but Microsoft.Extensions.Configuration has no binaries for .net 4.6.1 and rejects installation (i tried via NuGet)

Is there exists another way to install allure adapter?

• Allure adapter version: 0.2.1

NullReferenceException exception when running AllureNUnit

Hi,

I got the following stack trace:

Test Name:	Test2
Test FullName:	TestClass7.Test2
Test Source:	C:\Sources\AllureTest\AllureTest\AllureTest\UnitTest1.cs : line 26
Test Outcome:	Failed
Test Duration:	0:00:00.001

Result StackTrace:	
at NUnit.Allure.Core.AllureNUnitHelper.GetTestProperties(String name)
   at NUnit.Allure.Core.AllureNUnitHelper.UpdateTestDataFromAttributes()
   at NUnit.Allure.Core.AllureNUnitHelper.StopTestCase()
   at NUnit.Allure.Core.AllureNUnitHelper.StopAll(Boolean isWrappedIntoStep)
   at NUnit.Allure.Core.AllureNUnitAttribute.AfterTest(ITest test)
   at NUnit.Framework.Internal.Commands.TestActionCommand.<>c__DisplayClass0_0.<.ctor>b__1(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__1()
   at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)
Result Message:	System.NullReferenceException : Object reference not set to an instance of an object.

For the following code:

using NUnit.Allure.Core;
using NUnit.Framework;

[AllureNUnit]
[TestFixture(Category = "asd")]
public class TestClass7
{
    [Test(Description = "test")]
    [Category("Debug")]
    public void Test1()
    {
        Assert.AreEqual(1, 1);
    }

    [Category("Debug")]
    [Test(Description = "test")]
    public void Test2()
    {
        Assert.AreEqual(2, 2);
    }
}

I've installed the following NuGet in advance:

• Microsoft.NET.Test.Sdk
• NUnit
• NUnit3TestAdapter
• NUnit.Allure

and used scoop to install allure using the following cmd command:
scoop install allure

May you please assist?

Can't use StartStep and StopStep in TearDown and SetUp methods

Got exceptions:
For TearDown
Message: TearDown : System.ArgumentNullException : Value cannot be null.
Parameter name: key

For SetUp:
Message: System.ArgumentNullException : Value cannot be null.
Parameter name: key

I use NUnit 3.9.0

Could I configure path to allureConfig.json

I switched my project to NUnit.Allure. Before that my allureConfig.json file was located in project's root folder.
Now when i'm trying to run tests, I've got error "could not find file allureconfig.json", because it should be in the allure.coomons package folder. Is there a way to configure nunit to look for a allureConfig.json in the project folder?

NUnit [Theory] doesn't seem to work.

My tests are running but I'm not getting results in my json for tests methods with with [Theory] attributes instead of [Test] attributes. Not sure if this is bug or feature request but parameterized tests are essential these days. Would be nice if this library could support that.

Display OneTimeSetUp in report

Hello!

What you can say about showing up OneTimeSetup in report (I mean display OTS in test case near simple SetUp like at image below)

Also in current realization there is no link between test cases and OTS methods and if OTS fails it doesn't show at a report

Overview Environment

Hi Nick,
This is more of a question than issue...
In the overview section of the report, how can I pull the correct environment from a config file?
Say I have the following settings:

Then on next run I have iOS, etc.
How do I set that in the Environments section so they're grouped together like
Samsung Galaxy S6, Android OS 5.0
iPhone XS, iOS 10.0
.
..
...

Value cannot be null. (Parameter 'key') async Task

Hey! Error The value cannot be null. (Parameter key) if the test method is asynchronous

Attribute cannot be null in parametrized tests

allure-nunit/src/Core/AllureNUnitHelper.cs

Line 239 in 351e31e

value = _test.Arguments[i].ToString()

this._test.Arguments[i].toString() throws System.NullReferenceException if this._test.Arguments[i] == null

For example:
Code of test to reproduce:

    [Test, Combinatorial]
    [AllureStory("Example")]
    public void TestExample(
        [Range(1, 3)] int valA,
        [Values((object)null, 2)] int? valB
        )
    {
            Assert.Equals(valA, valB);
    }

Is it possible to add the method's parameters to report with the help of AllureStep?

        [AllureStep("I login with email = {email} and  pass {pass} ")]
        public BackgroundChecksSteps Login(string email, string pass)
        {
               new LoginPage().login(email, pass);
        }