modulexcite/verify

Verification tool to enable simple approval of complex models and documents.

Verify

Verification tool to enable simple approval of complex models and documents.

Support is available via a Tidelift Subscription.

Contents

• Verification versus Assertion
• Usage
  • Class being tested
  • xUnit
  • NUnit
  • MSTest
  • bunit
  • Initial Verification
  • Subsequent Verification
  • Disable Clipboard
  • AutoVerify
  • OnHandlers
• Received and Verified
• Not valid json
• Videos
• Extensions
• Alternatives
• Security contact information
• Serializer Settings
• File naming
• Parameterised tests
• Named Tuples
• Scrubbers
• Diff Engine
• Diff Tools
• Diff Tool Order
• Custom Diff Tool
• Using anonymous types
• Verifying binary data
• Comparers
• Compared to ApprovalTests

NuGet packages

• https://nuget.org/packages/Verify.Xunit/
• https://nuget.org/packages/Verify.NUnit/
• https://nuget.org/packages/Verify.MSTest/
• https://nuget.org/packages/Verify.Bunit/

Verification versus Assertion

Given the following method:

Class being tested

public static class ClassBeingTested
{
    public static Person FindPerson()
    {
        return new Person
        {
            Id = new Guid("ebced679-45d3-4653-8791-3d969c4a986c"),
            Title = Title.Mr,
            GivenNames = "John",
            FamilyName = "Smith",
            Spouse = "Jill",
            Children = new List<string>
            {
                "Sam",
                "Mary"
            },
            Address = new Address
            {
                Street = "4 Puddle Lane",
                Country = "USA"
            }
        };
    }
}

^{snippet source | anchor}

Tests

Compare a traditional assertion based test to a verification test.

Traditional assertion test:

[Fact]
public void TraditionalTest()
{
    var person = ClassBeingTested.FindPerson();
    Assert.Equal(new Guid("ebced679-45d3-4653-8791-3d969c4a986c"), person.Id);
    Assert.Equal(Title.Mr, person.Title);
    Assert.Equal("John", person.GivenNames);
    Assert.Equal("Smith", person.FamilyName);
    Assert.Equal("Jill", person.Spouse);
    Assert.Equal(2, person.Children.Count);
    Assert.Equal("Sam", person.Children[0]);
    Assert.Equal("Mary", person.Children[1]);
    Assert.Equal("4 Puddle Lane", person.Address.Street);
    Assert.Equal("USA", person.Address.Country);
}

^{snippet source | anchor}

Verification test

[Fact]
public Task Simple()
{
    var person = ClassBeingTested.FindPerson();
    return Verify(person);
}

^{snippet source | anchor}

Comparing Verification to Assertion

• Less test code: verification test require less code to write.
• Reduced risk of incorrect test code: Given the above assertion based test it would be difficult to ensure that no property is missing from the assertion. For example if a new property is added to the model. In the verification test that change would automatically be highlighted when the test is next run.
• Test failure visualization: Verification test allows visualization in a diff tool that works for complex models and binary documents.
• Multiple changes visualized in singe test run: In the assertion approach, if multiple assertions require changing, this only becomes apparent over multiple test runs. In the verification approach, multiple changes can be visualized in one test run.
• Simpler creation of test "contract": In the assertion approach, complex models can require significant code to do the initial assertion. In the verification approach, the actual test and code-under-test can be used to create that "contract". See initial verification.
• Verification files committed to source control: All resulting verified files are committed to source control in the most appropriate format. This means these files can be viewed at any time using any tooling. The files can also be diff'd over the history of the code base. This works for any file type, for example:
  • Html content can be committed as .html files.
  • Office documents can be committed as a rendered .png (see Verify.Aspose).
  • Database schema can be committed as .sql (see Verify.SqlServer).

Usage

Class being tested

Given a class to be tested:

public static class ClassBeingTested
{
    public static Person FindPerson()
    {
        return new Person
        {
            Id = new Guid("ebced679-45d3-4653-8791-3d969c4a986c"),
            Title = Title.Mr,
            GivenNames = "John",
            FamilyName = "Smith",
            Spouse = "Jill",
            Children = new List<string>
            {
                "Sam",
                "Mary"
            },
            Address = new Address
            {
                Street = "4 Puddle Lane",
                Country = "USA"
            }
        };
    }
}

^{snippet source | anchor}

xUnit

Support for xUnit

public class SampleTest :
    VerifyBase
{
    [Fact]
    public Task Simple()
    {
        var person = ClassBeingTested.FindPerson();
        return Verify(person);
    }

    public SampleTest(ITestOutputHelper output) :
        base(output)
    {
    }
}

^{snippet source | anchor}

• Verify Xunit Intro

NUnit

Support for NUnit

[TestFixture]
public class SampleTest
{
    [Test]
    public Task Simple()
    {
        var person = ClassBeingTested.FindPerson();
        return Verifier.Verify(person);
    }
}

^{snippet source | anchor}

MSTest

Support for MSTest

[TestClass]
public class SampleTest :
    VerifyBase
{
    [TestMethod]
    public Task Simple()
    {
        var person = ClassBeingTested.FindPerson();
        return Verify(person);
    }
}

^{snippet source | anchor}

bunit

Support for rendering a Blazor Component to a verified file via bunit.

Component test

Given the following Component:

<div>
    <h1>@Title</h1>
    <button>MyButton</button>
</div>

@code {
    [Parameter]
    public string Title { get; set; } = "My Test Component";
}

^{snippet source | anchor}

This test:

[Fact]
public Task Component()
{
    var component = RenderComponent<TestComponent>();
    return Verify(component);
}

^{snippet source | anchor}

Will produce:

The component rendered as html ...Component.verified.html:

<div>
    <h1>My Test Component</h1>
    <button>MyButton</button>
</div>

^{snippet source | anchor}

And the current model rendered as txt ...Component.info.verified.txt:

{
  Instance: {
    Title: 'My Test Component'
  }
}

^{snippet source | anchor}

Initial Verification

When the test is initially run will fail with:

First verification. SampleTest.Simple.verified.txt not found.
Verification command has been copied to the clipboard.

The clipboard will contain the following:

move /Y "C:\Code\Sample\SampleTest.Simple.received.txt" "C:\Code\Sample\SampleTest.Simple.verified.txt"

If a Diff Tool is detected it will display the diff:

To verify the result:

• Execute the command from the clipboard, or
• Use the diff tool to accept the changes, or
• Manually copy the text to the new file

This will result in the SampleTest.Simple.verified.txt being created:

{
  GivenNames: 'John',
  FamilyName: 'Smith',
  Spouse: 'Jill',
  Address: {
    Street: '4 Puddle Lane',
    Country: 'USA'
  },
  Children: [
    'Sam',
    'Mary'
  ],
  Id: Guid_1
}

^{snippet source | anchor}

Subsequent Verification

If the implementation of ClassBeingTested changes:

public static class ClassBeingTested
{
    public static Person FindPerson()
    {
        return new Person
        {
            Id = new Guid("ebced679-45d3-4653-8791-3d969c4a986c"),
            Title = Title.Mr,
            // Middle name added
            GivenNames = "John James",
            FamilyName = "Smith",
            Spouse = "Jill",
            Children = new List<string>
            {
                "Sam",
                "Mary"
            },
            Address = new Address
            {
                // Address changed
                Street = "64 Barnett Street",
                Country = "USA"
            }
        };
    }
}

^{snippet source | anchor}

And the test is re run it will fail with

Verification command has been copied to the clipboard.
Assert.Equal() Failure
                                  ↓ (pos 21)
Expected: ···\n  GivenNames: 'John',\n  FamilyName: 'Smith',\n  Spouse: 'Jill···
Actual:   ···\n  GivenNames: 'John James',\n  FamilyName: 'Smith',\n  Spouse:···
                                  ↑ (pos 21)

The clipboard will again contain the following:

move /Y "C:\Code\Sample\SampleTest.Simple.received.txt" "C:\Code\Sample\SampleTest.Simple.verified.txt"

And the Diff Tool is will display the diff:

The same approach can be used to verify the results and the change to SampleTest.Simple.verified.txt is committed to source control along with the change to ClassBeingTested.

Disable Clipboard

The clipboard behavior can be disable using the following:

Per Test

var settings = new VerifySettings();
settings.DisableClipboard();

^{snippet source | anchor}

For all tests

SharedVerifySettings.DisableClipboard();

^{snippet source | anchor}

If clipboard is disabled for all tests, it can be re-enabled at the test level:

var settings = new VerifySettings();
settings.EnableClipboard();

^{snippet source | anchor}

For a machine

Set a Verify.DisableClipboard environment variable to true. This overrides the above settings.

AutoVerify

In some scenarios it makes sense to auto-accept any changes as part of a given test run. For example:

• Keeping a text representation of a Database schema in a .verified.sql file (see Verify.SqlServer).

This can be done using AutoVerify():

var settings = new VerifySettings();
settings.AutoVerify();

^{snippet source | anchor}

Note that auto accepted changes in .verified. files remain visible in source control tooling.

OnHandlers

OnFirstVerify is called when there is no verified file.

OnVerifyMismatch is called when a received file does not match the existing verified file.

public async Task OnHandlersSample()
{
    var settings = new VerifySettings();
    settings.OnFirstVerify(
        receivedFile =>
        {
            Debug.WriteLine(receivedFile);
            return Task.CompletedTask;
        });
    settings.OnVerifyMismatch(
        (receivedFile, verifiedFile, message) =>
        {
            Debug.WriteLine(receivedFile);
            Debug.WriteLine(verifiedFile);
            Debug.WriteLine(message);
            return Task.CompletedTask;
        });
    await Verify("value", settings);
}

^{snippet source | anchor}

Received and Verified

• All *.verified.* files should be committed to source control.
• All *.received.* files should be excluded from source control.

Not valid json

Note that the output is technically not valid json. Single quotes are used and names are not quoted. The reason for this is to make the resulting output easier to read and understand.

Videos

• Verify Xunit Intro

Extensions

• Verify.AngleSharp.Diffing: Comparison of html files via AngleSharp.Diffing.
• Verify.Aspose: Verification of documents (pdf, docx, xslx, and pptx) via Aspose.
• Verify.EntityFramework: Verification of EntityFramework bits.
• Verify.ICSharpCode.Decompiler: Comparison of assemblies and types via ICSharpCode.Decompiler.
• Verify.ImageMagick: Verification and comparison of images via Magick.NET.
• Verify.ImageSharp: Verification of images via ImageSharp.
• Verify.NServiceBus: Verify NServiceBus Test Contexts.
• Verify.Phash: Comparison of documents via Phash.
• Verify.SqlServer: Verification of SqlServer bits.
• Verify.Web: Verification of web bits.
• Verify.WinForms: Verification of WinForms UIs.
• Verify.Xaml: Verification of Xaml UIs.

Alternatives

Projects/tools that may be a better alternative to Verify

• ApprovalTests
• Snapshooter
• Snapper

Security contact information

To report a security vulnerability, use the Tidelift security contact. Tidelift will coordinate the fix and disclosure.

Icon

Helmet designed by Leonidas Ikonomou from The Noun Project.