verifytests/verify

Verify is a snapshot tool that simplifies the assertion of complex data models and documents.

verify's Issues

Update Diffengine to 4.5

Add SharedVerifySettings.NameForParameter

https://github.com/VerifyTests/Verify/blob/master/docs/parameterised.md#complex-memberdata

UseParameters should allow nulls

43ccd63

Add xml docs to DiffEngine

Add support for IAsyncEnumerable

Add cleanup support to converters

Add more string scrubbing

https://github.com/SimonCropp/Verify/blob/master/docs/scrubbers.md#available-scrubbers

fix EmptyFiles transient files not included

missing tiff and jpeg file types for diff tool support

93064f1

add RequiresTarget to diffengine

allows diff tools to op out of creating an empty file for comparison purposes

Fix guids/datetimes in dictionary keys not being scrubbed

Partial namespace included in verified file name

for multipart namespaces (eg Foo.Bar.Bat) part of the namespace is being correctly included in the verified file name. Once upgrading to 1.27.0 all existing files, with incorrect names, will be renames to the correct file name.

Comparers not applying to text

Make Verify.Xunit static

Inheriting from VerifyBase is no longer required.

Instead use the static Verifier and add [UsesVerifyAttribute] to all xunit tests that use Verify

Use semantic HTML comparison instead of raw text/string comparison

Feature

HTML can change from a raw markup perspective, without changing the semantic meaning of it. Most common is changes to whitespace. This causes tests to break, when they do not need to.

Describe the solution

If Verify uses a semantic HTML comparison library, I created AngleSharp Diffing for this exact purpose, instead of just doing text comparison, then tests that uses Verify will be much more stable. There are some additional options that can be used in to modify the diffing, e.g. switch to case insensitive comparison, or ignore certain attributes (generated IDs), that are also very helpful.

I will be happy to help integrate AngleSharp Diffing into Verify. Just let me know if you have any questions.

Additional context

The context is primarily with the bUnit integration.

Here is a quick sample, that shows how Verify fails a test because of insignificant whitespace, and the bUnit tests passes, because it uses MarkupMatches to verify the rendered markup, using AngleSharp Diffing.

Fix suppress deterministic for mutli target

Tests pass when they should fail

Describe the bug

Running tests, the comparison tool is launched and the diff clearly shows that the verified vs received are different, but for some reason the tests pass.

• Visual Studio - 16.5.1
• dotnet --version = 3.1.200
• Project file is netcoreapp3.1 with:
• xunit 2.4.1 (and vs runner)
• Verify.xunit 1.27.0

Minimal Repro

Raising for now, will do when I can.

Submit a PR that fixes the bug

Raising for now, will do when I can.

Rename some input params to "target"

Add ability to customise build tool order via code

Fix dangling verified file

when verifying multiple streams, if the umber of files reduces, the test should fail and the clipboard should contain a command to remove extra files

Add DisableDiff and DisableClipboard to VerifySettings

• https://github.com/SimonCropp/Verify#disable-clipboard
• https://github.com/SimonCropp/Verify/blob/master/docs/diff-tool.md#disable-diff

incoming stream should be disposed

42a54e3

Some settings not respected when serializing converter info

Add MaxInstancesToLaunch

https://github.com/SimonCropp/Verify/blob/master/docs/diff-tool.md#maxinstancestolaunch

Move types in the Verify to VerifyTests

to unblock c# using static feature

Add AutoVerify

https://github.com/SimonCropp/Verify#autoverify

Add custom diff tool support

https://github.com/SimonCropp/Verify/blob/master/docs/diff-tool.custom.md

rename SharedVerifySettings to VerifierSettings

sync on snapshot/approvals/verification testing

@theramis @isidore @michaelstaib @nscheibe @claremacrae

hey peoples. Given our project overlaps i though it might be helpful for us to be aware of each other and have a possible sync so we can share ideas, lessons learnt, code etc.

• ApprovalTests
• Snapshooter
• Snapper
• Verify

Some discussion points i can think of

• the trade offs between stack trace walking and using [CallerFilePath]
• how diff tools work, which i think Snapper and Snapshooter dont have?
• verification of files/binary data. eg image or office document verification
• the possibility of sharing code. i have already created a stand alone nuget for EmptyFiles (which makes comparison of files easier). i was also considering splitting out the whole diff tool detection+management into a nuget.

So would people like to get on a call? or perhaps discuss things here? or some other medium?

Is there anyone else I should have brought in? or tools i should have listed?

Add OnHandlers

https://github.com/SimonCropp/Verify/#onhandlers

Add more diff tool support

• KDiff3
• TkDiff
• TortoiseIDiff
• TortoiseMerge
• DiffMerge
• Kaleidoscope

pass settings to comparer

Add rider diff support

Fix more null references

using nullable attributes from net5

Remove non-async stream compare

cache ncrunch env variable

add DisableNewLineEscaping

https://github.com/SimonCropp/Verify/blob/master/docs/serializer-settings.md#disablenewlineescaping

File comparison should not lock file

New file name convention logic when Verify is called in base classes

Is the feature request related to a problem

Recent changes wrt to [UsesVerify] and how Verifier.Verify (the method) uses [CallerFilePath] string sourceFile = "" have meant that when upgrading some of my projects the file names generated overlapped each other.

The issue is because when calling Verify from the base class, the CallerFilePath is the base class' file path.

Essentially I had:

[UsesVerify]
public class TestsBase
{
    protected async Task UtilityAssertMethod(...)
    {
        //... A bunch of other assertions / mock verifications, etc.
       
        await Verifier.Verify(actual, settings);
    }

    protected async Task Some_Theory_Should_Work_Impl(bool someParameter)
    {
        //...arrange
        //...act
        settings.UseParameter(someParameter);
        await Verifier.Verify(actual, settings);
    }
}

public class SomethingTests : TestsBase
{
    [Fact]
    public async Task Some_Test_Should_Work()
    {
        // ...arrange
        // ...act
        base.UtilityAssertMethod(...);
    } 

    [Theory]
    [InlineData(true)
    public async Task Some_Theory_Should_Work(bool someParameter)
        => await base.Some_Theory_Should_Work_Impl(someParameter);
}

The initial way around it I found was to have the base class define an abstract method for it's utility method to call Verify from:

public abstract Task Verify(string target, VerifySettings settings);

And then each leaf test class actually implement that, so their file path would be used:

public override Task Verify(string target, VerifySettings settings)
            => Verifier.Verify(target, settings);

Potentially another workaround would be to pass that value explicitly, rather than let the runtime pass it through. Here's a dotnetfiddle that seems to show that should work: https://dotnetfiddle.net/BT27ry

But I'd need to supply the full path of the file, and I guess getting that is just as challenging. Actually I could just update my protected async Task UtilityAssertMethod(...) to accept [CallerFilePath] string callerFilePath = "" and pass that on to Verify. Will give that a go, as it's a lot cleaner. One issue might be that then the verify assertions on UseParameters might want that to be a parameter...

Describe the solution

Don't really have a solution, was just talking through the issue at the moment, but potentially we can just immediately close this and leave it as an example solution for others.

Describe alternatives considered

Two solutions above.

Additional context

None at this stage.

Explanation on how to get actual Json Format

I've spent a good 15 mins trying to figure out which api changes the JsonTextWriter.

Also I think if you have something called "Json" then it should be json otherwise make a new format. The problem is you might "think" it looks better but if it's not json then other json based tooling will not work with it.

Also while we're at it where can I find how to change the source directory for all the received files?

release process testing

add TeamCity detection

Expose Namer `Runtime` or `RuntimeAndVersion` strings

https://github.com/SimonCropp/Verify/blob/master/docs/naming.md#namerruntimeandversion

Should use ToString on Expression

Update DiffEngine to fix teamcity detection

fe4e8e8

Issue with Verify creating empty file

Preamble

• Visual Studio - 16.5.4
• dotnet - 3.1.latest
• Verify.Xunit

Full refs on the test project:

Describe the bug

Running the test fails.

• Seems to create the *.received.txt file
• Interestingly, if I rename the *.received.txt to *.verified.txt, the test the passes.

So my guess is it's choking trying to create the empty verified one, which the stack trace certainly attests to.

• Note, I went looking into that .nuget packages path, and what struck me as odd is the path it can't find a part of is:
• C:\Users\Developer\.nuget\packages\emptyfiles\2.0.1\lib\netstandard2.0\EmptyFiles
  But the file it could be looking for is:
• C:\Users\Developer\.nuget\packages\emptyfiles\2.0.1\lib\netstandard2.0\EmptyFiles.dll

i.e. it's a dll not a folder.

    System.TypeInitializationException : The type initializer for 'EmptyFiles.AllFiles' threw an exception.
    ---- System.IO.DirectoryNotFoundException : Could not find a part of the path 'C:\Users\Developer\.nuget\packages\emptyfiles\2.0.1\lib\netstandard2.0\EmptyFiles'.
  Stack Trace: 
    AllFiles.TryCreateFile(String path, Boolean useEmptyStringForTextFiles) line 209
    DiffRunner.Launch(ResolvedDiffTool diffTool, String tempFile, String targetFile) line 79
    DiffRunner.Launch(String tempFile, String targetFile) line 65
    VerifyEngine.ProcessMissing(StringBuilder builder, FilePair item) line 278
    VerifyEngine.ProcessMissing(StringBuilder builder) line 238
    VerifyEngine.ThrowIfRequired(String message) line 123
    InnerVerifier.Verify(String input, VerifySettings settings) line 25
    InnerVerifier.Verify[T](T input, VerifySettings settings) line 43
    --- End of stack trace from previous location where exception was thrown ---
    ----- Inner Stack Trace -----
    FileSystemEnumerator`1.CreateDirectoryHandle(String path, Boolean ignoreNotFound)
    FileSystemEnumerator`1.ctor(String directory, EnumerationOptions options)
    FileSystemEnumerable`1.ctor(String directory, FindTransform transform, EnumerationOptions options)
    FileSystemEnumerableFactory.UserFiles(String directory, String expression, EnumerationOptions options)
    Directory.InternalEnumeratePaths(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options)
    Directory.EnumerateFiles(String path, String searchPattern, SearchOption searchOption)
    AllFiles.cctor() line 63

Minimal Repro

Incoming when I get a moment.

Submit a PR that fixes the bug

Incoming when I get a moment.

use resulting file type for diff engine

Update EmptyFiles to V1

Allow converters to have different extensions per stream

https://github.com/VerifyTests/Verify/blob/master/docs/converter.md

Before

VerifierSettings.RegisterFileConverter<Image>(
    toExtension: "png",
    canConvert: target => Equals(target.RawFormat, ImageFormat.Tiff),
    conversion: (image, settings) =>
    {
        var pages = image.GetFrameCount(FrameDimension.Page);

        var streams = new List<Stream>();
        for (var index = 0; index < pages; index++)
        {
            image.SelectActiveFrame(FrameDimension.Page, index);

            var page = new MemoryStream();
            image.Save(page, ImageFormat.Png);
            streams.Add(page);
        }

        return new ConversionResult(
            info: new
            {
                image.PixelFormat,
                image.Size
            },
            streams);
    });

VerifierSettings.RegisterFileConverter(
    fromExtension: "tif",
    toExtension: "png",
    conversion: (stream, settings) =>
    {
        using Image image = Image.FromStream(stream);
        var pages = image.GetFrameCount(FrameDimension.Page);

        var streams = new List<Stream>();
        for (var index = 0; index < pages; index++)
        {
            image.SelectActiveFrame(FrameDimension.Page, index);

            var page = new MemoryStream();
            image.Save(page, ImageFormat.Png);
            streams.Add(page);
        }

        return new ConversionResult(
            info: new
            {
                image.PixelFormat,
                image.Size
            },
            streams);
    });

After

VerifierSettings.RegisterFileConverter<Image>(
    canConvert: target => Equals(target.RawFormat, ImageFormat.Tiff),
    conversion: (image, settings) =>
    {
        var pages = image.GetFrameCount(FrameDimension.Page);

        var streams = new List<ConversionStream>();
        for (var index = 0; index < pages; index++)
        {
            image.SelectActiveFrame(FrameDimension.Page, index);

            var page = new MemoryStream();
            image.Save(page, ImageFormat.Png);
            streams.Add(new ConversionStream("png",page));
        }

        return new ConversionResult(
            info: new
            {
                image.PixelFormat,
                image.Size
            },
            streams);
    });

VerifierSettings.RegisterFileConverter(
    fromExtension: "tif",
    conversion: (stream, settings) =>
    {
        using Image image = Image.FromStream(stream);
        var pages = image.GetFrameCount(FrameDimension.Page);

        var streams = new List<ConversionStream>();
        for (var index = 0; index < pages; index++)
        {
            image.SelectActiveFrame(FrameDimension.Page, index);

            var page = new MemoryStream();
            image.Save(page, ImageFormat.Png);
            streams.Add(new ConversionStream("png",page));
        }

        return new ConversionResult(
            info: new
            {
                image.PixelFormat,
                image.Size
            },
            streams);
    });

Add TortoiseGitMerge support