vannevelj/roslyntester

A library that will help you unit test your Roslyn analyzers

RoslynTester

A library that will help you unit test your Roslyn analyzers. This package contains the default test helpers provided with the Diagnostics + CodeFix solution template but updated for the latest version of Roslyn and with a few enhancements.

NuGet

https://www.nuget.org/packages/RoslynTester/

Why should I use this?

The CodeFix + Diagnostics solution template is not updated at the same speed as Roslyn is. This means that if you want to test your analyzers, you are stuck with the older version of Roslyn which might (will?) contain bugs that have been fixed in later versions.

By providing these classes as a NuGet package I achieve two solutions:

• When you have an existing analyzer and want to update to a new version of Roslyn, you don't have to worry about backwards-compatibility issues introduced by a newer version.
• When you start a new project you can get started right away with the latest version of Roslyn by removing the default test files and including this package.

What guarantee is there that this is constantly updated?

There is none. I will try to keep up with new Roslyn releases and add features when I feel them to be necessary but in case I ever fall behind you're free to let me know (or send a PR) and I'll get right on it.

Can I contribute?

Yes, definitely. Create an issue or look for an open issue and provide me with a pull request. As always: if you're intending to make a big change, let's discuss it first to avoid unnecessary work.

roslyntester's Issues

Allow only specific new compiler diagnostics

It would be nice to be able to allow only a specific set of new diagnostics when testing new code fixes. This would ensure we knew exactly which new diagnostics were being triggered, and only allow acceptable ones to be triggered.

Invalid code yet test passed?

I had something along the lines of this:

object o;
var x = o is int ? (int) o : (string) o;

which is invalid code, yet the test passed. Look into this further.

Code with errors shows a different message than what is actually wrong

My theory: when we provide code in an uncompilable state that also violates one of the CSxxxx messages that we ignore when compiling (no main method, type not found, etc) it might return one of these instead of what is actually holding up.

This would explain why @Hosch250 saw a "type not found" message when the test had a protected member in a struct. Perhaps we're doing something like "errors.First()" instead of retrieving the correct one that is not one of the abovementioned error codes.

VB Unit Testing Issues

When unit testing in VB.NET, there are a few issues:

1. It tries to use the C# compiler.
2. It has missing references (fixed in the same location as above).

Reference PR #15

Includes more unit tests.

Remove Location requirement

Seriously. Who wants to enter this? Remove the entire need for the Locations property and simply verify that each expected diagnostic can be found in the actual diagnostics (verify by DiagnosticId and message) and that there is an equal amount of both.

Cleanup tests

The "tests" we have in this project are horrible and really should be cleaned up at some point.

Diagnostics are shown multiple times if multiple documents belonging to the same project are added

Title says it all.

I have it figured out and the culprit is

RoslynTester/RoslynTester/RoslynTester/Helpers/DiagnosticVerifier.cs

Line 325 in 6255f5c

foreach (var project in documents.Select(x => x.Project))

which does not take into account duplicated projects (e.g. documents belonging to the same project). I have confirmed this is the issue after I saw that the newest version of the helpers provided by Microsoft now use a HashSet<> prior to iterating. We can do the same or just add a Distinct() call after our Select().

Create VerifyFix method that allows applying a code fix to a source with multiple trees

We can test VerifyDiagnostic with code from multiple files, but not VerifyFix.

Refactor code

1. Make virtual CSharpCodeFixProvider() and related methods needed to override for testing abstract
2. Split the verifiers into subclasses CSharp and Visual Basic and get rid of the duplicated language-specific overloads.
3. When a codefix's result differs from the expected, show a complete well-formatted view of what was generated.

This will show people much earlier when they forgot to override those methods and it will make the public contract a lot prettier as well.

Allow running broken code

There may be a time when someone wishes to input broken code and output working code. This feature could be turned off by default, or it could just ensure that the output compiles.

Support for adding extra assembly references to the test project

When code analyzers are deployed in a NuGet package quite often the CodeFixes contain references to namespaces and code that are part of that NuGet package.

Therefore it should be possible to add the assembly as a reference to the test project other wise the codefix unit test will fail.

I've made a branch with my proposal for this change.

Allow for non-activated analyzers to be run

If your analyzer is set as isEnabledByDefault: false then it won't run in the tests. We should change this to have them always be run (otherwise what are you testing?)

Unsafe compilation flag

When testing code marked unsafe, there's an error message indicating we need to provide a compilation flag /unsafe. Look further into this and perhaps provide an option to specify compilation flags?

RoslynTester NuGet package pulls in NUnit, why?

I've replaced the default unit test code of my analyzers, with that of RoslynTester (thanks btw, for putting that in a nuget package ;). However, I noticed that it also pulled in NUnit. Is that really necessary?

I'm using Xunit, instead of NUnit, so I don't need it. Should RoslynTester really depend on NUnit?

Bring RoslynTester entirely to rc2

WE DONT WANT NO NIGHTLY BUILDS ANYMORE

update README

Update README to specify what RoslynTester's features are

Is RoslynTester hiding exceptions?

Are exceptions being caught somewhere in the RoslynTester? If they are: let everything through! I need that stuff to properly find out if something is broken inside the tester.

For clarification: I don't know but it was something I noticed through using RoslynTester. Maybe the exceptions were held up in another part of the pipeline? Just make sure it isn't here.

Throwing an exception from an analyzer doesn't fail the test

I think. I'm not sure but I think so. It would explain some of the behaviour I've seen anyway.

Invalid code doesn't act.. invalid?

See: Vannevelj/VSDiagnostics#42

Uncompilable code was tested yet.. it worked?