fatihsahin/test-flask

TestFlask - A record-replay-assert testing framework and tools for .net

TestFlask

Visit TestFlask GitHub Page for more details

TestFlask is a tool to create tests for your .net WCF Service or REST API. It is based on a very simple idea. Record your any .net method invocation in runtime with arguments and the return type. After recording, replay the same method invocation without actually calling the real implementation when testing or debugging.

It is super useful for creating auto-mocks by actually calling your WCF service or your REST API in an integration testing or real production environment (if you supply proper TestFlask Http headers). The end result is that you get a lot of recorded method calls that are invoked in a hierarchy.

All you need to do is add a Playback attribute on top of the method that you want to record/replay. Just look at this

[Playback(typeof(MovieNameIdentifier))]
public Movie GetMovieWithStockCount(string name)
{
    //gets movie info from info service
    var movie = infoService.GetMovieInfo(name);
    //obtain stock info from stock service
    movie.StockCount = stockService.GetStock(name);
    return movie;
}

After you build your project, TestFlask will weave your code and turn it into something like below . You can see it if you decompile your assembly with a decompiler tool.

[Playback(typeof (MovieNameIdentifier), null)]
public Movie GetMovieWithStockCount(string name)
{
    FuncPlayer<string, Movie> player = new FuncPlayer<string, Movie>("MovieRental.Models.Movie MovieRental.Business.RentalManager::GetMovieWithStockCount(System.String)", (IRequestIdentifier<string>) new MovieNameIdentifier(), (IResponseIdentifier<Movie>) null);
    player.BeginInvocation(name);
    switch (player.DetermineTestMode(name))
    {
        case TestModes.NoMock:
            return player.CallOriginal(name, new Func<string, Movie>(this.GetMovieWithStockCount__Original));
        case TestModes.Record:
            return player.Record(name, new Func<string, Movie>(this.GetMovieWithStockCount__Original));
        case TestModes.Play:
            return player.Play(name);
        default:
            throw new Exception("Invalid TestFlask test mode detected!");
    }
}

public Movie GetMovieWithStockCount__Original(string name)
{
    Movie movieInfo = this.infoService.GetMovieInfo(name);
    movieInfo.StockCount = this.stockService.GetStock(name);
    return movieInfo;
}

TestFlask uses Fody library to plug-in to MS Build process for weaving. It also depends on Mono.Cecil library to manipulate IL.

Then, you can now either replay like a mock while you are trigerring a regression test or just developing a new feature while replaying a previously recorded scenario and having the comfort that you already easily created (recorded) mocks for all your marked external dependencies while debugging.

The advantage of using TestFlask over service virtualization is that you actually do not have to deal with protocols or external testing environments, tools. It all happens inside your code and your IDE. You are also not limited to mocking your external SOAP, XML or JSON services. You can now mock a database call, or a totally different external resource (ftp, tcp, sna you name it). It actually does not matter, because your method is now intercepted and mocked. It will not do IO over the wire now.

TestFlask not just records your method invocations but also provides a way to create organized business scenarios just like you are creating a isolated testing session. Look at how a recorded scenario looks like

This is the scenario view for the Test Flask Manager web app. Notice that little Replay checkbox? It gives you flexibility to mock or not to mock a .net method depending on your choice next time. Also you can alter the response of a method and replay as your method will now return that object the we way you modified. Test data is encapsulated in that scenario with a scenario number and will not affect your other recorded scenarios.

Using the combination of these alterations, you can easily and flexibly develop a complex and highly integrated project without worrying about actual integration problems and test data. It is now all isolated for you and encapsulated in a TestFlask scenario. TestFlask is never a tool to replace your real unit tests, but complement them and provide you a hybrid testing power inside your IDE.

Go check out Movie Rental Sample App for a complete demonstration. I also created a Quick Pay Demo, a simpler demo version that is explained in the video here.

TestFlask also has built-in CLI tooling (checkout TestFlask.CLI) for auto creating unit-tests that trigger your recorded scenario, so that you can instantiate a testing scenario with mocks ready inside your IDE. MS Tests are supported right now, but I will add NUnit test support when requested.

TestFlask determines what to do by looking up to custom https headers that the client has sent to the service. There are actually five test modes.

TestFlask-Mode	Description
Record	Calls the original method and then persists request and response objects through TestFlask.API into a mongoDB database.
Play	Calls TestFlask.API to look for a recorded response for the current request and returns that response.
NoMock	Calls the original method with no mocking.
Assert	Same as Play, however in this case TestFlask stores last response as an assertion result to assert later on.
IntelliRecord	Just like record, however in this case if a matching invocation already exists it just replays it, but patches missing ones on to existing invocation tree.

As you can see TestFlask talks to a REST API to do recording and playing. This component is called TestFlask.API

How to instantiate a TestFlask.API Host

In order to persist your intercepted calls from your weaved methods, you should initiate a TestFlask.API host.

This project is actually an ASP.NET MVC API project. As a persistence mechanism, it uses mongoDB. Therefore, configure your TestFlask.API project's web.config with a running mongoDB instance

<appSettings>
    <add key="testFlaskMongoDbServer" value="mongodb://localhost" />
    <add key="testFlaskMongoDbName" value="test" />
</appSettings>

How will I send proper TestFlask HttpHeaders to determine testing mode?

There are four http headers that TestFlask looks up to determine how to store your intercepted request & response.

Http Header	Description
TestFlask-Mode	See above
TestFlask-ProjectKey	This is the key that you created for your backend service to categorize scenarios
TestFlask-ScenarioNo	This is the number for your test scenario. It is the main test data encapsulation identifier.
TestFlask-StepNo	This is optional, if you provide a step no, it will override that step. If you do not, TestFlask will create an auto step under that scenario.

You can also examine a SoapUI project inside that sample app to trigger your backend without UI.

Where does TestFlask actually persist all the stuff?

It persists all scenarios, steps and invocations in a MongoDB database.

Lastly, TestFlask.Assistant.Mvc project (nuget package) is an ASP.NET MVC extension to ease integrating your ASP.NET MVC client to send proper headers to your TestFlask ready backend service.

TestFlask still needs a lot of development and hopefully it will go on. I did upload v1.0.0 packages to nuget.org | TestFlask.

Please feel free to contribute with PRs. Thanks!

Icon

Erlenmeyer Flask by Rockicon from the Noun Project

test-flask's Issues

Add dynamic pattern matching strategy

When an invocation is requested from TestFlask API, currently it uses invocation instance hash code to fetch single invocation. Invocation instance hash code is a composite hash code to define a invocation instance uniquely inside the whole recorded call tree. However, different invocation matching strategies can be used such as using signature hash code, signature+request identifier hash code, and depth hash code. That way TestFlask could return same response for methods with same signature, same signature with same request identifier or same signature+request identifier on same depth level. This way api could return another invocation response and it may help regression tests to be independent from call stack tree changes. Strategy can be set on many levels including on project level, scenario level and step level. Lower levels would be dominant.

Prepare a nuget 1.0.0 package for nuget.org

TestFlask is now mature enough for a 1.0.0 version package release to nuget.org.

weave static methods

testflask shoud be able to weave static methods

Dynamic playback marking

A visual studio extension can be implemented to mark methods with Playback functionality on VS editor. Marked methods with proper identifier can be persisted to a xml file and weaver can rely on this file to weave methods

Cascade response updates using declared matching strategy

When a response is being updated for a invocation, user can declare cascading options. For example user can trigger the update for all responses with matching depth, sibling, request or signature inside the step, with matching request or signature inside a scenario or whole project with a single update action.

Labeling scenarios

Scenarios can have one or more labels to categorize the scenario trair explicitly. Unit test generator tool or other complementary features can rely on these labels to use the scenario or not in their specific context

Add request replacement variables on different levels

Users can define variables on project, scenario and step levels to replace any part of their recorded root request to be dynamically replaced with variable values. Moustache style {{variableName}} can be preffered. By altering variable values, request can be dynamically modified before any assertion or playback. This could help to trigger some secure service or apis with changing token or user credentials or with custom soap or http headers, idempotency keys etc.

Update libs to target .net standard

Update libs to target .net standard latest version

Cloning a scenario

Scenario cloning is currently not implemented. A clone should update stepno and scenariono containing fields in whole scenario object tree as well as variables to be cloned to the new scenario

Auto-generated unit tests

A side TestFlask project potentially could auto generate unit tests to trigger each scenario as a seperate test method. Generator can use root invocation request object and method signature to set up the test. It could also use expected assertion object to truly assert the output. It could help increasing the test code coverage.

New TestFlask mode - IntelliRecord

Introduce a new testflask mode like IntelliRecord/HybridRecord. When testFlask is set to this mode, inner player will record unmatched invocations as new and call original record wrapper, yet it will reuse recorded responses for matching invocations. Therefore it will be possible to record a diff patch to reduce the cost of setting up integration environments for re-recording. Only newly introduced integration nodes need to be online.

Generate variables by applying generator regex pattern

Raw http requests that are kept inside invocations can be manipulated and parameterized dynamically during recording if a variable that can be applied is already declared and have a matching generator regex pattern.

Update Fody version

Upgrade TestFlask to use new Fody version. New Fody licence is asking package consumers to be a patreon. TestFlask can contain its own plugin to Ms Build and therefore there would be no need for Fody. Worth considering.

Support variable processing for cross site raw requests

Currently cross site raw requests are not persisted and therefore it is not possible to use variable replacement processing. Consider a solution.

Support IntelliRecord mode on cross service requests

As invocations are cleared before hand on initial service root depth, cross service cannot find matching invocation to replay in intellirecord mode even if the invocation was marked as replayable. Consider an alternative solution to clearing all invocations. Each service may manage its ow invocations inside a step, a service key should be introduced in configuration for services

Clone scenario variables while cloning scenario

Clone scenario implementation inside StepController must also clone scenario variables.

Java version for TestFlask

This is a very early brainstorming. Java services can be manipulated using maybe Javassist and a maven task plugged in for javassist

Support for weaving async methods

Create a POC branch for weaving async functions that returns Task or Task<T>. TestFlask context (such as depth, invocation index or hierarchy) can be synced with continuation tasks that are appended inside player. HttpContext may or may not be available inside thread pool. An alternative solution can be necessary. Also consider what can be done with async void methods

weave generic methods

testFlask should be able to weave generic methods such as
public TRes DoSomething<TReq1, TReq2, TRes>(TReq1 req1, TReq2 req2, int someOtherParam) { }

Patch invocations to a step

An invocation totally declared from scratch or cloned from other step invocation trees can be patched to an existing invocation tree to make steps more flexible to react to underlying service version/implementation changes. UI Views can be implemented in test-flask-web repo, or a new rich client (like WPF maybe?) can be implemented for specific patching/cascading variations.

unit test documentation

auto generated unit test methods can have summary doc sections feeded from scenario and step descriptions

Resolve evaluated variables

Some variables can have a basic C# snippet to resolve the value in runtime. While declaring a variable, we can use the value field to store expression and a IsEvaluated flag to determine whether this is a static value or must be evaluated. Probably a very basic C# compiler would work and there is no need to support advanced syntax like LINQ at the beginning. It would be great if expression like this works => System.DateTime.Now.ToString("dd.MM.yyyy")

Gather played invocation call tree

While playing a current step, current played invocation tree can be gathered just like in record mode. This tree can be used to log or report a difference between recorded tree and replayed one. Also unmatched invocations can be marked to alert changes in play

add full json type name to request response fields

in order to fully deserialize recorded request and response objects, type names must be included in json strings. This way we can support reinstantiating objects with inherited types

CI integration with CLI tooling

A CLI tool can be written to run a console app to trigger recorded steps by actually calling TestFlask ready backend service host to support regression testing automation. Console app can stream errors to ErrorOutputStream, so a defined TFS build task can plug in and fail the entire build process if any assertions fail

Generic identifier compiler problem

For generic methods, we cannot define a generic attribute. It is not allowed in C#. Therefore it makes sense to declare an identifier like below with an object type for generic args.

public class GetGenericReqIdentifier : IRequestIdentifier<object, FooRequest>
{
	public string ResolveDisplayInfo(object arg0, FooRequest arg1)
	{
		return "ooh";
	}

	public string ResolveIdentifierKey(object arg0, FooRequest arg1)
	{
		return "whaaat";
	}
}

However, testFlask weaver generates something like

[Playback(typeof(GetGenericReqIdentifier), null)]
public TRes GetSomeGeneric<TReq, TRes>(TReq req, FooRequest fooReq) where TRes : new()
{
	FuncPlayer<TReq, FooRequest, TRes> funcPlayer = new FuncPlayer<TReq, FooRequest, TRes>
("TRes AssemblyToProcess.SomeClient`1::GetSomeGeneric(TReq,AssemblyToProcess.FooRequest)", 
		new GetGenericReqIdentifier(), null);
	//the rest of body
}

which generates a compiler error that is Argument 2: cannot convert from 'AssemblyToProcess.GetGenericReqIdentifier' to 'TestFlask.Aspects.Identifiers.IRequestIdentifier<TReq, AssemblyToProcess.FooRequest>'.

TestFlask must support generic identifier implementations

Get rid of HttpContext.Current

HttpContext.Current is blocking migration to .net core. It also disables support for async call interceptions. I will use a static Dictionary and keep it simple. In async calls they will be directly appended to root call. In sync calls to distinguish initial invocation from childs or siblings I plan to mark initial invocation with sth like [Playback(IsRoot: true)]. A timer thread with some idle time configuration may clean up static dictionary after a recording session is complete and override the scenario if IsRoot:true invocation is recorded again or replayed.