justeattakeaway/httpclient-interception

A .NET library for intercepting server-side HTTP requests

httpclient-interception's Issues

Question: Should HttpClient created from HttpClientInterceptorOptions auto-follow redirects?

Expected behaviour

I have created a HttpClientInterceptorOptions instance, defined 2 interceptions (one to capture requests to http://foo.com and return a 302 pointing to https://google.com and another to capture requests for https://google.com and response with some text content), and registered both of those interceptions with the HttpClientInterceptorOptions instance. My expectation is that if I create an HttpClient via HttpClientInterceptorOptions.CreateHttpClient() and then call client.GetAsync("http://foo.com"), the client should get the 302 from the initial request, redirect to google.com, and the final response should contain my text content from the google.com interception.

Actual behaviour

But...the resulting response is the 301 to google.com. The client does not auto-follow the redirect.

Steps to reproduce

using System.Net;
using JustEat.HttpClientInterception;
using Microsoft.Net.Http.Headers;

var clientOptions = new HttpClientInterceptorOptions();

new HttpRequestInterceptionBuilder()
    .Requests()
    .ForHttp()
    .ForHost("foo.com")
    .ForPath("dosomething")
    .Responds()
    .WithInterceptionCallback(msg =>
    {
        Console.WriteLine("Request for foo.com intercepted!!!");
    })
    .WithStatus(HttpStatusCode.Redirect)
    .WithResponseHeader(HeaderNames.Location, "https://google.com")
    .RegisterWith(clientOptions); 

new HttpRequestInterceptionBuilder()
    .Requests()
    .ForHttps()
    .ForHost("google.com")
    .Responds()
    .WithInterceptionCallback(msg =>
    {
        Console.WriteLine("Request for google.com intercepted!!!");
    })
    .WithContent("Hello!")
    .RegisterWith(clientOptions);

var client = clientOptions.CreateHttpClient();
var response = client.GetAsync("http://foo.com/dosomething").Result;
var content = response.Content.ReadAsStringAsync().Result;

request: more examples for response changing based on how many times called so far

My current situation is that I want to test a Polly policy for handling 429 (too many requests) responses. I'm hoping to be able to have the interceptor return 429 on the first (or maybe the first two) calls, then 200 on the ones after.

Describe the solution you'd like
This is probably just a sample to be included

Describe alternatives you've considered
I tried keeping a count local var and creating 2 options builders, using WithInterceptionCallback overload passing a Predicate, with one that returned 429 with ++count < 3 then another with count >= 3 that returned 200, but that didn't work (very possible bug on my side and it would have worked fine if I'd done it correctly)

Additional context
While there are certainly scopes to change behavior, in this case I need the behavior to change in the context of a single HttpClient.GetAsync call, so AFAICT I can't use scopes to change the behavior in the "middle of the call". Ideally the sample would maybe include something like an array of objects/tuples of status code and content and as each call came in, it used the next item in the array to determine how to respond.

Ignore order of query params

Is your feature request related to a problem? Please describe.

We have urls generated by a 3rd party framework and the order some query params are added is not always deterministic.

It would be helpful if the matcher was able to determine if query params were the same when not added in the same order.

Describe the solution you'd like

Ideally this would just work. If there's concern of introducing a breaking change, consider adding a property like HttpClientInterceptorOptions.IgnoreQueryParamOrder.

The matcher would split the query string into parts before comparing.

Describe alternatives you've considered

I created a custom matcher to do this, but I don't want to copy that code into every solution that uses this package and it seems silly to create a custom nuget just for that.

after upgrade, build error

Visual Studio 2022

byte[] content = await response.ContentFactory!().ConfigureAwait(false) ?? []; line 479

internal sealed class StringSyntaxAttribute(string syntax, params object?[] arguments) : Attribute line10

Question: How to simulate a timeout/no response from downstream service

I want to write a test for validating behaviour of a class once it has made a call to downstream service that is currently down/unavailable (ie, it's never going to process the incoming request)

Am I right in thinking that you would do this as shown in this sample test:

https://github.com/justeat/httpclient-interception/blob/3632d2f3d62ff51e2e3e3119d254d0f6d35ab10f/tests/HttpClientInterception.Tests/Examples.cs#L284

or is there another way?

Revise documentation covering usage together with IHttpClientFactory

I was reading through the documentation that describes how to use HttpClientInterception together with IHttpClientFactory, and I found the recommended solution to be a bit misleading and quite over-engineered, especially having to handroll your own implementation of IHttpMessageHandlerBuilderFilter.

I was able to make it work just by doing this:

var builder = new HttpRequestInterceptionBuilder();

builder.Requests()
       .ForGet()
       .ForAnyHost()
       .Responds()
       .WithStatus(HttpStatusCode.OK)
       .RegisterWith(options);

var services = new ServiceCollection();
services.AddHttpClient("myclient")
        .AddHttpMessageHandler(options.CreateHttpMessageHandler));    // 👈🏻 This line is all you need

var serviceProvider = services.BuildServiceProvider();

var httpClientFactory = serviceProvider.GetRequiredService<IHttpClientFactory>();
var client = httpClientFactory.CreateClient("myclient");

Perhaps the documentation could be updated to reflect this?

Response sequence

I would like to have an option to setup a sequence of responses for certain request. It would certainly help with testing retry policies.

Describe the solution you'd like

Moq allows for user to setup a sequence of responses like:

consumerService.SetupSequence(c => c.Consume(It.IsAny<CancellationToken>()))
    .Returns(payload)
    .Throws<OperationCanceledException>()
    .Returns(payload);

It would be great if we would be able to do the same in this library:

builder
    .Requests()
    .ForPost()
    .ForHttps()
    .ForHost(host)
    .ForPath(endpoint)
    .RespondsInSequence()
        .WithStatus(HttpStatusCode.InternalServerError)
    .Then()
        .WithStatus(HttpStatusCode.InternalServerError)
    .Then()
        .WithStatus(HttpStatusCode.OK)
    .RegisterWith(options);

Add [StringSyntax] attributes

Is your feature request related to a problem? Please describe.

A new attribute, [StringSyntax], is being added to .NET 7 that allows tools such as Visual Studio to provide syntax highlighting etc. for string variables that expect certain formats (JSON, URI, XML, Regex).

Describe the solution you'd like

Add [StringSyntax] to all methods where it would be appropriate. Examples include:

URI

https://github.com/justeat/httpclient-interception/blob/961bfc07a358f58ed2d1af6dace413ef826470e7/src/HttpClientInterception/HttpRequestInterceptionBuilderExtensions.cs#L224

Describe alternatives you've considered

None.

Additional context

Unless Visual Studio supports consuming the attribute by name rather than the concrete type, this would only work for .NET 7+.

WithResponseHeaders and WithContentHeaders overloads

Is your feature request related to a problem? Please describe.

I have an extension method which allows me to register responses for intercepted requests such that each subsequent request that matches, returns a different response content. I do this by using a closure of a Counter object, which gets incremented using WithInterceptionCallback, as follows:

[EditorBrowsable(EditorBrowsableState.Never)]
static class HttpRequestInterceptionBuilderExtensions
{
    public static HttpRequestInterceptionBuilder WithJsonResponses<T>(
        this HttpRequestInterceptionBuilder builder,
        Func<int, T, CancellationToken, Task> callback,
        T[] content,
        JsonSerializerOptions? options = null)
    {
        var counter = new Counter(content.Length);
        T item = default!;

        return builder
            .WithInterceptionCallback(async (_, cancellationToken) =>
            {
                counter.Increment();
                item = content[counter.Value - 1];
                await callback(counter.Value, item, cancellationToken);
            })
            .WithMediaType("application/json")
            .WithContent(() =>
            {
                if (item is Exception exception)
                    throw exception;
                    
                return JsonSerializer.SerializeToUtf8Bytes(item, options);
            });
    }

    class Counter
    {
        readonly int _maxValue;

        public Counter(int maxValue) => _maxValue = maxValue;

        public int Value { get; private set; }

        public void Increment()
        {
            if (Value == _maxValue) return;

            Value++;
        }
    }
}

I would like to do the same for the response headers and content headers as well. This requires new overloads for WithResponseHeaders and WithContentHeaders, and some associated code changes.

Describe the solution you'd like

New overloads for WithResponseHeaders and WithContentHeaders, which allow passing a Func<IDictionary<string, ICollection<string>>>.

Describe alternatives you've considered

No alternatives exist.

Additional context

The reason for requiring different responses to the same request is that the API being "mocked" is one that returns data which changes over time. My test suite is set up so that I can verify this behaviour.

Remove Newtonsoft.Json

Consider removing Newtonsoft.Json as a dependency and just using System.Text.Json instead.

Support for intercepting all requests sent to HttpClient

Would it be possible to easily implement intercepting all requests (i.e. not specifying a host) for the purposes of then doing some sort of assertions on the request.

For example if I am unit testing a class which uses HttpClient and I don't care about the host or the verb that is being sent to (this is covered by other tests) I simply want to test that the correct headers are being applied.

Verify calls to the registered end points have been called [discussion]

There are times where we are setting up a number of requests to match in a test and as part of using the tooling to write the functionality we want to know when the setup calls have not been called when they should have been. So it can be used in a "test first" type of a way to make sure the setup urls are called when the fix has been developed. The ability to throw exceptions when calls are made which aren't configured is great, it's the opposite side of that functionality as I want it to fail or at least check when they're configured and not called.

Describe the solution you'd like
Some sort of public API which allows for verifying the registered calls have been called as part of the request and/or a way to verify all the calls which have been registered have been called.

Initial thoughts would be similar to the Verify behaviour functionality on Moq - https://github.com/moq/moq4/blob/7848788b5968eb29056599db91f9e9b9d16f9ba9/src/Moq/Mock.cs#L255

Describe alternatives you've considered
I was thinking you could create custom content predicates which when matched could keep track of the url and request. Then using the lookup functionality at the end of the processing you could manually check that all the entries had been called as expected. The downside of this it would be custom for each test and would have to be manually setup each time.

I was thinking maybe it could be based off the key creation routine and how the registrations are added to the mapping dictionary and then when they are called through the GetResponseAsync method it could be set as matched. Then could either pass in the builder instance for the request (so it could generate the key again) to check, or some other mechanism to verify that the call had been requested.

The other check that it would need to do would be the ability to use some sort of predicate to make sure the correct call had been requested. For example when the same url needs to be called multiple times with different payloads. So I was thinking it could interrogate the payload as well so you could match it in the same way that you can register predicates in ForContent to match on the content as well as the url etc.

Maybe add an override to the Register method to something like ...

public HttpClientInterceptorOptions Register(HttpRequestInterceptionBuilder builder, out string registrationKey)

So on each registration you can get access to the key value which has been used so you can then request back in later but not overly a fan of this API change.

Thoughts

Just wanted to add this here in case there was an appetite to do something like this. I would be happy to contribute a PR if I could get some guidance on naming and approach to make sure it fitted with the plan for the library.

Test Failures Due to Authorization Issues After Upgrading to .NET 8 and HttpClient-Interception

We recently upgraded our project to .NET 8 and updated the HttpClient-Interception package to its latest version. Following this upgrade, we are encountering an issue where multiple tests are failing due to authorization failures.

Environment
.NET Version: 8.0
HttpClient-Interception Version: 4.1.0
OS: Windows

Issue Description
Prior to the upgrade, all our tests were passing consistently. The tests that are failing are specifically those that rely on HttpClient-Interception for mocking HTTP requests and responses. These tests are now failing with an "unauthorized" error, although no changes were made to the test setup or the codebase, apart from the version upgrades.

Steps to Reproduce
Upgrade the project to .NET 8.
Upgrade HttpClient-Interception to the latest version.
Run the existing tests which involve HTTP request mocking.

Expected Behavior
Tests that were passing previously should continue to pass after the upgrade, assuming no changes to the test logic or setup.

Actual Behavior
Tests involving mocked HTTP requests are failing with an "unauthorized" error.

Additional Information
Error logs/output:
Shouldly.ShouldAssertException : response.response.StatusCode
should be
HttpStatusCode.Created
but was
HttpStatusCode.Unauthorized

Could you please provide any insights into why this might be happening? Are there any breaking changes in the latest version of HttpClient-Interception that could affect authorization? Also, are there any additional configurations or steps that we need to consider after upgrading to .NET 8 and the latest version of HttpClient-Interception?

Any guidance or suggestions would be greatly appreciated.

Thank you for your help!

MissingMethodException when running on mono with newer Newtonsoft.Json

This is quite an exotic setup I know, but this has my tests failing in CI, so I'll be looking into this too.

I have an example here:
a776327#diff-ad2944e2db1bf6a75b8602f406da7846R13

When running under mono, if I call .WithJsonContent(... then it pops with a runtime exception:

Error Message:
   System.MissingMethodException : Method not found: JustEat.HttpClientInterception.HttpRequestInterceptionBuilder JustEat.HttpClientInterception.HttpRequestInterceptionBuilderExtensions.WithJsonContent(JustEat.HttpClientInterception.HttpRequestInterceptionBuilder,object,Newtonsoft.Json.JsonSerializerSettings)

In recreating this, I discovered that it's not an issue if you are referencing Newtonsoft.Json 9.0.1, but if you reference a higher version such as 12.x then this issue pops up.

ILogger<T> Integration?

When IHttpClientFactory is used with this library, the other built-in message handlers log things. InterceptingHttpMessageHandler on the other hand has no logging at all, though it does have some hooks to allow the integrator to do some logging themselves.

Maybe it would be beneficial to provider a way to wire in ILogger<T>/ILoggerFactory so the message handler had some built-in logging of it's own? The downside to this is creating a dependency on the logging abstractions.

Allow access to CancellationToken

The CancellationToken that is passed into the delegating handler is flowed into the inner delegating handler which is cool, however it isn't passed into the interception DSL, so there's no way to say in our tests "did the request get aborted".

It would be handy if the WithInterceptionCallback gave us that CancellationToken, what are your thoughts on this?

Consider registering HttpClient as a Singleton, not Transient

Hi guys,

Thank you for sharing this really good idea, testing HttpClient is always a pain :)

One comment though: in the Sample and documentation the HttpClient is being registered as transient. That could cause a lot of problems with the server (performance, TCP/IP Port Exhaustion, etc.), as HttpClient is intended to be instantiated once and reused throughout the lifetime of an application or context.

Take a look here for more information:
https://docs.microsoft.com/en-us/azure/architecture/antipatterns/improper-instantiation/

Cheers!

Make mutability clearer in the documentation

The usage documentation/examples should make it clearer that builders are mutable.

Same request but with different responses

When reading the documentation, I fail to find a solution to how I can register different responses from the same combination of http method and uri.

I want to mimic the pattern:
GET /items -> empty list
POST /items -> 201
GET /items -> 200 list with the new item

Pardon, if I have overlooked something.

Add exception list for urls to ignore when ThrowOnMissingRegistration is true

Is your feature request related to a problem? Please describe.

I'm using httpclient-interception and WireMock.Net together.

The scenario:

My application talks to an external (central) API (SaltBae), and from SaltBae I receive a list of different servers (SteakServers). In my integration tests, I intercept all requests going to SaltBae with httpclient-interception and all SteakServers are mocked and created by using WireMock.Net. The HttpClient used for the testing is created from _clientOptions.CreateHttpClient(); and is injected with Autofac in my test setup. This all works nicely so far.

The problem:

However, to make the above work I had to set ThrowOnMissingRegistration to false because httpclient-interception would throw on requests that are meant to be handled by WireMock.Net. But I would still like to have ThrowOnMissingRegistration = true while it ignores any requests that go to any of the Steak servers. ThrowOnMissingRegistration is very useful while writing tests as it shows exactly which requests still need to be mocked.

Describe the solution you'd like

Basically use the fluent API for creating requests that are then added to a special property IgnoredRequests, which are then taken into account before throwing an MissingRegistrationException

_clientOptions = new HttpClientInterceptorOptions
{
    ThrowOnMissingRegistration = true,
};

        new HttpRequestInterceptionBuilder()
            .Requests()
            .ForHttp()
            .ForHost("localhost")
            .ForPort(-1)
            .IgnoringPath()
            .IgnoringQuery()
            .RegisterWith(_clientOptions.IgnoredRequests);

or a simpler solution might be a property that set all requests to localhost to be ignored by the MissingRegistrationException

Describe alternatives you've considered

I tried the following to whitelist all localhost requests but that still intercepts the requests and doesn't return any response

new HttpRequestInterceptionBuilder()
    .Requests()
    .ForGet()
    .ForHttp()
    .ForHost("localhost")
    .ForPort(-1)
    .IgnoringPath()
    .IgnoringQuery()
    .RegisterWith(_clientOptions);

I also tried moving away from httpclient-interception and replace that with WireMock.Net but that caused more problems as the urls werent intercepted anymore and httpclient-interception is just exactly what I need.

Additional context

Nope

Update documentation to work with HttpClientFactory

Once https://github.com/aspnet/HttpClientFactory releases as part of ASP.NET Core 2.1, update the documentation to show its use with the library to set up DI registrations.

This should greatly simplify the boilerplate required as well as the amount of "getting started" documentation required in the README.

NullReferenceException when Deregister builder with a custom matcher

Hi Martin,

Deregister a builder from HttpClientInterceptorOptions object has not work correctly. This edge case has occurred when a custom matcher was added to a builder and then tried to deregister it from options.

This exception is raised because it should be configured matcher before calling buildKey in the Deregister method.

Here is a sample scenario for this bug:

var builder = new HttpRequestInterceptionBuilder()
    .Requests().For(async (request) => await Task.FromResult(true));

var options = new HttpClientInterceptorOptions()
    .Register(builder);

options.Deregister(builder); // NullReferenceException is raised

Fix remote certificate is invalid for https

Is your feature request related to a problem? Please describe.

How-to fix System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch ?

Describe the solution you'd like

https://stackoverflow.com/questions/72211623/how-to-fix-remote-certificate-is-invalid-according-to-the-validation-procedure
https://stackoverflow.com/questions/9983265/the-remote-certificate-is-invalid-according-to-the-validation-procedure
ServicePointManager.ServerCertificateValidationCallback += (o, c, ch, er) => true;

Describe alternatives you've considered

ForHttpsIgnoreCertificate method

Additional context

Update README for bundles

Before 2.0.0 is published to NuGet, update the README to include information about bundles (see #72).

Add ability to add multiple hosts

The Problem

I have some tests that run across multiple environments and I would like to codify that into the test. Initially I've left of the ForHost declaration but that introduces potential issues with it running against any service that matches the other builder routes configured.

Proposed Solution
I'm just wondering if there's been thought/discussions around adding the following overload to enable you to configure multiple hosts:

ForHost(params string[] host)

Alternatives

One alternative I was considering (which I feel could be a better option) is a regular expression overload as specifying multiple hosts feels a bit inconsistent with the idea of mocking out an HTTP endpoint (as opposed to many, which host implies).

Would be happy to contribute the PR if this is something you feel is valuable outside of my single use case.

Async RegisterBundle() overloads

Consider adding asynchronous variants of the RegisterBundle() methods in BundleExtensions.

Add source link support

As per a number of other Just Eat libraries, should add support for Source Link embedding to improve "debuggability".

Support multiple builder registration

To make it easier to register multiple requests at once, should add an registration overload that supports, for example, IEnumerable<T>.

There's maybe also mileage in some sort of clone/copy functionality so you can use one builder to seed others without needing to worry about conflicting result object registrations etc.

Improve custom matching

Issues #247 and #248 have highlighted that making conditional behaviour can be too difficult and unintuitive to achieve.

Consider implementing additional functionality to make it easy to add extra matching conditions to requests so that registrations for the same URL can be matched depending on arbitrary conditions. For example this could be used to cause fuzzing by having requests randomly fail.

Something like this where 50% of requests on average get rate-limited:

var builder200 = new HttpRequestInterceptionBuilder()
    .Requests()
    .ForHttps()
    .ForHost("api.github.com")
    .ForPath("orgs/justeat")
    .AndFor(() => DateTime.UtcNow.Millisecond % 2 != 0)
    .Responds()
    .WithJsonContent(new { id = 1516790, login = "justeat", url = "https://api.github.com/orgs/justeat" });

var builder429 = new HttpRequestInterceptionBuilder()
    .Requests()
    .ForHttps()
    .ForHost("api.github.com")
    .ForPath("orgs/justeat")
    .AndFor(() => DateTime.UtcNow.Millisecond % 2 == 0)
    .Responds()
    .WithStatus(HttpStatusCode.TooManyRequests)
    .WithJsonContent(new { error = "Too many requests" });

var options = new HttpClientInterceptorOptions()
    .Register(builder200)
    .Register(builder429);

Support conditional interception based on the body

Expected behaviour

Enough infrastructure should be available in the library so that a user may elect to read the body before it is sent (at their own risk, for example if it is in a write-only stream) and inform the interception message handler to intercept (or not) a request.

Actual behaviour

While it is possible to intercept the request before it is sent and inspect the body, it is not possible to change whether or not the request is intercepted.