Question: How to reuse the same mocked APIs to run Jest tests?

[andreawyss]

Question: How to reuse the same mocked APIs to run Jest tests?

[kettanaito]

I've published the logic that allows to intercept HTTP/HTTPS requests in NodeJS, called node-requests-interceptor.

During the implementation I've found and referenced multiple similar purposed libraries, yet they all seem to provide a high-level API (include request matching, routing, etc.). My goal was to have a low-level interception API, which I think I've successfully achieved.

As the next step, I'll use that library to kick off the interception and resolve mocked responses using request handlers as a part of setupServer in msw. Stay tuned.

[kettanaito]

Hi, Andrea. Thank you for brining this up, it's a good question.

Since Service Workers operate in a browser environment, it makes browser a primary operation surface for Mock Service Worker. Although modern test frameworks like Jest allow to run tests in DOM-like environment, that doesn't come with all the browser features implemented. As of now, you cannot reuse the mock definition of MSW in a node/js-dom environment. However, I'll try to elaborate on what you can do, as well as brainstorm a little on how to achieve what you want on the library's side.

Consider browser environment

I understand this may not be suitable for some use cases, but if you can, consider running a test in a browser environment. Tools like Cypress or Puppeteer provide a superb API to run about any piece of logic in a browser, be it a UI component, or a standalone function.

You can also refer to the MSW testing setup, which utilizes Jest+Puppeteer.

Personally, I'm a fan of Storybook, so, assuming you're resting a UI component, an ideal setup would be a dedicated story + running a browser automation tool on top of it. You can have a per-story mock definition, or import the mocks on a top level of Storybook, to affect all stories (which I would recommend to prevent activation of multiple Service Workers).

Drawbacks

• Browser environment is not suitable for a logic that's not designed for it, while still communicates with an API.
• This effectively makes an E2E test, and for the cases when it's not applicable, it creates an unnecessary setup overload.

How MSW can improve

Provide a dedicated API

One way I can imagine it, is that calling a composeMocks() function would also return a function to spawn a mocking for NodeJS environments.

// test/mocks.js
import { composeMocks, rest } from 'msw'

export default composeMocks(
  rest.get('/user', (req, res, ctx) => res(ctx.status(404))
)

// test/user.test.js
import { startServer } from './mocks'

describe('User', () => {
  beforeAll(() => {
    // Return a Promise so a testing framework may await it
    return startServer()
  })
})

Benefits

• Reusing the same mocking logic for another environment
• NodeJS support

Drawbacks

• MSW would have to figure out an alternative way to intercept request in a non-browser environment. This is usually done by stubbing a global fetch(), which I can't say I like.

---

I don't see anything that would stop MSW from providing an API that would allow you to use the same mocks in a node environment, yet I'd love to discuss the implementation detail of that. Please, if you have your opinion on this, or an idea in mind, share it with me so we can find a suitable solution. Thanks!

[kettanaito]

🎉 I'm so excited to finally release the NodeJS support!

Getting started

• Update to 0.16.0
• See the Usage in Node documentation for instructions and examples.
• See the REST API unit test example with React: https://github.com/mswjs/examples/tree/master/examples/rest-react-jest

What's supported?

• Ability to reuse the same mock definitions in the NodeJS environment.
• Explicit server.listen() and server.close() for establishing requests interception and clean up, respectively.
• Interception of any requests issued by http/https/XMLHttpRequest modules. This includes node-fetch, fetch (in jsdom), and generally about any other request issuing client, as it relies on one of aforementioned native modules.

It's been a huge amount of work, so I'd really appreciate your feedback on this! I know some edges may be rough, and I do expect issues, but I believe we can make this into an amazing developer experience. I'm grateful for your involvement and help!

[kettanaito]

Here's how @kentcdodds currently uses MSW for reusing the same mocking logic for testing in Jest:

https://github.com/kentcdodds/bookshelf/blob/b61e3ea0313a6da7fb648ad810bfd242797054bf/src/test/fetch-mock.js

I believe it can be a useful reference until MSW ships with this natively.

[andreawyss]

I use axios for all the API calls and use the axios-mock-adapter which works great because it works both in the browser and in node.

The drawback with axios-mock-adapter is that when running in the browser one cannot see the calls in the browser network tab, since these are intercepted by the adapter.
This is where 'msw' really shines!

I think we should create an "msw-axios-adapter" to be used only when the code runs in node/Jest process.env.NODE_ENV === 'test'
The adapter will intercept the calls and reply using the same mocked API code that is used by the service worker when the app runs in the browser.

This should make all axios users really happy.
For 'fetch' users, I do not see other solution than stubbing a global fetch

[kentcdodds]

During tests I think stubbing fetch works great. There's no real benefit to making actual network requests in a rest context.

With that in mind, whether your using axios or not, I think mocking fetch (or xhr) directly rather than axios is the better way to go.

[kettanaito]

As of now, #146 concludes the basics for NodeJS support 🎉

I'm planning on releasing that after #157 is fixed, not to transfer any existing issues into the next release. You can try it out on that branch already, but please wait a little once the blocking issue is resolved. Hope to hear a feedback from you soon.

[kentcdodds]

Personally I think it would be best if msw work directly with http rather than something like fetch or axios. That way it can work with whatever people end up using.

[andreawyss]

Thank you @kettanaito for making this happen!

I'm still trying to get it to work inside my project but then I see that axios is not yet supported issue #180.

I have created this tester project that you can use to reproduce the axios vs. fetch issue.
https://github.com/andreawyss/msw-tester
npm i
npm run test fetch works but axios does NOT work.
npm start fetch and axios both work in the browser.

[andreawyss]

@kettanaito what we need is an additional contexts options parameter in setupWorker and setupServer to pass in values that are then made available in the ctx to be used by customs context functions and custom resolvers.

In my case the failRate, failCode and failText are custom configuration values.

Also the default delay amount is another possible custom configuration value used by a ctx.defaltDelay() custom context function.

[kettanaito]

I see! That's a nice use case, I wonder what it implies to make such axios adapter. I'll research the topic and will try to come up with a viable solution. I suppose a custom adapter is a good choice for axios users, and regarding general node usage I'd give it some more thoughts.

Thanks for sharing your usage!

[andreawyss]

Yes. Good point.
Then msw could leverage something like this https://github.com/jameslnewell/xhr-mock or similar to expose a single way to write mock apis code for both browser and node.

[kettanaito]

I think we will go with the support of NodeJS environment in a way of passing the same list of request handlers to a different initialize function.

Usage

Request handlers

// src/mocks/handlers.js
import { rest } from 'msw'

export default [
  rest.get('/user', (req, res, ctx) => {...}),
  rest.get('/login', (req, res, ctx) => {...}),
]

Browser usage

// src/App.js
import { useWorker } from 'msw'
import handlers from './mocks/handlers'

if (process.env.NODE_ENV === 'development') {
  useWorker(...handlers).start()
}

useWorker() is a renamed composeMocks() function (discussion in #100).

Node usage

// test/setup.js
import { useServer } from 'msw'
import handlers from '../src/handlers'

// No detailed thoughts yet over this API
useServer(...handlers)

Technical insights

I believe useServer() can take the same request handlers you have and use them when stubbing XHR. To my best knowledge fetch() uses XHR under the hood, so stubbing it on the lowest level is the most battle-proof approach.

I still dislike an idea of stubbing, but I would like for MSW users to be able to test their non-browser environments as well, while keeping the same handlers logic.

@andreawyss what do you think about this proposal?

[andreawyss]

This is great and it is exactly what I was looking for.
"Write a common set of request handlers that can be used to respond to Browser and Node XHR requests".

For the names of the initialize functions see discussion at issue #100:

• configMockWorker(options)
• configMockServer(options)

For fetch() in Node one still need to handle that with some stub. Maybe leverage other libraries already available to stub fetch in Node like: https://github.com/node-fetch/node-fetch

[kettanaito]

Implementation

I'd like to brainstorm on the implementation of node support for Mock Service Worker.

Stubs

Stubbing Service Worker

• service-worker-mock

If one is able to use the Service Worker API in node environment, there is no need to establish any specific API on the library's side.

Benefits

• No extra API, use the same request handlers and compose functions.

Drawbacks

• Creates a coupling with the Service Worker spec
• May not be fully possible technically, since Service Worker righteously executes in its own environment (it's not even DOM).

Stubbing network

• xhr-mock

When going with stubbing a network communication (either XHR, or fetch), MSW should provide such API that would do stubbing explicitly.

Usage

// test/login.test.js
import { setupServer } from 'msw'
import handlers from './mocks/handlers'

describe('Login', () => {
  let mocks

  beforeAll(async () => {
    mocks = setupServer(handlers)
    await mocks.open()
  })

  afterAll(async () => {
    await mocks.close()
  })
})

Benefits

• If stubbed at the lowest level, can be sufficient in handling requests regardless of their issuing type (XHR, fetch, etc.).

Drawbacks

• Requires to provide a dedicated API from the library's side
• Requires to manage stubbing and cleanup carefully to prevent unexpected side-effects.

Runner plugins

Alternatively, the network communication doesn't have to be stubbed in order to reuse the same request handlers from the library. I was thinking about something like a test runner plugin (i.e. for Jest) that would leverage that testing framework's API to stub resources.

Jest

Advantages

• Node-specific implementation is required explicitly, not being distributed as a part of the library

Disadvantages

• Each test framework would likely need to have its own plugin/adapter, which increases a maintenance surface.

Usage

For example, stubbing a global.fetch = jest.fn(/* resolve request handlers */).

Third-party libraries

There are existing third-party libraries for about any stub you can think of. However, they often come with a built-in API for routing requests, which I would like to avoid, since MSW has its own request matching logic. I'd rather prefer not to perform any mapping from one matching to another, as it increases the coupling between libraries.

If it comes to implementing a stub, I think MSW should create its own stub logic that is open-ended towards what you do with an intercepted request. This can also be published as a separate package.

[andreawyss]

It is best to ask the node.js comunity for some help.

These are my assumptions:

• When one writes code with fetch() then in node one would use https://github.com/node-fetch/node-fetch
• When one writes code with axios then in node axios uses the built in xmlhttprequest module.

Both node-fetch and xmlhttprequest modules use the node's http module.

I recommend researching how to have msw integrate/leverage an existing node's http mock library.

I just found these with a Google search:
https://github.com/nock/nock
https://github.com/moll/node-mitm

There could be other/better ones.

[kettanaito]

I've spin out a little PoC and can confirm that stubbing http/https directly works just great. I think I will provide this functionality as a low-level abstraction in a separate library. Initially there's going to be support for HTTP/HTTPS/XHR, which should be sufficient for most use cases.

I will roll out tests for plain requests, node-fetch, and things like axios to make sure all are covered. I'm expecting them to be, as I'm taking inspiration from nock and other tools, reverse engineering how they're working internally.

[kettanaito]

A quick update on Node support: I've integrated the interceptor library, and see that mocking works in jsdom environment.

Unavailable Headers

Since the Headers class is unavailable in Node environment, the modules that rely on it need to be adjusted in order to run in Node:

• res() uses Headers to set the custom x-powered-by header;
• ctx.fetch() uses Headers to digest any headers input format (#149);
• req.headers reference you have in a response resolver also promises to expose you a Headers instance.

Solutions

1. Drop the usage of Headers

Pros:

• Internal code remains the same for browser/node usage.

Cons:

• Breaking change
• Missing the native handling of headers, such as headers with multiple values

2. Use polyfill

Pros:

• No breaking changes

Cons:

• All drawbacks of polyfills (out of sync with the spec, issues in certain browsers, overall a mocked implementation)
• Larger bundle size for Node usage, including the polyfill

---

Please, if you have any expertise or idea on how to handle this environment deviation, let me know.

[andreawyss]

In most Jest unit tests there is no need to have mocked APIs with headers.

I would document this limitation and ensure that the mock code can still runs with no errors also when there is no support for headers. Maybe just a console log/warn message informing that headers are currently not available in node environment.

If this issue becomes a real problem, many users feel that this limitation is unacceptable, than we evaluate a possible solution at that point in time.

[kettanaito]

@andreawyss, that's a good gradual development strategy. My main concern is that in order to ship Node support, even with limited headers, I'd have to modify the browser-related logic. This is mainly because both browser and server logic uses the same res() and ctx.fetch().

I think the Headers class is primarily designed for creating headers. I can see no practical limitations if the req.headers are represented as Record<string, string | string[]> in request handlers. What do you think about it?

[andreawyss]

It should probably be represented with a Record<string, string> or { [k: string]: string }
Even if some headers may have multiple value separated by some delimiter like "," or ";" I do not know if this is part of the HTTP headers specification.

It should be the responsibility of the consumer to split and join these multi values headers with whatever separator is needed for a certain header.

[kettanaito]

I've decided to use a custom Headers polyfill that's based on the polyfill from fetch. Per your suggestion, @andreawyss, that headers instance is going to keep the values in a string, with multiple header's values separated by comma (","). That is also to maintain a backwards-compatibility with a native Headers class.

The reason for choosing a polyfill was its straightforward implementation and the goal to provide the best developer experience when working with headers. If the usage proves it's more suitable to expose headers as Record<string, string>, there'll be no issue to do so.

[kentcdodds]

Will there be docs on the right way to approach this? I've got my own work here and I'd love to swap it with something more official and built-in.

[kettanaito]

@kentcdodds, that's a good question. I'm on the finish line with a brand new website for the library, that includes re-designed and vastly rewritten documentation, so I'd like not to repeat things. I think I'll add the instructions on how to use MSW in Node to the existing docs, so it's documented, but maybe not in the fine-grained level of detail, which you would get in the next version of the docs. I hope this makes sense.

[kentcdodds]

Yup, makes total sense. Thanks for all your work!

[kettanaito]

@andreawyss, thanks for putting up that reproduction repository! Much appreciated. I'll investigate that issue in more details to see what's wrong. Something is certainly off, as the axios interception test passes.

[kettanaito]

Interception of requests issued by axios should work as expected in [email protected].

[andreawyss]

@kentcdodds is there a better way to test the failure code handling that what I show in this sample?
I'm using a separate test suite for the err tests.
https://github.com/andreawyss/msw-tester

[kettanaito]

@andreawyss you can technically create your own res composition function, that will have a failure rate embedded into it.

// src/mocks/unreliableRes.ts
import { ResponseComposition, response as res, defaultContext as ctx } from 'msw'

const FAILURE_RATE = 0.2

export const unreliableRes: ResponseComposition = (...transformers) => {
  const shouldFail = decide(FAILURE_RATE)

  if (shouldFail) {
    // Respond with a fixed failure response
    return res(ctx.status(500), ctx.json({ message: 'Error!' }))
  }

  // Use the default `res` composition
  return res(...transformers)
})

// src/mocks.js
import { setupWorker, rest } from 'msw'
import { unreliableRes } from './unreliableRes'

setupWorker(
  rest.get('/products', (req, _, ctx) => {
    return unreliableRes(ctx.json([{...}, {...}])
  })
)

Since the call signature of this custom response function is up to you, you can make it accept a failure rate or other options to your liking.

[kettanaito]

@andreawyss, yes, I was thinking about this too. At the moment it's possible to define/extend the context utilities by creating a custom request handler. As of extending setupWorker/setupServer options, I'm not sure, as context utilities are specific to the request handler being used.

I'd say the easiest way is to create your own set of utility functions (you can do it even now), and explicitly import them in the mocks. As long as a utility function has a compliant call signature, you can call your custom utilities as a part of res() composition just fine. I'm elaborating on this in custom context utility.

[andreawyss]

@kettanaito
I solved the configurable setup this way:

1. A mock.utils file where one declares the SetupOptions and custom util functions.
2. Write the MockedResponse generator files like this.
3. Setup the service worker like this.
4. In success tests setup the mock Server like this and in error tests setup the mock Server like that.

With this approach the SetupOptions are available to the MockedResponse generators and are passed to the util functions.

[kettanaito]

Looks solid!

I know there are common things one'd expect when mocking, but I'd be vary cautious here in bringing them into the library's API. I'll try to maintain an API that's straightforward to extend, than ship some built-in context function, for example. I do think about altering certain behaviors, however, like ctx.delay() call without arguments delaying the response by a random realistic latency time.

Nevertheless, please reach out when you feel you miss something, I, by no means, see all the possible usage, so there are things that may be embedded into library's API.

Thank you for those insights!