Support for validating API response messages against model schema

[apigee-127]

Support for validating API response messages against model schema

[JamesMessinger]

The problem with that is that some responses are outside of my control (i.e. I might be using third-party middleware for authentication, data retrieval, error handling, etc.) so there's no way to change those middleware to use res.validateAndSend instead of res.send. I think a better way might be to allow an opt-in (rather than opt-out). Like this:

app.use(swaggerValidator({ validateResponse: true }));

And swagger-tools would only do the monkey-patch if validateResponse is true.

[grahamj]

Agreed! In fact outgoing schema validation is on my ToDo for our project so it would be great to have it here.

[whitlockjc]

This is now one step closer to being complete. spec.composeModel(apiDeclarationOrSwaggerObject, modelIdOrModelPath) now returns a composed JSON Schema for the requested model. This can be used to validate a structure representing a model. I will be hooking this in where possible but just know you can use this after the next release is dropped.

[grahamj]

Using this now, it works pretty well, thanks!

[whitlockjc]

It's not extremely well tested and there is one part I don't think should work but I won't mention it until someone else brings it up. I did a lot of personal testing and the unit tests test a few different scenarios but I've not tested every possible way to compose models. Keep me in the loop.

[grahamj]

Our models are fairly straightforward so hopefully I won't run into said issues. I'm guessing inheritance could get tricky so I'm purposefully keeping things simple.

[whitlockjc]

Ironically, inheritance is covered the best. It's the model references in properties, like arrays of Pets, that worries me. I tested multiple levels deep.

[grahamj]

Ah ok, I only do that in a couple of places.

[grahamj]

FYI what I'm doing is monkey-patching res.send to get the outgoing content then validating that. I call composeModel with req.swagger.apiDeclaration and either req.swagger.operation.type or if that's "array" then req.swagger.operation.items.$ref.

For a more general implementation I'd also check operation.type for primitives and patch into your validator for that but we only ever return models and arrays so I didn't bother.

[whitlockjc]

So this is all on your side right? I should be handling all of that on the way in of course. Primitives, models, models with arrays of refs, models with refs, etc.

[grahamj]

It is, yes. I thought about adding to swagger-tools so I could submit it back to you but I didn't have much time and probably didn't do it in a way that would be ideal for other users. I'd be happy to pass along what I've got if you want to use it as the basis for output validation middleware.

[whitlockjc]

This was completed as part of #18. swagger-router will now validate request parameters that are models. (For 1.2, you can even have request parameters as an array of models and that is handled too. For 2.0, arrays are documented explicitly to disallow models so for 2.0, we do not handle that.) There is also a new API to call this programmatically outside of swagger-router: Specification#validateModel

[JamesMessinger]

Looks like the new commit (for issue #18) adds request validation (which is awesome!) but not response validation (which is what grahamj is referring to). For the time being, I'm doing the same monkey-patch of res.send that grahamj is doing, but it would be great to have swagger-tools do that automatically.

[whitlockjc]

Ah, glad you caught that. I've reopened the issue and will add support for this. What do you think about adding a new method to res that way you can easily opt out if something doesn't work right? I'd like to add some res.validateAndSend(data) or something like that. Thoughts?

[whitlockjc]

Thanks for the feedback. I'll figure out some way to reuse res.send and some sane way to disable it when necessary. Of course, any feedback/suggestions are welcome.

[jcferrer]

Can somebody please share more details as to how validating a response?. From what I can tell some of you have found an interim solution. A code snippet might help.

Thanks

[whitlockjc]

@jcferrer How would you want this to work? Would you want res.send to validate the outgoing payload with some option to turn it off as described above? If so, I'll knock it out right now.

[jcferrer]

Personally, I like the proposed option of:
app.use(swaggerValidator({ validateResponse: true }));

or something like it.

Thanks

[whitlockjc]

That's what I was suggesting. Here's how it would work:

1. A configuration option for swaggerValidator middleware to turn on/off response validation
2. Create a wrapper for res.send that will use that configuration option to either validate the response or use the wrapped res.send

Do you think response validation should be on by default?

[jcferrer]

I would say so. The whole spirit behind the tools is to provide decoupling. So I am assuming that you would want to enforce the spec by default.

Thanks

[jcferrer]

I ran the question through some people and some think that it should be off by default. I think as long as the option is there, we are all good.

[whitlockjc]

I appreciate your help. No guarantee but it shouldn't take very long to get this done...when I get back from my run. :)

[whitlockjc]

Whenever there is an invalid response, what would you prefer the error code be?

[whitlockjc]

Mentioning you so you get the message, just to be sure: @euoia @grahamj @BigstickCarpet @jcferrer

I've got this working for Swagger 1.2 locally and before I get Swagger 2.0 support working with this as well, I wanted to run something by you. First thing is that res.send is not a Node.js/Connect thing so my implementation wraps res.end. Since res.send calls res.end, this should be fine. My tests thus far are these:

• Validate response content type
• Validate response type (void, primitives, array of primitives, models, array of models)
• Validate response type constraints (maximum, minimum, enum, uniqueItems, etc.)

All response validation errors get a 500 status code and all error messages, sent downstream via typical Connect middleware fashion, has a message starting with Response validation failed so that the consumer knows the cause of the issue. (Anytime middleware validation fails, it also throws errors, warnings and possibly apiDeclarations properties on the Error object as well so use those however you need downstream, my middleware does.)

Thoughts?

[jcferrer]

Based on your notes, my understanding is that we should be able to trap the response validation errors via a simple handler like:

function errorHandler(err, req, res, next) {
// handle response error 500;
}
app.use(errorHandler);

I think this is a good approach. Thinking in some use case scenarios, I'd say this provides the most flexibility.

[whitlockjc]

That is correct. All Swagger middleware errors are sent downstream via next and that allows you to use an error handler to do something more with the error. All validation errors have the validation results attached to error properties. I'll document this in the next release.

[avishaan]

This is awesome!

[avishaan]

Can we consider having an option to still send the response? It would make debugging a little easier.

For instance right now I am going back and forth between my swagger spec and the response and my controller to figure out where the problem is. Not having easy access to the response that would have been sent had it not failed would seem to be important. Error.body or Error.originalResponse.

[whitlockjc]

That's a great idea. If the above issue isn't what's described above, please file an issue.

[ovr]

BTW I started to work on similar idea on https://github.com/ovr/swagger-assert-helper, for PHP

Now It's supports

• Asserting Swagger Response Scheme to HttpResponse
• Making HttpRequest by Swagger Path

Still working on it, maybe will be helpful for someone