seglo/connect-prism

Record, mock, and proxy HTTP traffic as middleware for the connect middleware framework.

connect-prism's Issues

[question] Is it possible to remove dependency on di?

Hey,

Just opening up the discussion on removing the di dependency. It depends on a very old version of traceur that is causing problems in other plugins when running through gulp, due to a bug in traceur at that version (0.0.33 vs current version of 0.0.111). I've got some workarounds for fixing it, but not working around might be a plus too.

No biggie if not, love the tool, thanks for the work!

Collision with compiled Traceur runtime in Node Enviroment

Please reference: sindresorhus/gulp-traceur#46 (comment)

I am using var prism = require('connect-prism'); in a task name connect.js. In a separate task, if I var traceur = require('gulp-traceur'); it causes the compiled traceur in gulp-traceur to complain about the traceur runtime environment. But if I remove var prism = require('connect-prism');, I do not have this issue.

This goes the same with inside the node shell:
with var prism = require('connect-prism');

> var prism = require('connect-prism');
undefined
>  var traceur = require('gulp-traceur');
TypeError: undefined is not a function
    at eval (eval at <anonymous> (/home/one/github/encore-ui-template/node_modules/gulp-traceur/node_modules/traceur/src/node/traceur.js:24:17), <anonymous>:498:17)
    at eval (eval at <anonymous> (/home/one/github/encore-ui-template/node_modules/gulp-traceur/node_modules/traceur/src/node/traceur.js:24:17), <anonymous>:601:3)
    at eval (native)
    at Object.<anonymous> (/home/one/github/encore-ui-template/node_modules/gulp-traceur/node_modules/traceur/src/node/traceur.js:24:17)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
> 

---

without var prism = require('connect-prism');

>  var traceur = require('gulp-traceur');
undefined
> 

Test failed on Windows

When i try to run the tests, i got :

  2 failing

  1) prism task initialization clear mocks on start "before each" hook:
     Error: EPERM, operation not permitted 'D:\connect-prism\mocks\clearOnStartTest\foo.json'
      at Object.fs.openSync (fs.js:427:18)
      at Context.<anonymous> (D:\connect-prism\test\integration\prism-init-spec.js:78:12)
      at callFn (D:\connect-prism\node_modules\grunt-mocha-test\node_modules\mocha\lib\runnable.js:223:21)
      at Hook.Runnable.run (D:\connect-prism\node_modules\grunt-mocha-test\node_modules\mocha\lib\runnable.js:216:7)
      at next (D:\connect-prism\node_modules\grunt-mocha-test\node_modules\mocha\lib\runner.js:258:10)
      at Object._onImmediate (D:\connect-prism\node_modules\grunt-mocha-test\node_modules\mocha\lib\runner.js:275:5)
      at processImmediate [as _immediateCallback] (timers.js:330:15)

  2) mock filename generator should support a human readable filename generator function with truncation:
     TypeError: Cannot read property 'length' of undefined
      at Context.<anonymous> (D:\connect-prism\test\unit\mock-filename-generator-spec.js:65:48)
      at callFn (D:\connect-prism\node_modules\grunt-mocha-test\node_modules\mocha\lib\runnable.js:223:21)
      at Test.Runnable.run (D:\connect-prism\node_modules\grunt-mocha-test\node_modules\mocha\lib\runnable.js:216:7)
      at Runner.runTest (D:\connect-prism\node_modules\grunt-mocha-test\node_modules\mocha\lib\runner.js:373:10)
      at D:\connect-prism\node_modules\grunt-mocha-test\node_modules\mocha\lib\runner.js:451:12
      at next (D:\connect-prism\node_modules\grunt-mocha-test\node_modules\mocha\lib\runner.js:298:14)
      at D:\connect-prism\node_modules\grunt-mocha-test\node_modules\mocha\lib\runner.js:308:7
      at next (D:\connect-prism\node_modules\grunt-mocha-test\node_modules\mocha\lib\runner.js:246:23)
      at Object._onImmediate (D:\connect-prism\node_modules\grunt-mocha-test\node_modules\mocha\lib\runner.js:275:5)
      at processImmediate [as _immediateCallback] (timers.js:330:15)


npm ERR! Test failed.  See above for more details.
npm ERR! not ok code 0

Related to nodejs/node-v0.x-archive#6599 ?

Recording decompressed gzip responses

The services I want to record return gzipped JSON, making it difficult to adapt the data.
Could you implement an option to decompress gzipped response data if set (or do it by default)?

Anyways, thanks for this very useful middleware! ;-)

Human readable filenames for recorded responses

It would be great if the responses would be stored with their URL as filename. This would help finding the special response you might want to manipulate.
Special characters like the forward slash could be replaced to generate a filename that's still very similar to the original URL.

client and server on different ports?

does this library allow for the server to be on a different port as the client?

I'm running into some issues, and I think this might be the problem...

finer granularity on defining mock responses

Currently connect-prism defines requests/responses purely based off of sha1 of the request url. This is constraining in most use cases where request body data dictates data returned. Hashing based off of GET and body params as well as url would be much more effective at recording / reproducing e2e test output.

mock response file for request with query params

Issue
I try to mock a response for a HTTP GET with two query parameters. When prism tries to create a serialized response file, the "&" between query parameters is replaced with "?", but Windows 10 does not allow to create files with this character. So prism can not create the file and thus can't mock the response.

Code:
let params = { params: { lang: 'en', type: 'car' } };
return this.$http.get("service/vehicles", params).then(res => res.data);

Error:
info: Serialized empty 404 response for /service/vehicles?lang=en&type=car to mocks\serve\service\_service_vehicles_lang=en?type=car_8002ccd3109ba61c96894a5a5f04b68e0636b73a.json.404 fs.js:60
throw err; // Forgot a callback but don't know where? Use NODE_DEBUG=fs
^

Error: ENOENT: no such file or directory, open '...path....\mocks\serve\service\_service_vehicles_lang=en?type=car_8002ccd3109ba61c96894a5a5f04b68e0636b73a.json.404' at Error (native)

I already tried to create the file manually, with "&" between the params, but prism does not see it.
Is there any solution?
Any help is highly appreciated.

ignoreParameters let prism crash on windows

A small bug in the MockFilenameGenerator causes prism to crash under windows as soon as the URL to mock contains a question mark (so basically all requests with query parameters) and the ignoreParameters flag is used .

Reason: the sanitizing of the url is done before the filtering for ignored parameters:

var url = prismUtils.filterUrl(config, req.url.replace(/\/|\_|\?|\<|\>|\\|\:|\*|\||\"/g,'_'));

But in the filter-method, the question-mark is added again:

if (!_.isEmpty(filteredParameters)) {
  urlPath += "?" + filteredParameters.join("&");
}

As the question mark is forbidden in windows file names, prism crashes the hard way when it tries to create the file.

Another side effect of this bug is that the first parameter will never be ignored, as after the sanitizing it is seen as part of the base-URL by the filterUrl() method

In my opinion the expected result in test

it('should support a human readable filename generator function and support ignoreParameters configuration', function() {

is currently not correct. With ignoreParameters = true all parameters should get ignored.

This pull request (#66) fixes the bug. I wrote a test for the question-mark issue and adapted the test mentioned above according to the specification in README.md.

Ignore some param

Some plugins (like select2) add param &_=current_time_here to prevent caching, which uses prism useless in some scenarios. I think we need option to ignore such parameters.

Remove change origin config from examples in README

No longer required with new version of http-proxy

Option to clear mocks from mock directory on start

Conflict with grunt-sass

Gruntfile.js

var prism = require('connect-prism')

module.exports = function(grunt) {
  grunt.loadNpmTasks('grunt-sass')

  grunt.initConfig({
    sass: {
      dev: {
        files: {
          'foo.css': 'foo.sass'
        }
      }
    }
  })
}

Output:

Fatal error: Path must be a string. Received undefined

node v4.2.1, npm 2.14.7

Option to not save on error

It would be nice to have the option to not save mocks on error. A typical scenario is if you've set to mockrecord (in case you hit the API), then the API is temporarily down and you end up with a lot of errors, despite the API coming up again (before you realize why). It's extremely rare that we want to save mocks for errors.

hashFullRequest break the proxy

When i set hashFullRequest to true and try to mockrecord a request with a request body, the proxy wait forever.

It seems that the call of mockFilenameGenerator.getBody(req); in proxy.js break prism.server.web(req, res); behaviour.

Any relation with http-party/node-http-proxy#667 ?

The rewrite method doesn't work

When creating a prism with rewrite it doesn't have any impact on the requests. (using it as gulp plugin)

Don't record 500 or proxy error responses

While in record mode it may make sense to not record actual 500 level responses from the server or when the proxy server reports a timeout or error.

Act as a true web proxy

As connect-prism features are pretty good, I would like to use connect-prism to setup a true web proxy for my Karma tests. Actually, connect-prism is more a kind of reverse-proxy than a proxy.

My requirement ? "target url" should be read from the client request instead of being hard written in Gruntfile.

Set StatusCode through URL

Hi there,

Is there a way to get a different response from the same URL?

For example: I've got /api/login and I want to get a 200 for a successful login and a 401 when login failed.

Thanks for your help.

Regards,
Julien

connect-prism depends on grunt

for starts, thanks for this grate package it rocks.
I have a request, this package adds a huge bundle of grunt dependencies, including coffee script, as I use glup this is a big over head.
is it really necessary? can't you do without it?

Thanks,
Shay

Mix with dynamic data

Is it possible to override/add some data like data/time info to static mock data? Or to use placeholders in JSON data?

Default filename generator ignores HTTP verb

When using the default function for generating a mock filename, the HTTP verb used in the request is not considered. It would be useful to consider this - many REST API's use POST/DELETE/GET on one path to respectively create, delete or get a list of entities.

Relying on the request payload (using hashFullRequest) isn't sufficient, since the request payload can be equal and yet the verbs could be different.

In case this isn't desirable for backward compatibility reasons, maybe it could be an option (just like hashFullRequest)?

Am i misusing connect-prism? Can it do proxying and recording for real 3rd party APIs?

What I'm trying to achieve:

Record real API responses to mock them to the whole flock of apps for stress/latency testing.

I've setup things from example the following way:

prism.middleware set for record/replay
the proxying middleware that should return the real response if prism hasn't recorded it yet
the microservices are setup to use this prism app as a proxy

the mocks folders are created but they're empty, while the requests actually work and return the real api data

source and example output below

var connect = require('connect');
var http = require('http');
var prism = require('connect-prism');
var request = require('request');
prism.useVerboseLog();

prism.create({
    name: 'ok.ru',
    mode: 'mockrecord',
    host: 'api.ok.ru',
    context: '/',
    https: true,
    delay: 'slow',
    port: 443,
    mocksPath: './mocks/apiokru',
    hashFullRequest: true,
    humanReadable: true
});

prism.create({
    name: 'api.vk.com',
    mode: 'mockrecord',
    host: 'vk.com',
    context: '/',
    https: true,
    delay: 'slow',
    port: 443,
    mocksPath: './mocks/vkcom',
    hashFullRequest: true,
    humanReadable: true
});

var app = connect()
  .use(prism.middleware) // middleware
  .use(function(req, res){ // proxying recording if not mocked
      console.log(req.method.toLowerCase(), req.url);
      req.pipe(request[req.method.toLowerCase()](req.url)).pipe(res);
  });

http.createServer(app).listen(3100);

prism_1        | [nodemon] restarting due to changes...
prism_1        | [nodemon] starting `node --debug=3010 index.js`
prism_1        | Debugger listening on [::]:3010
prism_1        | info: Prism created for: / to api.ok.ru:443
prism_1        | info: Prism created for: / to vk.com:443
prism_1        | get https://api.ok.ru/fb.do?access_token=tknXXXX... (lots of similar real request urls that return data)

Can this task be accomplished with connect-prism? I need the delay feature. Any advice?

TY in advance.

Image record do not work

Hi,

I noticed that images binary data were recorded after a .toString() conversion.
Mocks of images requests resulted in display issues in browsers.

Please find a PR that fix this problem : #55.

Allow header configuration

Created as per comment #9 (comment)

The current configuration only plucks certain headers (content-encoding, content-type and location from what I can see).

I think it would be better to allow the user to control this. Perhaps something like I'm doing for issue #8 where we have a headers field and;
false // don't record any
true // record all
array // only record these / only exclude these

Or else you could just record all headers.

Sometimes no mock created when part of the url start with digits

Issue
When I call the api with part of url starting with digits, sometimes I have no mock recorded

• "/v1/business-rules/?client_id=2" => create a mock file
• "/v1/business-rules/6/" => no mock file created
• "/v1/business-rules/8/" => no mock file created
• "/v1/business-rules/3/" => no mock file created
• "/v1/business-rules/7/" => create a mock file
• "/v1/business-rules/28/" => create a mock file
• "/v1/business-rules/33/" => no mock file created
• "/v1/business-rules/1/" => create a mock file
  ...

I've tried to launch same queries in different orders, the result is the same.

Context:
I have an api who works like this:
"/v1/business-rules/?client_id=3" => list of items
"/v1/business-rules/6/" => data for item with ID: 6
"/v1/business-rules/4/" => data for item with ID: 4
...

Code

prism.create({
  mode: "mockrecord",
  host: ".......",
  port: 443,
  mockFilenameGenerator: "humanReadable",
  mocksPath: "./server/mocks",
  delay: "auto",
  https: true,
  name: "business_rules",
  context: '/v1/business-rules/',
  ignoreParameters: true,
});

Grunt peerDependency, why?

Heya, I wanna try this one in a gulp-based project (ish), but it wants grunt to be installed as a peer, which I don't want, it doesn't even make sense to me :)

Can it be removed?

Cheerios!

---> https://github.com/seglo/connect-prism/blob/master/package.json#L55

grunt should be a dev dependency

I have noticed grunt is added as peer dependency. Can you change it to use dev dependency unless their is a reason behind it? People who are using Gulp no needed to have grunt in their project as a dev dependency

Extra mock files created using humanReadable + ignoreParameters

I'm using the ignoreParameters option and setting the mock filename to be humanReadable.

I expected to see one file created after submitting two requests where the only difference between requests is the value of an ignored parameter. That wasn't the case, instead the filename is generated with the parameters included and so two mock files are created.

A brief look at the source code made me think mock-filename-generator.js:34 should use a prismUtils.filteredUrl() instead of the URL right out of the request?

Thanks!

proxy is removing content-length from header

We're using prism to add a specific header field to requests. When loading files the content-length header field is removed which causes broken downloads. This happens only if Content-disposition attachment; is set.

This could be an issue with prism, because accessing the server directly via postman worked. When accessing via prism the content-length field was missing.

Deprecate scheme/host/port config in favour of URL string

connect-prism doesn't use existing mocks

Issue

Connect-prism doesn't use previously committed (and renamed, for human readability) mock files. Of course, this files have been created on another computer, by another developer, in the same git project.

Workaround

For now, we have to :

1. run connect-prim in mock, mockrecord or record mode in order to get the 404 files or a new recorded mock
2. if needed, remove the 404 extension, then replace the content of this mock with the one previously committed
3. ignore this new mocks in every commit after that

This workaround needs to be repeated every time a mock is created or modified.

Context

As I used BrowserSync in the first place, and wanted to use connect-prism, my gulp "serve" task create a BrowserSync server, proxying requests to the backend (thanks to http-proxy-middleware) to a connect server wich use connect-prism for mocking.

The mocks are stored in the mocks/ folder.

I use :

• connect-prism 1.0.6
• browsersync ^2.12.5
• http-proxy-middleware ^0.17.1

You can find all the configuration in Sedona-Solutions/sdn-angularjs-seed (gulp/tasks/serve.js)

MockRecord mode doesn't work with hashFullRequest

When MockRecord mode tries to find a mocked request using hashFullRequest, it cannot properly do so because the body has not been read.
Thus the filename that is being searched will be based on req.body (undefined) + reqData.

Wrong dependency for di selected

Hi,

In version 1.0.5 the di dependency was changed from "2.0.0-pre-14" to "^2.0.0-pre-14"
(0192cda#diff-b9cfc7f2cdf78a7f4b91a753d10865a2R46)
What was the intent of the change? Right now, there is no newer di version and on my system, [email protected] is selected.

1265 silly addNameRange number 2 { name: 'di', range: '>=2.0.0-pre-14 <3.0.0', hasData: true }
1266 silly addNameRange versions [ 'di',
1266 silly addNameRange   [ '0.0.0',
1266 silly addNameRange     '0.0.1',
1266 silly addNameRange     '1.9.0',
1266 silly addNameRange     '2.0.0-pre-1',
1266 silly addNameRange     '2.0.0-pre-2',
1266 silly addNameRange     '2.0.0-pre-3',
1266 silly addNameRange     '2.0.0-pre-4',
1266 silly addNameRange     '2.0.0-pre-5',
1266 silly addNameRange     '2.0.0-pre-6',
1266 silly addNameRange     '2.0.0-pre-7',
1266 silly addNameRange     '2.0.0-pre-8',
1266 silly addNameRange     '2.0.0-pre-9',
1266 silly addNameRange     '2.0.0-pre-10',
1266 silly addNameRange     '2.0.0-pre-11',
1266 silly addNameRange     '2.0.0-pre-12',
1266 silly addNameRange     '2.0.0-pre-13',
1266 silly addNameRange     '2.0.0-pre-14' ] ]
1267 silly addNamed [email protected]
1268 verbose addNamed "2.0.0-pre-9" is a plain semver version for di
1269 silly mapToRegistry name di
1270 silly mapToRegistry using default registry
1271 silly mapToRegistry registry https://registry.npmjs.org/
1272 silly mapToRegistry uri https://registry.npmjs.org/di
1273 verbose addRemoteTarball https://registry.npmjs.org/di/-/di-2.0.0-pre-9.tgz not in flight; adding
1274 verbose addRemoteTarball [ 'https://registry.npmjs.org/di/-/di-2.0.0-pre-9.tgz',
1274 verbose addRemoteTarball   'b51fb4c3a7a1cb231396e1abce4f0ecfe187b6df' ]

The bad thing is that this version fails to install due to its dependency on "traceur" specified as "vojtajina/traceur-compiler#add-es6-pure-transformer-dist".

So right now I can do an "npm install [email protected]" but not "npm install [email protected]".

shouldRecord does not allow checking of response body

If I want to prevent recording of server response based on body. e.g.

statusCode: 200, body: { error: "No server" errorDescription: "Server is not serving" }

Since response body can only be captured thru a callback, shouldRecord does not work in the given scenario.

useapi() doesn't work after creating a prism server with the POST /_prism/create method

i would like to use the useapi() feature.
But when I enable it by using prism.useApi() ( just after prim = require('connect-prism') , and after created a prism server ( with POST /_prism/create ), i can't access to the API anymore.

the prism server proxifies the request ( like GET http://myprismConnectserver:prismPort/_prism/version ), and response with a 404 .

am'i doing it in the wrong way ?

Consequent requests

Need an ability to distinguish consequent requests within one tests.
Now it's recorded into one mock, because url and query string are the same.

Grunt is a full dependency, not a devDependency

You still require grunt in lib/prism.js and use it for limited file handling elsewhere in the file. In a project that does not use grunt, this causes an error:

Error: Cannot find module 'grunt'
...  at Object.<anonymous> (node_modules/connect-prism/lib/prism.js:8:13)

TypeError on POST in v.1.0.11/12

Issue
We use prism in proxy mode to redirect REST requests form an angular front end to a back end in docker. When doing a POST in the new version 1.0.12 (or 1.0.11) we got the following error:

TypeError: must start with number, buffer, array or string
    at fromObject (buffer.js:218:9)
    at new Buffer (buffer.js:65:10)
    at placeholder\node_modules\connect-prism\lib\modes\proxy.js:19:28

Reproduce

curl 'http://localhost:8088/bpm/activiti/save' -H 'Accept: application/json' -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"taskId":"50584","properties":[{"id":"abort_video","value":false}]}'

Reason
In api.js req.body is set to an object:

      try {
        req.body = JSON.parse(body);
      } catch (err) {}

Now later in proxy.js, the same req.body is a constructor parameter for a Buffer object.

 if (req.bodyRead) {
      //re-stream the request body, because it has already been consumed
      req.removeAllListeners('data');
      req.removeAllListeners('end');

      process.nextTick(function() {
        if (req.body) {
          req.emit('data', new Buffer(req.body))
        }
        req.emit('end')
      });
    }

But a buffer can not get initialized with an object (see error).
The Bug only appears in v 1.0.11/1.0.12, with v 1.0.6 the POST never runs into this error (aka never reaches this line in proxy.js).

I'm not sure which change caused the problem, but my guess would be Support hashFullRequest for mockrecord mode

We temporary downgrade the prism version to 1.0.6, but it would be nice to use the latest one. Let me know if I can help!

Mock file names can get so long that they break Windows

When a request has mulitple query strings the human readable file names can get really long.

My colleagues who use Windows or Ubuntu (with directory encryption) are unable to checkout a branch that contains mocks with those long names.

Maybe limit the length of the generated file names.

Proxying to proxy

It would be great if connect-prism could forward received requests to another proxy.

Use case: to work with a webapi behind a proxy with authentication

Maybe this can be achieved with toProxy option from node-http-proxy ? (see https://github.com/nodejitsu/node-http-proxy#options too)