flickr/yakbak

Record and playback HTTP responses

flickr-sdk

Almost certainly the best Flickr API client in the world for node and the browser

This SDK provides methods and type definitions for all methods listed on https://www.flickr.com/services/api/

To use this SDK, sign up for an API key here

install

$ npm install flickr-sdk

quickstart

Make a Flickr API call

import { createFlickr } from "flickr-sdk"

const { flickr } = createFlickr("<your Flickr API key>")

const res = await flickr("flickr.photos.getInfo", {
    photo_id: '12345',
})

Upload a photo

import { createFlickr } from "flickr-sdk"
import { resolve } from "node:path"

const { upload } = createFlickr({
    consumerKey: "<your API key>",
    consumerSecret: "<your API secret>",
    oauthToken: "<the oauth token>",
    oauthTokenSecret: "<the oauth token secret>",
})

const id = await upload(resolve("example.png"), {
    title: "Works on MY machine!",
})

auth

The Flickr SDK currently supports the following auth methods:

API Key

This is the simplest way to use the SDK. Just provide your API key as a string:

const { flickr } = createFlickr("<your API key>")

OAuth 1.0

OAuth lets users grant your application access and then you may act on their behalf. The OAuth flow is described here.

const { upload } = createFlickr({
    consumerKey: "<your API key>",
    consumerSecret: "<your API secret>",
    oauthToken: "<the oauth token>",
    oauthTokenSecret: "<the oauth token secret>",
})

💡 Use examples/oauth.mjs to quickly set up an OAuth flow and obtain a set of credentials

migrating from previous versions

Previous versions of this SDK depended on superagent for http requests. This version of the SDK uses node's native fetch instead, so you now only receive the response body back from an API call. This means the return value of an API call will only be the response body, not a superagent Request

Migrating existing code looks like this:

//  old
const res = await flickr.test.login()
console.log(res.body)

// new
const body = await flickr('flickr.test.login')
console.log(body)

advanced

configuring fetch

import { createFlickr, FetchTransport } from 'flickr-sdk'

const transport = new FetchTransport({
    headers: {
        'user-agent': 'foo',
    }
})

const { flickr } = createFlickr('<your API key>', transport)

testing

import { createFlickr, MockTransport, NullAuth } from 'flickr-sdk'
import * as assert from 'node:assert'

// mock transport returns the response you pass in the constructor
const transport = new MockTransport({
    stat: 'ok',
})

// null auth does nothing
const auth = NullAuth()

const { flickr } = createFlickr(auth, transport)

// makes no network request
const res = await flickr('flickr.photos.getInfo', {
    photo_id: '12345',
})

assert.deepStrictEqual(res, { stat: 'ok', foo: 'bar' })

yakbak's Issues

add more/better logging to help ease debugging

I'm currently debugging some tests that fail intermittently. The tests hit yakbak for each request and only sometimes have a matching tape. it'd be extremely useful if setting DEBUG=yakbak* when running my test suite would output more than this:

  yakbak:server req +0ms /some/path?with=some&query=strings

would you take a PR that adds a few more debug()s for things like:

• the request path
• the request hash
• if a tape was found
• if yakbak will respond to the request

Yakbak is dead.. Long live Talkback!! [Replacement Library]

I was banging my head against the wall trying to make Yakbak work for my use case. First off the tapes are in base64 format which makes it really hard to debug the tapes & also this library is not maintained anymore.

I came across a replacement called Talkback which serves the same functionality but was written from scratch by @ijpiantanida! 👏

Hence this led me to replace my mock Yakbak server with Talkback:

Yakbak Cons:

• The tapes were in base64 format - made it hard to debug the request/response
• It would re-record tapes when x-request-id changed - an unintended side-effect
• Needed express & morgan libraries in addition to yakbak
• Not maintained anymore

Talkback Pros:

• Easy setup - took less than 2 mins to replace Yakbak
• Logging requests by default - even shows whether it served a cached request or is recording a new tape
• Human-readable tapes - you can see the req/res in JSON5 format
• Ability to rename tapes - this is great as you can know exactly what your tapes your doing

Hence if you are considereing Yakbak for your use case, I would highly suggest you checkout Talkback and use that instead.

Provide live call option

Consider providing a value in the options to always make a live call and not use existing tapes. I have a situation where certain API's may need to be called live whereas others may be replayed. A case could be made that the caller ought to not route to the yakbak server those calls that need to be live. However, for my scenario, the caller is running against dev (with the ability to route to the yakbak server as a toggle option via qs param) and yakbak is setup to record/replay against prod so that the dev client code can be tested against prod data. In this case, the dev client wouldn't be configured to call prod API's, so any calls not routed to yakbak would go to dev instead of prod.

option to create custom hash/tape names

First off, this package is awesome! Great work here =)

I'm currently moving an enormous codebase (all segment.com integrations) away from hitting 3rd party APIs in tests to use yakbak, but I'm hitting issues with APIs that require cache-busting query parameters (example: google-analytics).

Would you be open to a (backwards compatible) addition to the API that allows users to create their own hash function?

"no record" mode that will error on an unrecorded request

This would be super useful on CI where we absolutely don't want to record anything new. This should have the same behavior as vcr's record :none mode.

Allow for rewriting responses before writing tape

Yakbak is great - it's really easy to setup, and just works. It'd be nice if there was a custom hook to modify the response before writing to the tape - specifically in the case where the response returns references to the proxied server.

For example:

1. Using yakbak server to proxy through to some other server
2. Request made, html returned with references to js/css/img assets that point to the original server url
3. Ideally, I can write a function to replace these references with the yakbak proxy so that they are then captured through yakbak as well.

I submitted a PR with an attempt at this here: #27

Expose the `tapename` function

I would like to be able to human-read the names of the tape, and I thought of monkeypatching the tapename function to have more flexibility.

This is not possible however as this function is not exposed.

Is project alive? => New library

First of all, thank you for building this tool. I've been using it for a long time now.

Unfortunately it looks like the project lost a little bit of attention and a lot of good PRs and issues have been open for quite some time. (I've been actually running of my PR's branch for more than a year now)

I decided to create my own version of "yakbak" called talkback to address some of the issues posted here and add some extra features.

If you're planning on still maintaining this project please close this issue and sorry for the spam, otherwise I think it might be a good successor.

Again, thank you for your efforts!

Why not nock-vcr?

Hi there,

In the blog post linked in the README, there's mention of a few approaches to the same problem, including Ruby's vcr library.

I'm curious how yakbak compares with existing node libraries that mimic vcr, such as https://github.com/poetic/nock-vcr-recorder or https://github.com/carbonfive/nock-vcr

What's different or better about yakbak? I think it would be helpful to clarify this in the README, if not a follow-up blog post.

Thanks!

Consider using end-of-line comments in tapes instead of escaping "*/"

It might be a bit more elegant to use "//" style comments instead of "/*", since you'd be able to drop the escaping. This could also allow the data to be deterministically parsed in the future if need be.

(related: #4)

Odd behaviour when connecting to an https backend

I'm attempting to create a proxy to an https API that sits behind Cloudflare:

http.createServer(yakbak('https://--domain--', {
dirname: __dirname + '/tapes'
})).listen(3000);

When I do this, and curl the proxy:

curl -i localhost:3000/status

I get an error from cloudflare "The request's Host header does not match the request's TLS SNI Host header."

I've got a code change for the proxy to fix this; updating the headers collection prior to mod.request rather than using preq.setHeader('Host', uri.host), and this fixes it, but wasn't able to reproduce it with a test.

yakbak('http://host.com/subdir') does not work

In this case:

yakbak('http://host.com/subdir', {dirname});

It seems like yakbak drops the '/subdir' part when it proxies requests and only sends 'http://host.com/'.

Filter sensitive data

vcr has an option for this. I generally prefer to check in tapes so this would be really useful.

Respect Content-Encoding and Content-Type in tapes

It would be fantastic if the tape format took content-encoding and content-type into account when reading/writing tapes so that the recorded request body is human-readable and easily editable.

yakbak sometimes does not record tapes

I sometimes -- often, in fact -- find that when I run tests with no tapes in place, the tests work correctly (falling through to the underlying HTTP service) but no tapes get written. I have no idea what causes this. There are no diagnostics of any kind, just a silent failure -- or, worse, I might characterise it as a silent success, but one that doesn't give me the side-effect that I need.

I realise this is a hopelessly underspecified bug report: it's more of a cry for help, really. I'm posting it in the hope that someone else has had this experience, recognises what I am describing, and can put me on the right path.

The proxy returned from yakbak() should return its Promise chain

This allows us to use express-promise-router to catch any unhandled Promise rejections that happen to occur in the rest of the code.

Plans for using native Promises?

Hi Jeremy,

Do you have any plans to use native promises with your library? Greater node 8 they are pretty much faster then js implementations like bluebird.

Today we encountered a strange problem with bluebird which we could not
nail down to a PoC. While recording a tape the response was not finally written
to disk but the Promise already resolved, such require(file) returned an empty response resulting in a "TypeError tape is not a function". Removing bluebird from lib/buffer.js solved the issue for us.
For the urgency we had to move forward, we created the fork at https://github.com/spurreiter/yakbak-native-promise which has bluebird replaced with native promises. Are you interested in a PR?

Option to record response in different format

Sometimes, when I record an HTTP interaction, I'll want to modify the body of the response later. Usually, I do not want to re-record the entire interaction, I just want to modify the currently recorded response.

One way to do this would be to modify this so that, instead of:

res.write(new Buffer('cafebeef', 'base64'))

we'd have something like:

var obj = '{"v":2}'
res.write(new Buffer(JSON.stringify(obj), 'utf8'))

It could then be surfaced in a configuration, like:

yakbak('http://myServer:3060', {
  dirname: __dirname + '/../fixtures',
  fixtureEncoding: 'utf8', // default would be 'base64'
}));

Is that functionality that would be beneficial? If so, I'd be happy to add it.

Thank you!

Just started using this in a project. So far this is the most seamless http mocking tool I've used.

Thank you for releasing it 😄

Beginner level error: done is not defined

Hello,
I am a beginner in nodejs and require some help regarding the usage of yakbak. Hope, you don't mind :-)

Situation:
At present, I develop ASP.NET webmethods via Visual studio and manually test them. Once I start a visual studio project, the test url becomes: http://localhost/<random-but-valid-port-number>. This link can be opened via any web browser (e.g. chrome, firefox), for which I think node modules can use it also.

Code I tried:
I have started the visual studio project, and it has started on port 52280. Therefore, going to http://localhost/WebService1.asmx/Ping yields a valid timestamp (code written by me which I want to test). Code I have written in index.js:

// Contents of index.js
try {
    var yakbak = require('yakbak');
    var http = require('http');

    var validPort = 52280;

    var proxy = http.createServer(yakbak('http://api.example.com', {
        dirname: __dirname
    }));
    proxy.listen(validPort, done);

    var options = {
        host: 'http://localhost',
        port: validPort,
        path: '/WebService1.asmx/Ping'
    };

    http.get(options, function (resp) {
        resp.on('data', function (chunk) {
            console.log(chunk);
        });
    }).on("error", function (e) {
        console.log("Got error: " + e.message);
    });

} catch (e) {
    console.log(e.message);
    console.trace();
}

Example
If I try 'WebService1.asmx/Ping' then I would expect 2016-07-18T15:50:04.483+06:00 (current timestamp) as response and this response should be cached so that if I call this url later, then I will get the cached response instead of current timestamp.

Desired output
I want to control the variable options (i.e. path) and fetch response from that particular path. Then, I will compare the result in resp.on which will be same as the example output above. I expect yakbak to mock/cache the response.

Actual output

$ node index.js

done is not defined
Trace
    at Object.<anonymous> (C:\Test\index.js:28:13)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:456:32)
    at tryModuleLoad (module.js:415:12)
    at Function.Module._load (module.js:407:3)
    at Function.Module.runMain (module.js:575:10)
    at startup (node.js:159:18)
    at node.js:444:3

How to use it with React Native?

I am hoping to use Yakbak along with Detox (integration testing framework) for React Native.

I understand that since this package is built for node.js applications it assumes access to node.js core modules such as assert and fs etc

I was able to get around this by using node-libs-browser which provides these core node.js modules in a React Native environment.

Although I am running into trouble with fs package. Here is the error I get:

error: bundling failed: Error: Unable to resolve module `fs` from `pillow_app/node_modules/yakbak/lib/record.js`: Module does not exist in the module map

So in the record.js file it does this var fs = require('fs'); although the only way to get access to the native filesystem in React Native is through this var fs = require('react-native-fs');

Hence I was wondering if anyone had any thoughts on how to resolve this or a possible workaround to get Yakbak working in React Native?

Allow tape to be re-recorded if tape file is deleted

If any tapes are deleted while the yakbak server is running with the intention of refreshing them, subsequent requests for that tape will result in retrieving the cached tape module instead of re-recording. This is because require.resolve does not throw an error even if the tape file doesn't exist and the tape module was cleared from the require.cache. It successfully resolves the tape file and doesn't throw any error. This, however, causes the require to be called which then throws a file not found error.

A simple fix is to check if the file exists post-request.resolve and to throw a MODULE_NOT_FOUND exception if it doesn't which causes control to pass into the catch(ModuleNotFoundError, ...) block, and the request to be re-recorded.

PR for this request:
#39

differentiate from other libraries

It would be helpful if you add a note to your README explaining why you created a new project instead of using Nock's record feature or Node Replay.

Better tape metadata

I don't think tapes need better names (that's what the dirname is for!) but I do think the tapes themselves could use some better metadata, either in the header comment or exported from the module itself.

• query string as a hash
• datetime the tape was recorded
• possibly who recorded it?

Dynamic responsa

Would you be interested in adding documentation on how to edit a yakbak response to make it dynamic?

I had a usecase where I wanted to test an API client that should cache a response if passed a flag and not otherwise, but obviously yakbak "caches" the server response anyway. So I edited the "tape" to return a dynamic response so I could test that.

If it's something you'd like to add docs for, I'm happy to write up what I did with an example so it's clear?

Use template-literal instead of EJS

Template Literal is fastest, smallest and simplest template engine, because it use JS's literal template feature.

It's 55 times faster than EJS, and it also use less CPU and RAM ressources, so it may be a good idea to use it instead of EJS 😀