vercel / async-retry Goto Github PK

View Code? Open in Web Editor NEW

1.8K 53.0 52.0 651 KB

Retrying made simple, easy and async

Home Page: https://npmjs.com/async-retry

License: MIT License

JavaScript 93.30% Dockerfile 6.70%

async retry await javascript

async-retry's Introduction

async-retry

Retrying made simple, easy, and async.

Usage

// Packages
const retry = require('async-retry');
const fetch = require('node-fetch');

await retry(
  async (bail) => {
    // if anything throws, we retry
    const res = await fetch('https://google.com');

    if (403 === res.status) {
      // don't retry upon 403
      bail(new Error('Unauthorized'));
      return;
    }

    const data = await res.text();
    return data.substr(0, 500);
  },
  {
    retries: 5,
  }
);

API

retry(retrier : Function, opts : Object) => Promise

The supplied function can be async or not. In other words, it can be a function that returns a Promise or a value.
The supplied function receives two parameters
1. A Function you can invoke to abort the retrying (bail)
2. A Number identifying the attempt. The absolute first attempt (before any retries) is 1.
The opts are passed to node-retry. Read its docs
- retries: The maximum amount of times to retry the operation. Default is 10.
- factor: The exponential factor to use. Default is 2.
- minTimeout: The number of milliseconds before starting the first retry. Default is 1000.
- maxTimeout: The maximum number of milliseconds between two retries. Default is Infinity.
- randomize: Randomizes the timeouts by multiplying with a factor between 1 to 2. Default is true.
- onRetry: an optional Function that is invoked after a new retry is performed. It's passed the Error that triggered it as a parameter.

Authors

Guillermo Rauch (@rauchg) - Vercel
Leo Lamprecht (@notquiteleo) - Vercel

async-retry's People

Contributors

Stargazers

Watchers

Forkers

jfmengels modulexcite gmiroshnykov brandonmp harendranathvegi9 rubythonode timstott mfilenko destenson whitecrownclown madmax983 javascriptexpert honeycrypto dandv heshanlk mellamopablo ramalhoes johannlai hpurra amanoooo igorklopov the-cc-dev tarpsvo timfpark git-thinh pauloprestes paraself t1nky voratham pathakdarshana dexter0201 h2atecnologia apustobaev supersheep isabella232 oooh-name playerdata descire lukasknutti vrootebal dhensby cgb-frontend brandonros hungnd6282 samarth-rapido freekrai ssafayet vuongtd1834 iamsreeragkp swain ericallam vitaly-z

async-retry's Issues

FR: pass attempt number to onRetry

it would be nice to change the signature to this onRetry: (err, i) => {}

Package size increase after async-retry

adding async-retry into our package dependencies increases bundle size by 10kb.Any suggestions to reduce this size?

the way i am importing is import retry from 'async-retry'; the version of the package i am using is "async-retry": "1.3.1". Please let me know.

NPM Version Outdated

When will a new version get published to npm?

Types for Typescript

Are you interested in types? We made some for using your library and I can share them. I can make a pull request, otherwise I can try and toss it up on DefinitelyTyped.

Thanks!

Why isn’t it using vercel retry as its base?

When parsing the options I wandered why they differ from the ones seen in vercel retry.

Then I checked that this library uses retry instead of vercel retry. Any reason why?

(0 , async_retry_1.retry) is not a function

Hello
I am getting this error while using retry

(0 , async_retry_1.retry) is not a function

Here is my code:

 await retry(
        async () => {
          await this.myMethod(arguments)
        },
        { retries: 2 },
      );

Could you please help?

Add option to decide if retry should be attempted?

I know there's feature to bail out conditionally, but I think it would be useful to have a function that is called as soon as the operation fails, and the return value of that function should decide whether or not the next retry should happen.

So if (!op.retry(err)) { would just become something like if (!options.shouldRetry(err) || !op.retry(err)) {.

Current alternative to this would be to have a try/catch in the retrier function, decide in catch, throw if need to retry, and swallow and bail if needed. But this is a lot of boilerplate and can not be extracted into a separate function.

While having a function in option would be much easier.

Did it make sense?

Inaccurate/confusing example?

In example,

// ...
return await retry(async (bail) => {
  // ...
  if (403 === res.status) {
    // don't retry upon 403
    bail(new Error('Unauthorized'));
  }
  // ...
}
// ...

Shouldn't that be return bail? Otherwise, it passes through, and attempts to also return the text substring, which wastes resources.

Bail automatically upon `TypeError`

And make it optional.

UnhandledPromiseRejectionWarning

Hello,

I have these warnings when the retry is trigerred :

(node:27189) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 44): TypeError: Cannot read property 'onRetry' of undefined

This is how I use retry :

const _retryGeocode = (address) => (
  retry((bail) => (
    _geocode(address).catch(e =>
      e === 'OVER_QUERY_LIMIT'
        ? Promise.reject(e)
        : bail(e)
    )
  ))
)

I'm on node v6.9.1

As I don't provide any option to the retry function, I don't understand why these warnings occur as the opt is initialized as an empty object by default.

Does the example code work?

await retry(async bail => {
  // if anything throws, we retry
  const res = await fetch('https://google.com')
  // IF THIS CODE THROWS THEN THE CODE UNDERNEATH WON'T EVER RUN RIGHT?
 //  SO WHY IS THERE LOGIC TO BAIL IN THERE, BECAUSE SURELY TO RUN THAT CODE
// THE PROMISE WOULD HAVE RESOLVED CORRECTLY AND THEREFORE NOT NEED
// TO BE RETRIED?

  if (403 === res.status) {
    // don't retry upon 403
    bail(new Error('Unauthorized'))
    return
  }

  const data = await res.text()
  return data.substr(0, 500)
}, {
  retries: 5
})

my question is in caps above 😄

Jest test fails without retry

"dependencies": {
"@azure/event-hubs": "^5.6.0",
"@azure/eventgrid": "^4.5.0",
"@azure/eventhubs-checkpointstore-blob": "^1.0.1",
"@azure/storage-blob": "^12.8.0",
"@hapi/joi": "^17.1.1",
"@nestjs/common": "^8.2.6",
"@nestjs/core": "^8.2.6",
"@nestjs/mongoose": "^8.0.1",
"@nestjs/platform-express": "^8.2.6",
"@nestjs/swagger": "^5.2.0",
"@sendgrid/client": "^7.4.2",
"async-retry": "1.3.1",
"axios": "^0.21.1",
"azure-eventgrid": "^1.6.0",
"azure-iothub": "^1.14.1",
"azure-storage": "^2.10.4",
"class-transformer": "^0.5.1",
"class-validator": "^0.13.2",
"config": "^3.3.6",
"flat": "^5.0.2",
"fs-extra": "^10.0.0",
"hot-shots": "^8.3.0",
"mongoose": "^5.13.9",
"ms-rest-azure": "^3.0.0",
"nanoid": "^3.3.1",
"nestjs-pino": "^2.5.0",
"pino-noir": "^2.2.1",
"qs": "^6.10.1",
"raw-body": "^2.4.1",
"reflect-metadata": "^0.1.13",
"rimraf": "^3.0.2",
"rxjs": "^7.5.4",
"swagger-ui-express": "^4.1.6"
},
"devDependencies": {
"@nestjs/testing": "^8.1.2",
"@types/express": "^4.17.13",
"@types/flat": "5.0.2",
"@types/fs-extra": "^9.0.13",
"@types/hapi__joi": "^17.1.4",
"@types/jest": "^27.0.2",
"@types/multer": "^1.4.2",
"@types/nanoid": "^2.1.0",
"@types/node": "^14.17.20",
"@types/pino": "^6.3.11",
"@types/supertest": "^2.0.10",
"@typescript-eslint/eslint-plugin": "^3.7.1",
"@typescript-eslint/parser": "^3.7.1",
"eslint": "^7.8.1",
"eslint-plugin-import": "^2.24.2",
"eslint-plugin-prefer-arrow": "^1.2.2",
"jest": "^27.2.4",
"prettier": "^1.15.3",
"supertest": "^4.0.2",
"ts-jest": "^27.0.5",
"ts-node": "9.0.0",
"tsc-watch": "^4.2.9",
"tsconfig-paths": "3.9.0",
"typescript": "^4.0.2",
"wait-on": "^5.3.0"
},
"jest": {
"moduleFileExtensions": [
"js",
"json",
"ts"
],
"rootDir": "src",
"testRegex": ".spec.ts$",
"transform": {
"^.+\.(t|j)s$": "ts-jest"
},
"coveragePathIgnorePatterns": [
"/src/core/validators"
],
"coverageDirectory": "../coverage",
"testEnvironment": "node"
}

please refer to my above configurations. I am facing an issue while testing the async-retry library as below.

const flushPromises = (): unknown => new Promise(resolve => setImmediate(resolve));

it('should retry method execution till success', async () => {
const verifyMethod = jest.fn().mockResolvedValue('verified');
const testObject = {
testKey: 'testValue',
methodToExecute : jest.fn()
.mockRejectedValueOnce(new errors.ThrottlingError('ThrottlingError'))
.mockRejectedValueOnce(new errors.ThrottlingError('ThrottlingError'))
.mockImplementationOnce(function(input) {
// eslint-disable-next-line no-invalid-this
return verifyMethod(input, this.testKey);
})
};
const promise = throttlingErrorHandler.handleOnError(
testObject.methodToExecute.bind(testObject),
[ 'testInput' ],
{ iothub: 'iothub1', deviceId: 'testdevice', operation: 'verifyMethod' }
);
await flushPromises();
jest.runAllTimers();
await flushPromises();
jest.runAllTimers();
await flushPromises();
const result = await promise;
expect(result).toEqual('verified');
expect(verifyMethod).toHaveBeenCalledWith('testInput', testObject.testKey);
expect(testObject.methodToExecute).toHaveBeenCalledTimes(3);
expect(mockMailService.sendMail).not.toHaveBeenCalled();
});

while executing this getting jest timeout exceed error.

below are my code snipped.
public async handleOnError(
asyncMethodToExecute: (...args) => Promise,
methodArguments: unknown[],
context: { iothub: string; deviceId: string; operation: string },
runForever = false
): Promise {

	this.appLoggerService.warn(`Throttling/IotHubQuotaExceededError on ${context.iothub}`, ThrottlingErrorHandler.name);

	const throttlingErrorConfiguration = await this.throttlingErrorConfigurationDbService.getConfiguration();

	if (!throttlingErrorConfiguration) {
		this.appLoggerService.warn('Throttling error configuration not available', ThrottlingErrorHandler.name);
		return null;
	}
	const isMaxRetriesEnabled = throttlingErrorConfiguration.maxRetriesEnabled;
	const maxRetries = isMaxRetriesEnabled ? throttlingErrorConfiguration.maxRetries : -1;
	// fixed interval in seconds after which operation is tried again
	const maxJitterIntervalInSeconds = throttlingErrorConfiguration.maxJitterIntervalInSeconds;
	const sendMailAfterNumberOfAttempts = throttlingErrorConfiguration.sendMailAfterNumberOfAttempts;
	const milliseconds = 1000;

	return await retry(async (bail, attempt) => {
		try {
			const result = await asyncMethodToExecute(...methodArguments); // method context (this) should already be set by caller
			return result;
		} catch (error) {
			if (error instanceof errors.ThrottlingError
                || error instanceof errors.IotHubQuotaExceededError) {
				this.appLoggerService.warn(`Error: ${error.name} on ${context.iothub}`, ThrottlingErrorHandler.name);
				if (attempt % sendMailAfterNumberOfAttempts === 0 && throttlingErrorConfiguration.sendGrid.isEnabled) {
					this.appLoggerService.error(`Error: ${error.name} on ${context.iothub}, sending mail`, ThrottlingErrorHandler.name);
					void this.mailService.sendMail(
						throttlingErrorConfiguration.sendGrid.mailTo,
						throttlingErrorConfiguration.sendGrid.mailCc,
						throttlingErrorConfiguration.sendGrid.templateId,
						{
							iothub: context.iothub,
							deviceId: context.deviceId,
							operation: context.operation,
							error: error.name
						}
					);
				}
				throw error;
			} else {
				bail(error);
			}
		}
	},
	{
		retries: maxRetries > 0 && !runForever ? maxRetries - 1 : undefined, // subtract 1 as initial attempt itself is a retry
		forever: !isMaxRetriesEnabled || runForever,
		maxTimeout: maxJitterIntervalInSeconds * milliseconds,
		randomize: true
	});
}

the above retry function doesn't retry when an error is thrown.

kindly review and suggest what is wrong. it was working with jest 25.* version but not working with 27.* version.

my node version is 16.11.0 & npm version is 8.0.0

Can onRetry handle async functions?

I have this situation where I download the file, and want to delete this file (async operation) if the downloaded file is corrupted and needs retry.

    return retry(
      async (_, attempt) => {
        await downloadFileFromInternet();
      },
      {
        onRetry: err => {
         // It happens that downloaded file has an error
         // I want to delete the downloaded corrupted file
         await fs.unlink(pathToCorruptedFile);
        },
        retries: 3
      }
    );

Is it possible to have async-retry retry the next attempt only once the onRetry resolves (if the return value is a Promise)?

bail on certain errors?

is there a way to check a thrown error before deciding to retry? I tried wrapping my await line in a try/catch & bail()/throw depending on error.message, but that's making my unit tests go nuts for reasons that aren't clear to me

Use of `eslintConfig` leaks out, affecting downstream projects

I faced this today, with my repo that uses firebase-admin.

Would like to hear your comments about it. Is the use of eslintConfig in package.json supposed to be dev/internal detail? Currently (async-retry 1.3.3), It's not.

TypeError error e is not a function and TypeError error Function expected

Hi @rauchg and @leo,

I recently added async-retry to the project I'm working on because I was getting a lot of ChunkLoadError Loading chunk 6 failed in Sentry. This is how I currently use it:

import loadable from '@loadable/component'
import retry from 'async-retry'

const Dashboard = loadable(() =>
  retry(() => import(/* webpackChunkName: "Dashboard" */ 'containers/pages/Dashboard'), {
    factor: 1,
    maxTimeout: 10000,
  })
)

I'm now getting a new error on Sentry from async-retry package. I'm not sure to understand it:

function runAttempt(num) {
  32       var val;
  33
  34       try {
  35         val = fn(bail, num);
  36       } catch (err) {
  37         onError(err, num);
  38         return;
  39       }
  40
  41       Promise.resolve(val)
  42         .then(resolve)
  43         .catch(function catchIt(err) {
  44           onError(err, num);
  45         });
  46     }

According to Sentry the problem is on line 35 of this function. I get 3 different error messages:

TypeError error e is not a function
TypeError error Function expected
TypeError e is not a function. (In 'e(c,t)', 'e' is an instance of Promise)

Am I doing something wrong or is it a bug?
I would be happy to help either way.

How to retry if response data is empty

i want to retry the http request if the response data i am receiving is empty. The backend service takes some time to process the data, it return me an empty array if it has not processed yet. The array will hold data if it is successfully processed.

 public async processClassification(instanceId: any, requestId: any): Promise<any> {

        const url = this.config.backendUrl + "/check/classification";

        const options = {
            uri: url,
            headers: {
                "X-IDCHECK-SESSION_ID": instanceId,
            },
            body: {},
            json: true,
            resolveWithFullResponse: true,
        };

        const data = await retry(async (bail) => {

            const res = await request.get(options);

            if (res.body.data.classification.length > 0) {
              bail("got data");
              return;
            }

            return res;
          }, {
            retries: 5,
            });

        return data;
    }

Also to note i using the following library https://github.com/request/request-promise-native to make the actual request

Set dynamic retry timeout in onRetry

I am currently trying to retry requests made to Notion. If I am rate limited, their API returns the number of seconds I should wait before retrying. I would like to use this value in onRetry (or anywhere else, though here it makes sense) to dynamically alter how long to wait before the next retry. If I am not mistaken, this kind of behaviour is not supported by async-retry at the moment.

Add ability to override the delay for a particular failed invocation

It would be nice to allow the called function to override the amount of time to delay between invocations, for the next invocation only. The rationale for this request is that some REST APIs will return a retry-after header for some calls indicating how long one should wait before retrying the request. There is currently no way to communicate that directive to this module.

The documentation for onRetry is wrong

The documentation currently says this:

onRetry: an optional Function that is invoked after a new retry is performed. It's passed the Error that triggered it as a parameter.

But it should say this:

onRetry: an optional Function that is invoked before a new retry is performed. It's passed the Error that triggered it as a parameter.

It's a small but significant difference.

I thought onRetry was useless because the docs say it runs after the retry. Then I tested and found it actually runs before the retry, which is much more useful. The docs should reflect that.

Feature: set a timeout for the entire retry operation

I find myself doing this a lot when using async-retry:

await Promise.race(
  retry(/* ... */),
  new Promise((_, reject) => {
    setTimeout(() => {
      reject(new Error('FAILED to complete async retry in less than 1 minute'))
    }, 60000)
  }),
)

It would be awesome if there were a clean way to provide an option (retryTimeout?) that allowed you to say ("time out after the retry went for this long, regardless of the attempt number").

Option has no retries

I haven't followed the details of whether the type of @types/retry has changed, but I got an error that option has no "retries", and I couldn't use this package.

const retry = require('async-retry');
const fetch = require('node-fetch');

await retry(
  async (bail) => {
    // if anything throws, we retry
    const res = await fetch('https://google.com');

    if (403 === res.status) {
      // don't retry upon 403
      bail(new Error('Unauthorized'));
      return;
    }

    const data = await res.text();
    return data.substr(0, 500);
  },
  {
   // Not Found
    retries: 5,
  }
);

bail should prevent any further calls to the retrier function

From the example:

// Packages
const retry = require('async-retry')
const fetch = require('node-fetch')
 
await retry(async bail => {
  // if anything throws, we retry
  const res = await fetch('https://google.com')
 
  if (403 === res.status) {
    // don't retry upon 403
    bail(new Error('Unauthorized'))
    // return  <---- don't immediately return here
    throw new Error('Throw after bail'); 
  }
 
  const data = await res.text()
  return data.substr(0, 500)
}, {
  retries: 5
})

Calling bail will immediately reject the promise returned by retry, but if the promise returned by the retirer function is then itself rejected, attempts will continue running in the background until e.g. the max retry count is reached.

This can be surprising and result in obscure bugs.

Passing function as argument does not work

This is most likely because I am not familiar enough with promises... but, how can I pass a function in a variable and then retry it if it fails the first time?:

// works ok:
await retry(
  async (bail) => {
    const res = await fetch('https://google.com')
  }
);

// does not work 
const Fn = fetch('https://google.com')

await retry(
  async (bail) => {
    const res = await Fn; // if fails first time, it is not called again
  }
);

Throw previous error if there's a timeout

Right now, code like this will actually throw the error 'RetryOperation timeout occurred' which originates in node-retry. It should throw the error I made ("HI" + Math.random()), but doesn't because of a timeout error.

import * as retry from 'async-retry'

async function throwsErrors(){
  await delay(1000)
  throw new Error("HI" + Math.random())
}

retry(
  async (bail, attempt) => {
      await throwsErrors()
  } , {
    maxRetryTime: 100,
    minTimeout: 2,
    randomize: true,
    retries: 100,
    onRetry: console.log
  }
).then(console.log)

async function delay(milliseconds: number) {
  return new Promise(resolve => setTimeout(resolve, milliseconds))
}

I made an issue in node-retry that describes why this happens in detail, but the gist is that if you call retryOperation.retry(err) and there's a timeout error, node-retry drops err. I'm not sure if this is something that node-retry needs to support or if it's something that you can/should handle here.

Make sure `randomize` defaults to true

Ref: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/