Giter Site home page Giter Site logo

app-store-scraper's Introduction

app-store-scraper Build Status

Node.js module to scrape application data from the iTunes/Mac App Store. The goal is to provide an interface as close as possible to the google-play-scraper module.

Installation

npm install app-store-scraper

Usage

Available methods:

  • app: Retrieves the full detail of an application.
  • list: Retrieves a list of applications from one of the collections at iTunes.
  • search: Retrieves a list of apps that results of searching by the given term.
  • developer: Retrieves a list of apps by the given developer id.
  • privacy: Display the privacy details for the app.
  • suggest: Given a string returns up to 50 suggestions to complete a search query term.
  • similar: Returns the list of "customers also bought" apps shown in the app's detail page.
  • reviews: Retrieves a page of reviews for the app.
  • ratings: Retrieves the ratings for the app.

app

Retrieves the full detail of an application. Options:

  • id: the iTunes "trackId" of the app, for example 553834731 for Candy Crush Saga. Either this or the appId should be provided.
  • appId: the iTunes "bundleId" of the app, for example com.midasplayer.apps.candycrushsaga for Candy Crush Saga. Either this or the id should be provided.
  • country: the two letter country code to get the app details from. Defaults to us. Note this also affects the language of the data.
  • lang: language code for the result text. Defaults to undefined, so country specific language should be used automatically.
  • ratings: load additional ratings information like ratings number and histogram

Example:

var store = require('app-store-scraper');

store.app({id: 553834731}).then(console.log).catch(console.log);

Results:

{ id: 553834731,
  appId: 'com.midasplayer.apps.candycrushsaga',
  title: 'Candy Crush Saga',
  url: 'https://itunes.apple.com/us/app/candy-crush-saga/id553834731?mt=8&uo=4',
  description: 'Candy Crush Saga, from the makers of Candy Crush ...',
  icon: 'http://is5.mzstatic.com/image/thumb/Purple30/v4/7a/e4/a9/7ae4a9a9-ff68-cbe4-eed6-fe0a246e625d/source/512x512bb.jpg',
  genres: [ 'Games', 'Entertainment', 'Puzzle', 'Arcade' ],
  genreIds: [ '6014', '6016', '7012', '7003' ],
  primaryGenre: 'Games',
  primaryGenreId: 6014,
  contentRating: '4+',
  languages: [ 'EN', 'JA' ],
  size: '73974859',
  requiredOsVersion: '5.1.1',
  released: '2012-11-14T14:41:32Z',
  updated: '2016-05-31T06:39:52Z',
  releaseNotes: 'We are back with a tasty Candy Crush Saga update ...',
  version: '1.76.1',
  price: 0,
  currency: 'USD',
  free: true,
  developerId: 526656015,
  developer: 'King',
  developerUrl: 'https://itunes.apple.com/us/developer/king/id526656015?uo=4',
  developerWebsite: undefined,
  score: 4,
  reviews: 818816,
  currentVersionScore: 4.5,
  currentVersionReviews: 1323,
  screenshots:
   [ 'http://a3.mzstatic.com/us/r30/Purple49/v4/7a/8a/a0/7a8aa0ec-976d-801f-0bd9-7b753fdaf93c/screen1136x1136.jpeg',
     ... ],
  ipadScreenshots:
   [ 'http://a1.mzstatic.com/us/r30/Purple49/v4/db/45/cf/db45cff9-bdb6-0832-157f-ac3f14565aef/screen480x480.jpeg',
     ... ],
  appletvScreenshots: [],
  supportedDevices:
   [ 'iPhone-3GS',
     'iPadWifi',
     ... ]}

Example with ratings option:

var store = require('app-store-scraper');

store.app({id: 553834731, ratings: true}).then(console.log).catch(console.log);

Results:

{ id: 553834731,
  appId: 'com.midasplayer.apps.candycrushsaga',

  // ... like above

  ratings: 652230,
  histogram: {
    '1': 7004,
    '2': 6650,
    '3': 26848,
    '4': 140625,
    '5': 471103
  }
}

list

Retrieves a list of applications from one of the collections at iTunes. Options:

  • collection: the collection to look up. Defaults to collection.TOP_FREE_IOS, available options can be found here.
  • category: the category to look up. This is a number associated with the genre for the application. Defaults to no specific category. Available options can be found here.
  • country: the two letter country code to get the list from. Defaults to us.
  • lang: language code for the result text. Defaults to undefined, so country specific language should be used automatically.
  • num: the amount of elements to retrieve. Defaults to 50, maximum allowed is 200.
  • fullDetail: If this is set to true, an extra request will be made to get extra attributes of the resulting applications (like those returned by the app method).

Example:

var store = require('app-store-scraper');

store.list({
  collection: store.collection.TOP_FREE_IPAD,
  category: store.category.GAMES_ACTION,
  num: 2
})
.then(console.log)
.catch(console.log);

Returns:

[ { id: '1091944550',
    appId: 'com.hypah.io.slither',
    title: 'slither.io',
    icon: 'http://is4.mzstatic.com/image/thumb/Purple30/v4/68/d7/4d/68d74df4-f4e7-d4a4-a8ea-dbab686e5554/mzl.ujmngosn.png/100x100bb-85.png',
    url: 'https://itunes.apple.com/us/app/slither.io/id1091944550?mt=8&uo=2',
    price: 0,
    currency: 'USD',
    free: true,
    description: 'Play against other people online! ...',
    developer: 'Steve Howse',
    developerUrl: 'https://itunes.apple.com/us/developer/steve-howse/id867992583?mt=8&uo=2',
    developerId: '867992583',
    genre: 'Games',
    genreId: '6014',
    released: '2016-03-25T10:01:46-07:00' },
  { id: '1046846443',
    appId: 'com.ubisoft.hungrysharkworld',
    title: 'Hungry Shark World',
    icon: 'http://is5.mzstatic.com/image/thumb/Purple60/v4/08/1a/8d/081a8d06-b4d5-528b-fa8e-f53646b6f797/mzl.ehtjvlft.png/100x100bb-85.png',
    url: 'https://itunes.apple.com/us/app/hungry-shark-world/id1046846443?mt=8&uo=2',
    price: 0,
    currency: 'USD',
    free: true,
    description: 'The stunning sequel to Hungry ...',
    developer: 'Ubisoft',
    developerUrl: 'https://itunes.apple.com/us/developer/ubisoft/id317644720?mt=8&uo=2',
    developerId: '317644720',
    genre: 'Games',
    genreId: '6014',
    released: '2016-05-04T09:43:06-07:00' } ]

search

Retrieves a list of apps that results of searching by the given term. Options:

  • term: the term to search for (required).
  • num: the amount of elements to retrieve. Defaults to 50.
  • page: page of results to retrieve. Defaults to to 1.
  • country: the two letter country code to get the similar apps from. Defaults to us.
  • lang: language code for the result text. Defaults to en-us.
  • idsOnly: (optional, defaults to false): skip extra lookup request. Search results will contain array of application ids.

Example:

var store = require('app-store-scraper');

store.search({
  term: 'panda',
  num: 2,
  page: 3,
  country : 'us',
  lang: 'lang'
})
.then(console.log)
.catch(console.log);

Results:

[
  { id: 903990394,
    appId: 'com.pandarg.pxmobileapp',
    title: 'Panda Express Chinese Kitchen',
    (...)
  },
  {
    id: 700970012,
    appId: 'com.sgn.pandapop',
    title: 'Panda Pop',
    (...)
  }
]

developer

Retrieves a list of applications by the give developer id. Options:

  • devId: the iTunes "artistId" of the developer, for example 284882218 for Facebook.
  • country: the two letter country code to get the app details from. Defaults to us. Note this also affects the language of the data.
  • lang: language code for the result text. Defaults to undefined, so country specific language should be used automatically.

Example:

var store = require('app-store-scraper');

store.developer({devId: 284882218}).then(console.log).catch(console.log);

Results:

[
  { id: 284882215,
    appId: 'com.facebook.Facebook',
    title: 'Facebook',
    (...)
  },
  { id: 454638411,
    appId: 'com.facebook.Messenger',
    title: 'Messenger',
    (...)
  },
  (...)
]

privacy

Retrieves the ratings for the app. Currently only for US App Store. Options:

  • id: the iTunes "trackId" of the app, for example 553834731 for Candy Crush Saga.

Example:

var store = require('app-store-scraper');

store.privacy({
  id: 324684580,
})
.then(console.log)
.catch(console.log);

Returns:

{
  "managePrivacyChoicesUrl": null,
  "privacyTypes": [
    {
      "privacyType": "Data Used to Track You",
      "identifier": "DATA_USED_TO_TRACK_YOU",
      "description": "The following data may be used to track you across apps and websites owned by other companies:",
      "dataCategories": [
        {
          "dataCategory": "Contact Info",
          "identifier": "CONTACT_INFO",
          "dataTypes": [
            "Email Address",
            "Phone Number"
          ]
        },
        ...
      ],
      "purposes": []
    },
    ...
  ]
}

suggest

Given a string returns up to 50 suggestions to complete a search query term. A priority index is also returned which goes from 0 for terms with low traffic to 10000 for the most searched terms.

Example:

var store = require('app-store-scraper');

store.suggest({term: 'panda'}).then(console.log).catch(console.log);

Results:

[
  { term: 'panda pop' },
  { term: 'panda pop free' },
  { term: 'panda' },
  { term: 'panda express' },
  { term: 'panda games' },
  { term: 'panda pop 2' },
  ...
]

similar

Returns the list of "customers also bought" apps shown in the app's detail page. Options:

  • id: the iTunes "trackId" of the app, for example 553834731 for Candy Crush Saga. Either this or the appId should be provided.
  • appId: the iTunes "bundleId" of the app, for example com.midasplayer.apps.candycrushsaga for Candy Crush Saga. Either this or the id should be provided.

Example:

var store = require('app-store-scraper');

store.similar({id: 553834731}).then(console.log).catch(console.log);

Results:

[
  {
    id: 632285588,
    appId: 'com.nerdyoctopus.dots',
    title: 'Dots: A Game About Connecting',
    (...)
  },
  {
    id: 727296976,
    appId: 'com.sgn.cookiejam',
    title: 'Cookie Jam',
    (...)
  }
  (...)
]

reviews

Retrieves a page of reviews for the app. Options:

  • id: the iTunes "trackId" of the app, for example 553834731 for Candy Crush Saga. Either this or the appId should be provided.
  • appId: the iTunes "bundleId" of the app, for example com.midasplayer.apps.candycrushsaga for Candy Crush Saga. Either this or the id should be provided.
  • country: the two letter country code to get the reviews from. Defaults to us.
  • page: the review page number to retrieve. Defaults to 1, maximum allowed is 10.
  • sort: the review sort order. Defaults to store.sort.RECENT, available options are store.sort.RECENT and store.sort.HELPFUL.

Example:

var store = require('app-store-scraper');

store.reviews({
  appId: 'com.midasplayer.apps.candycrushsaga',
  sort: store.sort.HELPFUL,
  page: 2
})
.then(console.log)
.catch(console.log);

Returns:

[ { id: '1472864600',
    userName: 'Linda D. Lopez',
    userUrl: 'https://itunes.apple.com/us/reviews/id324568166',
    version: '1.80.1',
    score: 5,
    title: 'Great way to pass time or unwind',
    text: 'I was a fan of Bejeweled many moons ago...',
    updated: '2021-07-26T18:26:24-07:00',
    url: 'https://itunes.apple.com/us/review?id=553834731&type=Purple%20Software' },,
  { id: '1472864708',
    userName: 'Jennamaxkidd',
    userUrl: 'https://itunes.apple.com/us/reviews/id223990784',
    version: '1.80.1',
    score: 1,
    title: 'Help! THE PROBLEM IS NOT FIXED!',
    text: 'STILL HAVING THE SAME ISSUE.  It\'s happening again...',
    updated: '2021-07-26T18:04:41-07:00',
    url: 'https://itunes.apple.com/us/review?id=553834731&type=Purple%20Software' },
  (...)
]

ratings

Retrieves the ratings for the app. Options:

  • id: the iTunes "trackId" of the app, for example 553834731 for Candy Crush Saga. Either this or the appId should be provided.
  • appId: the iTunes "bundleId" of the app, for example com.midasplayer.apps.candycrushsaga for Candy Crush Saga. Either this or the id should be provided.
  • country: the two letter country code to get the reviews from. Defaults to us.

Example:

var store = require('app-store-scraper');

store.ratings({
  appId: 'com.midasplayer.apps.candycrushsaga',
})
.then(console.log)
.catch(console.log);

Returns:

{
  ratings: 652719,
  histogram: {
    '1': 7012,
    '2': 6655,
    '3': 26876,
    '4': 140680,
    '5': 471496
  }
}

Memoization

Since every library call performs one or multiple requests to an iTunes API or web page, sometimes it can be useful to cache the results to avoid requesting the same data twice. The memoized function returns the store object that caches its results:

var store = require('app-store-scraper'); // regular non caching version
var memoized = require('app-store-scraper').memoized(); // cache with default options
var memoizedCustom = require('app-store-scraper').memoized({ maxAge: 1000 * 60 }); // cache with default options

memoized.app({id: 553834731}) // will make a request
  .then(() => memoized.app({id: 553834731})); // will resolve to the cached value without requesting

The options available are those supported by the memoizee module. By default up to 1000 values are cached by each method and they expire after 5 minutes.

app-store-scraper's People

Contributors

araratispiroglu avatar b-reynolds avatar cgwyllie avatar chernikovalexey avatar christian-fei avatar dahlgren avatar davidgustys avatar dependabot[bot] avatar elabpro avatar facundoolano avatar filoozom avatar gamebenchjake avatar giszmo avatar hcyildirim avatar icaroponce avatar jacob-meacham avatar jbigman avatar kasnder avatar kblcuk avatar koltclassic avatar marxsk avatar mrcoles avatar pdobrev avatar piercus avatar rchatterjee avatar secretrobotron avatar techinpark avatar tk1cntt avatar twalling avatar yura415 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

app-store-scraper's Issues

No Reviews?

Do you plan to add a review functionality?

100 items return on search

Hi,

Thanks for the scraper it works like a charm but the search function returns only 100 results and if there are some others it says "... 150 more items" and interrupt. Is there any method to see full search result?

Default value of page param on reviews method

Is there some special reason for the default value of the page param on reviews method to be 1 instead of 0?

My concerning is about the compatibility with the google-play-scraper module.

I think that if we could make it compatible and work the same way, it'd be great for both APIs.

No Date in Reviews Results

Is there any way to add date data in the Reviews output?
I see that the google-play-scraper has this but not that app-store-scraper.

Thanks!

Get IPA

it's possible to get the IPA without login?

Crash when single review is returned

If there is only one review then the server returns a single object, not an array and the parser crashes with:

TypeError: reviews.map is not a function
    at cleanList (.../node_modules/app-store-scraper/lib/reviews.js:15:18)
    at process.internalTickCallback (internal/process/next_tick.js:77:7)

Try this call to duplicate:

store.reviews({id:` 1092841332, country: 'IL'}).then(console.log).catch(console.log);

Error is here:

function cleanList (results) {
  let reviews = results.feed.entry || [];
  return reviews.map((review) => ({
// ...

I managed to fix the problem with something like the code below but perhaps there could be a better way.

function cleanList (results) {
  let reviews = results.feed.entry || [];
  // Check if it's a single object and if so wrap it into array
  if (results.feed.entry && !Array.isArray(results.feed.entry)) {
    reviews = [results.feed.entry];
  }
  return reviews.map((review) => ({
// ...

The "list" function returns an error.

The response body is output as shown below.

<HTML>
<HEAD>
<TITLE>Web Server Error</TITLE>
</HEAD>
<BODY BGCOLOR="white" FGCOLOR="black">
<H1>Web Server Error</H1>
<HR>

<FONT FACE="Helvetica,Arial"><B>
Description: The host did not return the document correctly.
</B></FONT>
<HR>
</BODY>

Check if reviews is a valid type array

If results.feed.entry exists but is not a valid type array (i.e. dictionary) in our case, this error occurs. The resulting json file returns a dictionary for this entry when there are no reviews in that page.

TypeError: reviews.slice is not a function
    at cleanList (.../node_modules/app-store-scraper/lib/reviews.js:10:18)
    at process._tickDomainCallback (internal/process/next_tick.js:129:7)

A simple check to see if it's a valid array would be sufficient for a fix in this case.

Returning old app info

After deploying a new version of the IOS app, app-store-scraper returns the previous version with its details.

Is there a reason that the updated details will take some time to propagate?

Test case: Opening the app store URL shows the update is live with new version, and the update is available to devices, although app-store-scraper returns the previous version with its details.

I need to check the version immediately when update is live.

App store version:
screenshot 2018-10-01 at 08 45 45

app-store-scraper version:
screenshot 2018-10-01 at 08 46 03

Retrieving app by "id" instead of "appId" seems to make a difference, why would this be the case?

Feature request: add ratings

Can the ratings also be added to this library, like the android version?
They are a bit more complex to fetch, but very useful.
You can fetch them with the call below:
By using: GET http://ax.phobos.apple.com.edgesuite.net/WebObjects/MZStore.woa/wa/viewContentsUserReviews?id=&pageNumber=0&sortOrdering=2&type=Purple+Software
With headers:
User-Agent: iTunes/9.1.1
X-Apple-Store-Front: 143441-1

X-Apple-Store-Front tells it which apple store to talk to 143441-1 is usa.
This page includes all reviews and all ratings, so it includes way more data then the rss feed and is literally the call that itunes makes.

Simple script does not terminate

Node v4.2.2
[email protected]

It appears that this library prevents a script from terminating. Here’s a super simple script that when I run from the command-line as node test.js it just hangs until I forcefully close it.

var store = require('app-store-scraper');

store.search({
    term: 'panda',
    num: 2,
    device: store.device.IOS,
    country : 'us'
})
.then(console.log)
.catch(console.log);

As a sanity check, the following does terminate on its own.

var store = require('app-store-scraper');

function foo() {
    return new Promise((resolve, reject) => {
	resolve('OK!');
    });
}

foo()
.then(console.log)
.catch(console.log);

iTunes will return no data sometimes unless country code is provided.

Hi there,
Thank you so much for putting this package together, saves me a ton of time. It's really great but there is a small issue where with some appIds, unless the country_code is provided in the lookup url, it will return nothing back in the json response even though the app is live in the store.

I am thinking of making a pull request to add in a feature so you can add the country code similar to how you can add throttle to the play-store-scrapper. My goal is that if the country code is provided it tacks it on to the url, if not, everything runs normally as it already does. Let me know if this is okay and I'll push up the PR.

Again, thanks for all your work on this package :)
Cheers!

Allow memoization for internal calls

  • the common.lookup helper is used by multiple methods, they should share the memoized results when possible. Pass it by parameter like it's done in google-play-scraper
  • the search method makes a first request that is common to all pages; this result should be reused when fetching subsequent pages.

"currentVersionScore" value gone

I just noticed that reviews no longer contain the "currentVersionScore" value. Is this something Apple deprecated or could I be missing something. Thanks!

implement fullDetail for list

The new itunes rss feeds returns much less information than before, so a fullDetail option should be implemented as it is in google-play-scraper.

make sure memoization properly works for the full detail calls (dont make a request if the app has been called via the app method)

unskip (and fix as necessary) the full detail list test.

Unable to get reviews sorted as RECENT

I am trying to sort reviews as RECENT but it is giving me an error.

TypeError: reviews.slice is not a function

var store = require('app-store-scraper');
store.reviews({ id: 'XXXXXXXXX',sort: store.sort.RECENT,page: 2}).then(console.log).catch(console.log);

XML Feed for Reviews?

I've been working with this scraper to collect app reviews, but my use case requires review dates. I noticed the XML version of the reviews feed provides date info, and was going to work on adding that functionality to this scraper, but then I saw there was already an accepted PR for that feature in March 2017. Is there any reason the scraper is back to using the JSON feed for reviews?

Here's the PR in question:
#19

Review date

Is it possible to get a date field for app reviews?

Get All Reviews?

Is there any way of knowing how many pages are able to be requested? is there a way of requesting all reviews?

Similar call not return any results

I have used the method Similar example, but it returned a null array. I have tested with other id but it was returned the same result. Can anyone help me?
I used the search method and it worked so i think i have installed it successfull.

Can't Get Reviews After Providing 'id' and 'appId'

Hi, Thanks for developing this scraper. I use 'app' to retrieve the full detail of an application with id: 1135011025 and find the appId. But after I providing both 'id' and 'appId' as options in 'reviews', there's nothing retrieved. The codes are as following:

var store = require('app-store-scraper');
store.reviews({
id: 1135011025,
appId: 'com.upside-travel.Winterfell',
sort: store.sort.HELPFUL,
page: 2
})
.then(console.log)
.catch(console.log);

Thanks for your help.

Compile to ES5

Hey,

Thanks a lot for your work here - we've been using app-store-scraper for some time with our Ionic app.

We have this issue with our app, the underlying cause being app-store-scraper. Until now it only affected development builds and we've just lived with it.

However, on Ionic's latest iteration, it's completely broken, so now our production builds will not run on iOS 9 at all and we need to get our release out ASAP.

After spending some time trying to monkey patch this morning, I've realised that the lesser of several evils was just to release a clone that's compiled to ES5: https://www.npmjs.com/package/app-store-scraper-compat.

I am just raising here for your information. I figured you might not be interested in supporting ES5 anyway.

If you end up releasing a backwards compatible version in the future I'll just kill off app-store-scraper-compat.

Thanks

Stephen

the method ".list" does not work

all other methods works well except the method ".list" does not work for me since yesterday knowing that it was functional...i can't get the app list from app store

App not found

App store Scrapper works pretty but on some app it shows error
that app is not found
for example
com.bestonlinepartners.udemy.udemy

Research itunes endpoint for app lists

Instead of using the feeds endpoint, we could use whatever endpoint itunes uses internally. The feed is slow (see #55), has limited results and currently doesn't allow filtering by category.

Apps position is not the same showed in the iTunes Store

Hello there,

I'm using the module to search for some apps, but the result returned is not the same that I get on the iTunes app (Yes, I had set the country correctly) which works perfectly in the google-play-scraper module. As I'm using it for ASO purposes, it's critical to know the real apps positions in the iTunes. (For example, I want to know if the app which i'm optimizing in ranking better, if it's in the first position for a given word...)

Talking to a friend of mine who works with ASO daily, he told me that the iTunes API is not reliable when it comes to the apps positioning (maybe for a delay to update, idk, but it isn't the same list) and another solution would be necessary.

Had you guys have the same problems or notice something like this? Or am I the only one?

Actually, if it's really a problem, I'm considering a solution: I've found out using a Sniffer (like Wireshark), to monitor the network when I'm using the iTunes app, and the app do requests to another url, setting different headers to fetch the content to be presented on it...

Please, let me know what you guys think about it.

[Question] Languages supported

Hi @facundoolano, Thanks for sharing app-store-scraper! I'm wondering if it is possible to use it with languages other than English. I'm interested in obtaining search suggestions in Chinese/Japanese.

bug in extractSuggestions

In extractSuggestions function in app-store-scraper/lib/suggest.js assumes there is always suggestions. I think it should check before applying list.map.

TypeError: Cannot read property 'map' of undefined
    at extractSuggestions (/Users/rahul/appscraper/node_modules/app-store-scraper/lib/suggest.js:27:14)
    at process._tickCallback (internal/process/next_tick.js:109:7)

(I am very new to nodejs, and pardon me if it shouold be how it is.)

ENOBUFS error after many requests at once (403 error after many frequent requests)

I'm trying to export reviews by looping through 2 arrays of data (10 games and 38 countries) and iTunes reviews pages. Something like this (I'm not good at coding):

for( var i = 0; i < g.length; i++) {
	for( var j = 0; j < c.length; j++) {
		for( var p = 0; p < 10; p++) {
			// Parse reviews
			})
			.then(console.log)
			.catch(console.log);
		};
	};
}

I'm getting this in the output file:

{ Error: connect ENOBUFS x.xx.xxx.xxx:xxx - Local (undefined:undefined)
    at Object._errnoException (util.js:992:11)
    at _exceptionWithHostPort (util.js:1014:20)
    at internalConnect (net.js:960:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:284:19)
    at GetAddrInfoReqWrap.emitLookup [as callback] (net.js:1106:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:97:10)
  code: 'ENOBUFS',
  errno: 'ENOBUFS',
  syscall: 'connect',
  address: 'x.xx.xxx.xxx',
  port: xxx }

I've searched for workaround and tried to delay each request with setTimeout (in many various ways) with no success.

I would like to know if it's possible to add throttle option like in google-play-scraper to avoid this error?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.