RateLimitError

[generalmills]

RateLimitError

[sarahjohns]

Actually, I tried again, now it gives the same traceback plus the following message:
Exceeded Google's Rate Limit. Please use time.sleep() to space requests.

I was using your old version of pytrends several months back. During that time, letting it sleep for 5-10 seconds worked well for me. Has Google increased the limit recently? If you know what the limit is, it would be very helpful for me. Thank you!

[dreyco676]

I've been successful with 6 seconds. I think they allow 10/min

On Tue, Sep 20, 2016 at 10:06 PM, sarahjohns [email protected]
wrote:

Actually, I tried again, now it gives the same traceback plus the
following message:
Exceeded Google's Rate Limit. Please use time.sleep() to space requests.

I was using your old version of pytrends several months back. During that
time, letting it sleep for 5-10 seconds worked well for me. Has Google
increased the limit recently? If you know what the limit is, it would be
very helpful for me. Thank you!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#66 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGxCB_psiY-oIl7bRW2tqY7IZ2qM7yOdks5qsJ84gaJpZM4KCWU4
.

[sarahjohns]

How many requests do you generally have? I remember that it used to be 1500 downloads per 24 hrs. Now it seems to be much lower. How many requests are you able to download with the 6 seconds wait in between? Thank you!

[sarahjohns]

I still experience the RateLimitError after 10 downloads even when I put time.sleep(30). One of the previous posts mentioned that they needed a cookie in the browser called PREF. Do you know how to add it? I could not figure out a way to add it... I would really appreciate it if you could help me figure out this issue.

[dreyco676]

And you are for sure logged into your Gmail account? I'm able to chug along
at 10/min. I've put it a loop to see how long I can keep going.

On Wed, Sep 21, 2016 at 11:16 AM, sarahjohns [email protected]
wrote:

I still experience the RateLimitError after 10 downloads even when I put
time.sleep(30). One of the previous posts mentioned that they needed a
cookie in the browser called PREF. Do you know how to add it? I could not
figure out a way to add it... I would really appreciate it if you could
help me figure out this issue.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#66 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGxCB7PGAE1xkWHF9jcv-tpTS0cWf4mrks5qsVhngaJpZM4KCWU4
.

[sarahjohns]

I have a loop that goes down a list of inputs. Is it important that I call TrendReq(google_username, google_password, custom_useragent=None) inside the loop, i.e., before every request?

[dreyco676]

No I don't think you would need to relogin each time. I just tested as was
able to run 203 requests spaced at 10/min but then it finally gave me the
error. So if I had to guess its probably 10/min, 200/hr.

# connect to Google
pytrend = TrendReq(google_username, google_password, custom_useragent='My Pytrends Script')

trend_payload = {'q': 'Pizza, Italian, Spaghetti, Breadsticks, Sausage', 'cat': '0-71'}

i = 1
while i <= 3000:
    # trend
    trend = pytrend.trend(trend_payload)
    #print(trend)
    print(i)
    time.sleep(6)
    i += 1

[sarahjohns]

I see. Thank you a lot!

[senielee]

I used time.sleep(randint(20, 30)) and I got Rate Limit error after 10 request. That seems weird or google change the rate limit again?

[urataj]

I've found that I'm limited to 12 requests per hour.
However, for some time this morning I've received the following response:
Response did not parse. See server response for details. Sorry, our systems are a little stressed out right now and need to take a deep breath. Please try again in a few moments.

[cdshinde]

Yes, I too saw this response, when I was trying to run the "https://www.google.com/trends/fetchComponent" URL with its required query parameters from the browser. Though it was working normally from the Google Trends URL, for the same URL/IP combination.

[paulwohlfarth]

I'm receiving the same error, funnily enough when scraping the data only. I could still download reports manually without issues even after I hit a suspected rate limit. I also agree with urataj, the rate limit, if that's what's causing this, must be much lower than 200/hr. Could there be some 24hr limit as well? I've also noticed that some of the data seems to be compromised. By that I don't mean the occasional sampling error but large parts of the data having zero values or being otherwise completely spurious.

[dreyco676]

I'm guessing that the library has a bug that is not correctly passing the session/cookies to google and hits an extremely low rate limit. I will look into it when I've got time. Until then if this is a showstopper check out a version before v3.0 it has a clunkier method to get the data.

[paulwohlfarth]

For all I know it is a showstopper for now as the URL used to scrape has changed completely. There is now a token added to the end, which looks very much like the session id's used in the developer platform for the official google api's (at least that's the way it works for maps apparently). And I believe this token hasn't been there before. I wonder if this means that we might get an official trends api soon though? I believe it was planned originally but pretty much forgotten for years. Let's hope the guys at google caught up with that now.

[gmlee7]

I have been struggling for two days on the rate limit. I should have read this thread before starting. I also found that the rate limit is very low: After 12 calls (with 30-second intervals), I got a rate limit error. I wish Google supports official API soon...

[colincolinkn]

Getting this error again... Can you help us to solve it soon? Thanks!
Response did not parse. See server response for details.

[patrickongwong]

Getting similar error... but it worked for the first maybe 4 queries.
Thanks for the great work though!

[Michelangelo1994]

Hi, I have a similar problem here. Can anyone help us out to resolve this issue? Thanks.

[dreyco676]

I spent some time investigating today. It looks like they changed the API and perhaps throttled the old one.

https://www.google.com/trends/api/widgetdata/multiline?

It looks like it has a ton more parameters that I'll have to wade through but hopefully using that I can resolve this.

[dreyco676]

Here is the URL for getting CSV's directly for Pizza & Bagel

https://www.google.com/trends/api/widgetdata/multiline/csv?req=%7B%22time%22%3A%222012-01-24%202017-01-24%22%2C%22resolution%22%3A%22WEEK%22%2C%22locale%22%3A%22en-US%22%2C%22comparisonItem%22%3A%5B%7B%22geo%22%3A%7B%7D%2C%22complexKeywordsRestriction%22%3A%7B%22keyword%22%3A%5B%7B%22type%22%3A%22BROAD%22%2C%22value%22%3A%22pizza%22%7D%5D%7D%7D%2C%7B%22geo%22%3A%7B%7D%2C%22complexKeywordsRestriction%22%3A%7B%22keyword%22%3A%5B%7B%22type%22%3A%22BROAD%22%2C%22value%22%3A%22bagel%22%7D%5D%7D%7D%5D%2C%22requestOptions%22%3A%7B%22property%22%3A%22%22%2C%22backend%22%3A%22IZG%22%2C%22category%22%3A0%7D%7D&token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&tz=360

[dreyco676]

URL decodes to:

https://www.google.com/trends/api/widgetdata/multiline/csv?req=
{
   "time":"2012-01-24 2017-01-24",
   "resolution":"WEEK",
   "locale":"en-US",
   "comparisonItem":[
      {
         "geo":{

         },
         "complexKeywordsRestriction":{
            "keyword":[
               {
                  "type":"BROAD",
                  "value":"pizza"
               }
            ]
         }
      },
      {
         "geo":{

         },
         "complexKeywordsRestriction":{
            "keyword":[
               {
                  "type":"BROAD",
                  "value":"bagel"
               }
            ]
         }
      }
   ],
   "requestOptions":{
      "property":"",
      "backend":"IZG",
      "category":0
   }
}
&token=XXXXXXXXXXXXXXX
&tz=360

[dreyco676]

Looks like you can maybe get the the Token from this url request

https://www.google.com/trends/api/explore?hl=en-US&tz=360&req=%7B%22comparisonItem%22:%5B%7B%22keyword%22:%22pizza%22,%22geo%22:%22%22,%22time%22:%22today+5-y%22%7D,%7B%22keyword%22:%22bagel%22,%22geo%22:%22%22,%22time%22:%22today+5-y%22%7D%5D,%22category%22:0,%22property%22:%22%22%7D&tz=360

which decodes to

https://www.google.com/trends/api/explore?hl=en-US&tz=360&req=
{
   "comparisonItem":[
      {
         "keyword":"pizza",
         "geo":"",
         "time":"today 5-y"
      },
      {
         "keyword":"bagel",
         "geo":"",
         "time":"today 5-y"
      }
   ],
   "category":0,
   "property":""
}
&tz=360

Cool we might have something to work with....

[dreyco676]

I'm able to pull back data this way but it is not a trivial fix and will take some time.

[cwallac]

@dreyco676 absolutely fantastic, thanks for discovering that.

[dreyco676]

In case people are curious as to how I'm doing this. I'm using Fiddler to inspect the web requests that go across when using my browser. From there its all about making PyTrends mimic those requests.

[jblemoine]

@dreyco676 Awesome !!! Thanks a lot.

[jblemoine]

@dreyco676 How did you get the url where I can find the token ? I would like to select a specific country.

[dreyco676]

Fiddler Just look for a URL that starts with https://www.google.com/trends/api/explore?hl=en-US&tz=360

You can specify the country with the geo param.

[dreyco676]

Alright I was able to test that I have request limit that is in the 100's. I will close this issue as it will be fixed in the next release.