HTTP Issues

[generalmills]

HTTP Issues

[ericross2]

John,

I believe I speak for all on this forum that we are all greatly appreciative of your time and effort, and of your graciousness in offering it for use.

Thank you.

Eric

[dreyco676]

New version released 1.2.0 both on github and Pypi. Functionality remains the same. Next major release will use dictionaries for parameters passed in, I will keep both up on Pypi once that happens.

Known issues:

• Google will nag you that you've logged in from a new device each time.

[mcapogreco]

I am also seeing the same issue - checked login details and they are correct, plus checked latest version..
The google trends site still works as expected when using the url's and same account. Hope this can be resolved soon - such a useful tool!

[ericross2]

First off: I greatly appreciate the work done on pytrends. Thank you for providing this.

Pytrends worked perfectly for several weeks with zero glitches on Python 3.5.2. On Monday I received the same 400 error, and since then I have been stuck. I don't know what (if anything) changed. I too tried what @adstastic did in January (closed 400 error topic) and saw the exact same results (same csv file dump s/he saw). I also tried the pip install pytrends --"upgrade" you had recommended in March for Python 3.5.1 to ravimevcha from "Script stopped working.. 400 Bad Request error #30" but still the same 400 error.

I believe I've traced the issue to the self.opener.open(self.url_CookieCheck). My Google credentials are correct and work everywhere else (and I have no messages from Google relating to security or proving I am not a bot, and thus I don't believe Google locked my account. ) so I am at a loss as to how to proceed.

If I manually paste the google trends request url directly into the browser (I am using Opera; already logged into the same Google account for email/etc.) it gives me a good csv. I cleared the Opera cache, and don't show any cookies listed.

To be clear: I am only downloading between 20-50 trends TOTAL per day (each a list of up to five terms to be compared as a single trend/url/csv); this is not a mass-trend downloader for me.

I apologize if accidentally broke protocol but I had originally commented on this via a closed issue: HTTP Error 400: Bad Request #24. Actually there have been a number of similar comments on this closed issue in the past several days.

Any help is appreciated.

Thank you.

[dreyco676]

I will be be taking a look at this today.

This package is something I support in my freetime so I appreciate
understanding in the time it takes to fix.

On Aug 27, 2016 07:59, "luc prieur" [email protected] wrote:

I am getting the same error. The problem is that Google has changed its
backend again. This means that this module, without some serious changes no
longer works. From my initial investigation, it goes beyond simple URL
change.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#47 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGxCB83lRT31J60moQCGpP4QFfAAZypdks5qkDSagaJpZM4JrzRB
.

[dreyco676]

Got it. It's messy so I'll refactor after some coffee tomorrow but here is the gist in case I get hit by a meteor while I sleep.

class SessionGoogle:
def init(self, url_login, url_auth, login, pwd):
self.ses = requests.session()
login_html = self.ses.get(url_login)
soup_login = BeautifulSoup(login_html.content, "lxml").find('form').find_all('input')
dico = {}
for u in soup_login:
if u.has_attr('value'):
dico[u['name']] = u['value']
# override the inputs with out login and pwd:
dico['Email'] = login
dico['Passwd'] = pwd
self.ses.post(url_auth, data=dico)

def get(self, URL):
    return self.ses.get(URL).text

url_login = "https://accounts.google.com/ServiceLogin"
url_auth = "https://accounts.google.com/ServiceLoginAuth"
session = SessionGoogle(url_login, url_auth, google_username, google_password)
req = session.get("http://www.google.com/trends/trendsReport?q=Pizza&hl=en-US&cmpt=q&content=1&export=1")
print(req)

[iethan]

Here is the adapted code for an easy copy / paste job:

from bs4 import BeautifulSoup

class SessionGoogle:
    def __init__(self, email, pwd):
        self.url_login = "https://accounts.google.com/ServiceLogin"
        self.url_auth = "https://accounts.google.com/ServiceLoginAuth"
        self.email = email      
        self.pwd = pwd      
        self.ses = requests.session()
        self.login_html = self.ses.get(self.url_login)
        self.soup_login = BeautifulSoup(self.login_html.content, "lxml").find("form").find_all("input")
        self.dico = {}

    def log(self):          

        for u in self.soup_login:
            if u.has_attr('value'):
                # print(u.encode('utf-8'))
                self.dico[u['name']] = u['value']
                # override the inputs with out login and pwd:
                self.dico['Email'] = self.email
                self.dico['Passwd'] = self.pwd
                self.ses.post(self.url_auth, data=self.dico)

    def get(self, URL):     

        output = self.ses.get(URL).text     
        return output

session = SessionGoogle(username, password)
session.log()
req = session.get("http://www.google.com/trends/trendsReport?q=Pizza&hl=en-US&cmpt=q&content=1&export=1")
print(req)

[shobute]

I am also seeing the same issue with Python 2. It worked on the 19th but not since then, with unchanged username/password and code.

[oshoma]

I get the same error with Python 3.4.3 i.e. the call to self.opener.open(self.url_CookieCheck) fails. My authentication credentials haven't changed since last use. Perhaps Google has changed the behavior of that page.

[lmclupr]

I am getting the same error. The problem is that Google has changed its backend again. This means that this module, without some serious changes no longer works. From my initial investigation, it goes beyond simple URL change.

[lmclupr]

John, I find it extraordinary that guys like you maintain free packages
like pygtrend. I am very grateful. Thanks

On Aug 27, 2016 2:12 PM, "John Hogue" [email protected] wrote:

I will be be taking a look at this today.

This package is something I support in my freetime so I appreciate
understanding in the time it takes to fix.

On Aug 27, 2016 07:59, "luc prieur" [email protected] wrote:

I am getting the same error. The problem is that Google has changed its
backend again. This means that this module, without some serious changes
no
longer works. From my initial investigation, it goes beyond simple URL
change.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<https://github.com/GeneralMills/pytrends/issues/
47#issuecomment-242915837>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/
AGxCB83lRT31J60moQCGpP4QFfAAZypdks5qkDSagaJpZM4JrzRB>
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#47 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAXoKIUAnS7bQchnKnnFaRr1O-OjghIOks5qkH39gaJpZM4JrzRB
.

[dreyco676]

Working on it now. Looks like its finally time to move to Requests. I'm able to log into my Google Account now I just need to pass the cookie to the other report request.

@ericross2 thanks for your help. The reason it is working when you paste the URL is because you are most likely already logged into Google through your browser.