salompas/pytrendsdaily

Fetches Daily Data from Google Trends

Table of Contents

1. pytrendsdaily
   1. Usage

pytrendsdaily

The Pytrends package provides an easy way to obtain data from Google Trends. This package, pytrendsdaily, uses the functionality from Pytrends to obtain data from Google Trends at the daily frequency, automatically dealing with scaling issues.

Usage

The main functionality of this package is available via the function getDailyData.

from pytrendsdaily import getDailyData

svi = getDailyData('GOLD PRICES', 2004, 2017)

The variable svi contains a Pandas data frame with the original unscaled search volume index (SVI) from Google Trends, and also the scaled data at the daily frequency.

pytrendsdaily's Issues

Time out issue

I use the example code:
from pytrendsdaily import getDailyData
svi = getDailyData('GOLD PRICES', 2004, 2017)

and then it shows timeout. as show below.
How to fix it?

GOLD PRICES:2004-01-01 2004-01-31
GOLD PRICES:2004-02-01 2004-02-29
GOLD PRICES:2004-03-01 2004-03-31
GOLD PRICES:2004-04-01 2004-04-30
GOLD PRICES:2004-05-01 2004-05-31

timeout Traceback (most recent call last)
~\Anaconda3\lib\site-packages\urllib3\connection.py in _new_conn(self)
158 conn = connection.create_connection(
--> 159 (self._dns_host, self.port), self.timeout, **extra_kw)
160

~\Anaconda3\lib\site-packages\urllib3\util\connection.py in create_connection(address, timeout, source_address, socket_options)
79 if err is not None:
---> 80 raise err
81

~\Anaconda3\lib\site-packages\urllib3\util\connection.py in create_connection(address, timeout, source_address, socket_options)
69 sock.bind(source_address)
---> 70 sock.connect(sa)
71 return sock

timeout: timed out

During handling of the above exception, another exception occurred:

ConnectTimeoutError Traceback (most recent call last)
~\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
599 body=body, headers=headers,
--> 600 chunked=chunked)
601

~\Anaconda3\lib\site-packages\urllib3\connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
342 try:
--> 343 self._validate_conn(conn)
344 except (SocketTimeout, BaseSSLError) as e:

~\Anaconda3\lib\site-packages\urllib3\connectionpool.py in _validate_conn(self, conn)
838 if not getattr(conn, 'sock', None): # AppEngine might not have .sock
--> 839 conn.connect()
840

~\Anaconda3\lib\site-packages\urllib3\connection.py in connect(self)
300 # Add certificate verification
--> 301 conn = self._new_conn()
302 hostname = self.host

~\Anaconda3\lib\site-packages\urllib3\connection.py in _new_conn(self)
163 self, "Connection to %s timed out. (connect timeout=%s)" %
--> 164 (self.host, self.timeout))
165

ConnectTimeoutError: (<urllib3.connection.VerifiedHTTPSConnection object at 0x000001AA2F2716A0>, 'Connection to trends.google.com timed out. (connect timeout=2)')

During handling of the above exception, another exception occurred:

MaxRetryError Traceback (most recent call last)
~\Anaconda3\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
448 retries=self.max_retries,
--> 449 timeout=timeout
450 )

~\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
637 retries = retries.increment(method, url, error=e, _pool=self,
--> 638 _stacktrace=sys.exc_info()[2])
639 retries.sleep()

~\Anaconda3\lib\site-packages\urllib3\util\retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
397 if new_retry.is_exhausted():
--> 398 raise MaxRetryError(_pool, url, error or ResponseError(cause))
399

MaxRetryError: HTTPSConnectionPool(host='trends.google.com', port=443): Max retries exceeded with url: /trends/api/explore?hl=en-US&tz=360&req=%7B%22comparisonItem%22%3A+%5B%7B%22keyword%22%3A+%22GOLD+PRICES%22%2C+%22time%22%3A+%222004-05-01+2004-05-31%22%2C+%22geo%22%3A+%22US%22%7D%5D%2C+%22category%22%3A+0%2C+%22property%22%3A+%22%22%7D (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x000001AA2F2716A0>, 'Connection to trends.google.com timed out. (connect timeout=2)'))

During handling of the above exception, another exception occurred:

ConnectTimeout Traceback (most recent call last)
in
----> 1 svi = getDailyData('GOLD PRICES', 2004, 2017)

~\Anaconda3\lib\site-packages\pytrendsdaily\dailydata.py in getDailyData(word, start_year, stop_year, verbose, wait_time)
109 if verbose:
110 print(f'{word}:{timeframe}')
--> 111 results[current] = _fetchData(pytrends, build_payload, timeframe)
112 current = lastDateOfMonth + timedelta(days=1)
113 sleep(wait_time) # don't go too fast or Google will send 429s

~\Anaconda3\lib\site-packages\pytrendsdaily\dailydata.py in _fetchData(pytrends, build_payload, timeframe)
35 while not fetched:
36 try:
---> 37 build_payload(timeframe=timeframe)
38 except ResponseError as err:
39 print(err)

~\Anaconda3\lib\site-packages\pytrends\request.py in build_payload(self, kw_list, cat, timeframe, geo, gprop)
163 self.token_payload['req'] = json.dumps(self.token_payload['req'])
164 # get tokens
--> 165 self._tokens()
166 return
167

~\Anaconda3\lib\site-packages\pytrends\request.py in _tokens(self)
173 method=TrendReq.GET_METHOD,
174 params=self.token_payload,
--> 175 trim_chars=4,
176 )['widgets']
177 # order of the json matters...

~\Anaconda3\lib\site-packages\pytrends\request.py in _get_data(self, url, method, trim_chars, **kwargs)
121 else:
122 response = s.get(url, timeout=self.timeout, cookies=self.cookies,
--> 123 **kwargs) # DO NOT USE retries or backoff_factor here
124 # check if the response contains json and throw an exception otherwise
125 # Google mostly sends 'application/json' in the Content-Type header,

~\Anaconda3\lib\site-packages\requests\sessions.py in get(self, url, **kwargs)
544
545 kwargs.setdefault('allow_redirects', True)
--> 546 return self.request('GET', url, **kwargs)
547
548 def options(self, url, **kwargs):

~\Anaconda3\lib\site-packages\requests\sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
531 }
532 send_kwargs.update(settings)
--> 533 resp = self.send(prep, **send_kwargs)
534
535 return resp

~\Anaconda3\lib\site-packages\requests\sessions.py in send(self, request, **kwargs)
644
645 # Send the request
--> 646 r = adapter.send(request, **kwargs)
647
648 # Total elapsed time of the request (approximately)

~\Anaconda3\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
502 # TODO: Remove this in 3.0.0: see #2811
503 if not isinstance(e.reason, NewConnectionError):
--> 504 raise ConnectTimeout(e, request=request)
505
506 if isinstance(e.reason, ResponseError):

ConnectTimeout: HTTPSConnectionPool(host='trends.google.com', port=443): Max retries exceeded with url: /trends/api/explore?hl=en-US&tz=360&req=%7B%22comparisonItem%22%3A+%5B%7B%22keyword%22%3A+%22GOLD+PRICES%22%2C+%22time%22%3A+%222004-05-01+2004-05-31%22%2C+%22geo%22%3A+%22US%22%7D%5D%2C+%22category%22%3A+0%2C+%22property%22%3A+%22%22%7D (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x000001AA2F2716A0>, 'Connection to trends.google.com timed out. (connect timeout=2)'))

Single keyword request

Trying to run the API using a single keyword and I am receiving code 400 error from Google. Works fine when using 2-5 keywords. Is there a way you know just to use 1 single keyword?

Invalid syntax

I'm getting an invalid syntax error in python 3.5

>>> from pytrendsdaily import getDailyData

Traceback (most recent call last):

  File "/home/adam/.pyenv/ep-chan/lib/python3.5/site-packages/ipython-7.4.0-py3.5.egg/IPython/core/interactiveshell.py", line 3296, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-3-531ae61cfc18>", line 1, in <module>
    from pytrendsdaily import getDailyData

  File "/home/adam/.pyenv/learn-r/lib/python3.5/site-packages/pytrendsdaily/__init__.py", line 1, in <module>
    from .dailydata import getDailyData

  File "/home/adam/.pyenv/learn-r/lib/python3.5/site-packages/pytrendsdaily/dailydata.py", line 29
    return f"{start.strftime('%Y-%m-%d')} {stop.strftime('%Y-%m-%d')}"
                                                                     ^
SyntaxError: invalid syntax

And I get a different invalid syntax error in python 2.7:

>>> from pytrendsdaily import getDailyData
File "/home/adam/.pyenv/learn-r/lib/python3.5/site-packages/pytrendsdaily/dailydata.py", line 10
    def getLastDateOfMonth(year: int, month: int) -> date:
                               ^
SyntaxError: invalid syntax