Comments (3)
I'd love the help. I was thinking about how to parse the csv response into separate files for consumption while being flexible enough to handle changes.
Another issue that was brought to my attention is that there is a quota limit and we should try handling it and retrying if we reach it.
from pytrends.
So thanks for the commit rights, I will take a look, I have a few things to do over the next week or so but I will get involved soon.
About the quota limit.... That is an issue, I am really not sure how to avoid it, I think the randomised timer you have imposed is a good idea and may throw Google off the scent. That was the reason I added a sleep in the first place, but it sometimes hits the limit still. We should certainty add an exception handler for this. I have found that the quota limit is very random...
The problem with making the csv response code too involved is that Google could change the csv file format and then we're screwed. So I think you're correct in that for now it can easily be split into several smaller files: Interest Over Time, Regions, Cities, Searchers, Rising Searches. So that is 4 in total and they're usually separated by "\n\n" at the very least. Speak soon, should be good! I think as long as we get it into a useful output, then that is the main thing. I was interested in the Interest over time column for a given date range so that was what my code ended up doing and passing it to a python application called pandas.
Speak soon. Matt
from pytrends.
We could just abstract the splitting of the csv response into just reading the first line of each section and then just write out the data until the next blank line and name the file searchterm_firstrow.csv. That way if they change City into Geographical Regions or switch them around we are still doing it correctly.
from pytrends.
Related Issues (20)
- MaxRetryErr HOT 8
- It broke again HOT 57
- Code documentation :)
- Broken for short timeframe - question for experts HOT 4
- related_queries and related_topics not working (infinite loop) HOT 4
- Anyone getting "Holes" in the data again? HOT 2
- Many tests fail
- Error with all request: HTTPSConnectionPool(host='trends.google.com', port=443): Max retries exceeded HOT 17
- Update to Selenium script HOT 4
- is that pytrends still unavailiable everyone? HOT 4
- Selenium script giving exception HOT 2
- Error 429 for the past few days HOT 14
- Interested in Maintainers
- KeyError: 'content-type' when calling build_payload()
- Is there an upper limit on display results? HOT 1
- 429 Error HOT 7
- I am interested in reverse engineering Google trends HOT 1
- daily serach trends
- Too many 429 error responses. Max retries exceeded. #578 HOT 3
- Search volume HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytrends.