Data differs from web interface

[generalmills]

Data differs from web interface

[dreyco676]

I've eliminated fake-user agent in 1.2.0 which might have been setting your location to another geo-graphical area (US) than your home location when you manually use the browser.

Please let me know if this is still an issue.

[Nzteb]

I do have the same issue. How is this possible? Does the data provided in the US differ from the data provided in Europe?

[dreyco676]

It is indexed by the number searches in the geographic region you filter it to, therefore the numbers would be different. You can set it to worldwide.

[Nzteb]

My problem is that the data differ, even if exactly the same parameters are entered in the module and the web interface respectively.

E.g I call:

trend({'q':'pizza','date': '1/2014 12m','geo':'US'}, 'dataframe')

The first value of the returned dataframe is:

2014-01-05 78.0

But when I go to Google Trends and in the menu I choose: United States, 2014 and then enter 'pizza' (search term), the value of the first node is:

05 Jan - 11 Jan: 79

The difference gets even higher for the following nodes and I dont see, why these can be different?

Interestingly I found following weird appearance: I checked the url that is requested by the module.
(I changed export to '5' so you can see tha graph, it has no effect on the data)

https://www.google.com/trends/fetchComponent?cid=TIMESERIES_GRAPH_0&date=1%2F2014+12m&q=pizza&export=5&geo=US

The graph shows the data, that is generated by the module. On the right bottom of the page there is a link that says 'View full report in Google Trends'. By clicking this link you reach the web interface of Google Trends with the parameteres already put in the menu. But the data differ to the data on the page before! Do you see the difference?

It would be very nice if you could help me with that.

Thank you.

[dreyco676]

If you download the CSV from the link on the chart you shared you will see
that the date ranges do not line up. One starts at Jan 5th, the other
starts at Jan 4th.

On Tue, Sep 20, 2016 at 1:16 PM, Nzteb [email protected] wrote:

My problem is that the data differ, even if exactly the same parameters
are entered in the module and the web interface respectively.

E.g I call:

trend({'q':'pizza','date': '1/2014 12m','geo':'US'}, 'dataframe')

The first value of the returned dataframe is:

2014-01-05 78.0

But when I go to Google Trends and in the menu I choose: United States,
2014 and then enter 'pizza' (search term), the value of the first node is:

05 Jan - 11 Jan: 79

The difference gets even higher for the following nodes and I dont see,
why these can be different?

Interestingly I found following weird appearance: I checked the url that
is requested by the module.
(I changed export to '5' so you can see tha graph, it has no effect on the
data)

https://www.google.com/trends/fetchComponent?cid=TIMESERIES_
GRAPH_0&date=1%2F2014+12m&q=pizza&export=5&geo=US

The graph shows the data, that is generated by the module. On the right
bottom of the page there is a link that says 'View full report in Google
Trends'. By clicking this link you reach the web interface of Google Trends
with the parameteres already put in the menu. But the data differ to the
data on the page before! Do you see the difference?

It would be very nice if you could help me with that.

Thank you.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#45 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGxCB-bllN0iQTm0_buCHVspF_iFqJvTks5qsCLtgaJpZM4JlZ4m
.

[Nzteb]

Thank you but I dont see this. When I download the CSV from Google Trends it starts at 5.Jan. I dont see that there could be a different CSV to download. The chart as well as the dataframe from the module says the range starts at 5. Jan, too. Could you show me an example where the data from the module exactly fit the data obtained by Google Trends?

[dreyco676]

• I go to the link you sent. The first point is Jan 5th - 11th 2014, the data value is 78.
• I click on the link in the corner "View full report in Google Trends" That one also says Jan 5th - 11th 2014 but the data value is 79.
• I click on the drop down and download to CSV. The first row says 2014-01-04, value 79.

My guess is Google is doing one inclusive of the date on visualization, and exclusive of the date on the other. Since the data is indexed it will be sensitive to the inclusive or exclusion of that date.

If I request the actual K,V data from Google Trends or inspect the JSON in the page source from the link you shared I'll see {"v":new Date(2014,0,5),"f":"Jan 4 \u2013 11, 2014"},{"v":78.0,"f":"78"} for the first record.

[Nzteb]

Thank you for going into this.

• I click on the drop down and download to CSV. The first row says 2014-01-04, value 79.

When I change my computer settings to an US timezone and re-log in into Google, the first row says 2014-01-04, value 79 , too. With Central Europe timezone settings (as in my last posts) it says 2014-01-05, value 79 .

My guess is Google is doing one inclusive of the date on visualization, and exclusive of the date on the other. Since the data is indexed it will be sensitive to the inclusive or exclusion of that date.

Yes, that sounds plausible. Yet we still dont know for sure which periods have been used for which type of data (Google Interface vs. Module/Json) because the Json has both dates in it as well. And I really need to fetch data with the module, that can exactly be replicated from the Google Interface, so Iam still curious about if that is possible?

• I go to the link you sent. The first point is Jan 5th - 11th 2014, the data value is 78.

Well, yes but... what is going on here:
When I click the link again (it is one day later now), the value is not 78 anymore but it changed to 77?! (And of course the same with requesting it with the pytrends module) I know that the interpretation of the data is not affected much in general because the relation between single datapoints is only weakly changed (indexing). Still, if the data fetched today for a specific period differs from the data fetched tomorrow for the same period, Iam wondering how exactly the data is created and if I can use it for research. What is it, that I am missing?

[shobute]

I have been researching this as I have also been having the same problem.

This comment on a different project may be relevant: PMassicotte/gtrendsR#117 (comment)

[Nzteb]

Thank you for the link. The comment clears up some questions. I observed the GTrends interface data on different days and it turns out that they are not consistent either. As the comment supposes, the difference seems to be higher for non-frequent search terms. If the reason for the inconsistency is caused by different samples, this probably can be fixed by fetching the same data at different days and averaging them and therefore getting lower variance of the 'estimate'.

The question now is if the difference between GTrends interface and the link I posted (again: which is used to provide the pytrends module data) has the same reason. If so, it would be fine. But I feel as there is still some uncertainty in it.

[dreyco676]

When I've got time I may re-release a temporary branch using the previous method of getting the data which is much clunkier, but apparently works.... If some one wants to get a headstart on that they can look at the version before v3.0

[dreyco676]

Google trends API changed. They are calculated differently. I will switch over to the new API soon but it will take time.