voxmedia/tap-instagram

Singer Tap for the Instagram Graph API

tap-instagram's Issues

Media stream fails for API Version v13.0+

As of v13.0+ of the API, video_title is deprecated and fails when requested, needs to be removed from the fields docs

user_insights_daily only capturing the last day when using target-sqlite

I am trying to get the history of insights by day over time. With the user_insights_* feeds, I am only seeing the latest days data persisted to sqlite, despite setting the appropriate date boundaries in the config.

I can see the relevant queries in the Graph API query history and meltano is paginating through previous dates, but only the latest gets inserted into the table.

Is it somehow related to the primary_keys = ["id"] set on these streams? Does the PK need to be [id, end_time] to retain all the rows? When I run the extractor using target-jsonl I can see all the previous days in the output file which makes me think it is PK/upsert related.

tap-instagram/tap_instagram/streams.py

Line 725 in a617855

primary_keys = ["id"]

EDIT: Just starting to realise I think this this is related to #13 :)

Incompatible metrics (engagement, impressions, video_views) with reel media

KeyError 'end_date' for user_insights_audience stream.

user_insights_audience stream returns:

{'data': [
{'name': 'audience_city', 'period': 'lifetime', 'values': [{'value': {'Salford, England': 21, 'Manchester, England': 189, 'London, England': 275, 'Lancaster, England': 36, 'Leicester, England': 65, 'Liverpool, England': 137, 'Hong Kong, Hong Kong': 27, 'York, England': 45, 'Newcastle upon Tyne, England': 54, 'Reading, England': 13, 'St Andrews, Scotland': 18, 'Dublin, Dublin': 12, 'Bristol, England': 69, 'Kingston upon Hull, England': 12, 'Jakarta, Jakarta': 13, 'Hatfield, England': 20, 'Paris, Île-de-France': 13, 'Coventry, England': 13, 'Aberdeen, Scotland': 30, 'Exeter, England': 48, 'Nottingham, England': 72, 'Durham, England': 13, 'Shanghai, Shanghai': 122, 'Glasgow, Scotland': 78, 'Delhi, Delhi': 14, 'Portsmouth, England': 57, 'Leeds, England': 80, 'Norwich, England': 15, 'Edinburgh, Scotland': 60, 'Stoke-on-Trent, England': 18, 'Sheffield, England': 58, 'Birmingham, England': 70, 'Bangkok, Bangkok': 14, 'Falmouth, England': 56, 'Stockport, England': 11, 'Northampton, England': 12, 'Cardiff, Wales': 98, 'Tehran, Tehran Province': 13, 'Belfast, Northern Ireland': 16, 'Huddersfield, England': 29, 'Southampton, England': 35, 'Mumbai, Maharashtra': 13, 'Bath, England': 38, 'Brighton and Hove, England': 18, 'Canterbury, England': 12}}], 'title': 'Audience town/city', 'description': "The towns/cities of this profile's followers", 'id': '***/insights/audience_city/lifetime'}, {'name': 'audience_country', 'period': 'lifetime', 'values': [{'value': {'DE': 56, 'BD': 6, 'RU': 10, 'TW': 17, 'HK': 27, 'PT': 8, 'JP': 7, 'FR': 105, 'SA': 13, 'QA': 7, 'BR': 11, 'MA': 6, 'SG': 8, 'DZ': 6, 'KE': 13, 'ID': 23, 'GB': 3133, 'IE': 23, 'OM': 7, 'CA': 18, 'US': 91, 'EG': 15, 'AE': 14, 'CH': 7, 'IN': 83, 'ZA': 10, 'MU': 6, 'IQ': 7, 'IR': 21, 'GR': 11, 'MX': 12, 'IT': 14, 'CN': 147, 'KW': 11, 'MY': 19, 'ES': 16, 'TH': 23, 'AU': 26, 'CY': 9, 'VN': 7, 'PH': 42, 'NG': 26, 'PK': 27, 'NL': 13, 'TR': 14}}], 'title': 'Audience country', 'description': "The countries of this profile's followers", 'id': '***/insights/audience_country/lifetime'}, {'name': 'audience_gender_age', 'period': 'lifetime', 'values': [{'value': {'F.13-17': 12, 'F.18-24': 1185, 'F.25-34': 1075, 'F.35-44': 140, 'F.45-54': 100, 'F.55-64': 39, 'F.65+': 18, 'M.13-17': 2, 'M.18-24': 349, 'M.25-34': 342, 'M.35-44': 82, 'M.45-54': 46, 'M.55-64': 14, 'M.65+': 3, 'U.13-17': 7, 'U.18-24': 425, 'U.25-34': 307, 'U.35-44': 89, 'U.45-54': 31, 'U.55-64': 17, 'U.65+': 9}}], 'title': 'Gender and age', 'description': "The gender and age distribution of this profile's followers", 'id': '***/insights/audience_gender_age/lifetime'}, {'name': 'audience_locale', 'period': 'lifetime', 'values': [{'value': {'el_GR': 8, 'ru_RU': 9, 'it_IT': 15, 'ro_RO': 3, 'tr_TR': 10, 'id_ID': 6, 'pt_BR': 14, 'en_PH': 1, 'th_TH': 6, 'ja_JP': 5, 'fr_FR': 98, 'cs_CZ': 3, 'hu_HU': 1, 'de_DE': 36, 'ms_MY': 1, 'nb_NO': 4, 'zh_HK': 14, 'zh_TW': 16, 'es_MX': 3, 'sk_SK': 2, 'es_ES': 14, 'es_CL': 1, 'nl_NL': 4, 'es_LA': 11, 'sv_SE': 2, 'fa_IR': 2, 'bg_BG': 2, 'vi_VN': 5, 'cy_GB': 2, 'fi_FI': 1, 'ar_AR': 13, 'en_GB': 2754, 'ko_KR': 5, 'en_US': 889, 'uk_UA': 1, 'en_IN': 48, 'zh_CN': 271, 'ar_AE': 1, 'pt_PT': 5}}], 'title': 'Location', 'description': "The locales by country code of this profile's followers", 'id': '***/insights/audience_locale/lifetime'}]}

parse_response fails with a KeyError looking for end_date in this data, but the period value is lifetime so not sure if this end_date key will be present.

Stack trace below

2023-03-08T15:23:25.171108Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.171860Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/bin/tap-instagram", line 8, in <module> cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.172429Z [info     ]     sys.exit(TapInstagram.cli()) cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.172861Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.173463Z [info     ]     return self.main(*args, **kwargs) cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.174023Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.174494Z [info     ]     rv = self.invoke(ctx)      cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.174871Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.175564Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.176171Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.176700Z [info     ]     return __callback(*args, **kwargs) cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.177238Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 499, in cli cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.177752Z [info     ]     tap.sync_all()             cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.178318Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 379, in sync_all cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.178880Z [info     ]     stream.sync()              cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.179322Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1020, in sync cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.180179Z [info     ]     self._sync_records(context) cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.183917Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 962, in _sync_records cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.184758Z [info     ]     self._sync_children(child_context) cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.185483Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1025, in _sync_children cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.186549Z [info     ]     child_stream.sync(context=child_context) cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.187657Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1020, in sync cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.189509Z [info     ]     self._sync_records(context) cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.190156Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 946, in _sync_records cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.190613Z [info     ]     for record_result in self.get_records(current_context): cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.191049Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/tap_instagram/client.py", line 105, in get_records cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.191371Z [info     ]     for record in self.request_records(context): cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.191704Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 323, in request_records cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.192008Z [info     ]     yield from self.parse_response(resp) cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.192314Z [info     ]   File "/home/***/vscode_projects/***_elt/***_meltano/.meltano/extractors/tap-instagram/venv/lib/python3.10/site-packages/tap_instagram/streams.py", line 851, in parse_response cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram
2023-03-08T15:23:25.192744Z [info     ]     "end_time": pendulum.parse(values["end_time"]).format( cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram

<\details

Deprecated metrics for carousel since Dec 23

It looks like some metrics required by this tap have been deprecated since December last year.

singer_sdk.exceptions.FatalAPIError: 400 Client Error: Bad Request - (#100) These metrics carousel_album_impressions, carousel_album_reach, carousel_album_engagement, carousel_album_saved, carousel_album_video_views are no longer supported on version v18+. For a list of alternative metrics, visit https://developers.facebook.com/docs/instagram-api/reference/ig-media/insights. for path: /{media_id}/insights cmd_type=elb consumer=False name=tap-instagram producer=True stdio=stderr string_id=tap-instagram

https://developers.facebook.com/docs/instagram-api/reference/ig-media/insights#metrics

Also on story_insights

singer_sdk.exceptions.FatalAPIError: 400 Client Error: Bad Request - (#100) These metrics taps_forward, taps_back, exits are no longer supported on version v18+. For a list of alternative metrics, visit https://developers.facebook.com/docs/instagram-api/reference/ig-media/insights. for path: /{media_id}/insights

Changelog info: https://developers.facebook.com/docs/instagram-api/changelog#september-12--2023