twarc2 timeline --no-context-annotations not pulling 500 tweets

[mr-devs]

twarc2 timeline --no-context-annotations not pulling 500 tweets

[igorbrigadir]

Ha! I literally just noticed the same "bug" yes.

So, --no-context-annotations is a shortcut to remove context_annotations from tweet.fields. This is something that causes the search endpoint to be limited to 100 results per page, which is slow due to the 1 request per second limit in academic access.

twarc2 timeline command uses the timeline API endpoint, which likewise has a 100 tweet per page limit, but can NOT have a 500 per page with or without context annotations.

twarc2 timeline --use-search will use the search API instead, to get around the last 3200 tweets limit of the timelines API, if you have academic access. However - it seems like --no-context-annotations doesn't seem to work here either, which is the actual bug i think needs fixing.

The temporary workaround for this is to use the search command explicitly, as these two should end up equivalent:

If you had:

twarc2 timeline --use-search --no-context-annotations 1312850357555539972 results.jsonl

run this instead:

twarc2 search --archive --no-context-annotations "from:1312850357555539972" results.jsonl

[mr-devs]

This makes perfect sense. Thanks for the workaround!