Are retweets truncated?

[altanner]

Are retweets truncated?

[altanner]

tweepy/tweepy#1085

[altanner]

Need to look at finding other field with full text. Really don't want a conditional in there if we can avoid it.

[altanner]

I've updated mongoexport to be making a csv with the fields

user.id_str,id_str,created_at,retweeted_status.full_text,full_text
that is
USER ID, TWEET ID, WHEN, CONTENTS OF RETWEET FIELD, CONTENTS OF BASE TEXT FIELD.

This loses no data (well none is lost anyway because it is all in mongodb), but it present a problem.

1. Tweets which are actual tweets generate a blank field (last field is correct).
2. Retweets are truncated in last field, but are in full and correct in penultimate field.
3. You can't just move retweets into final field, because then you can't tell if it is a tweet or not. Which would require an odd shell script to sort out, since conditional export is not possible for mongoexport.
4. Other option is for docker to contain a script, in javascript, which interacts with mongodb to say "if it IS a tweet take THIS field, if NOT take THAT field", but you are still left with the problem of tweets not having the "RT @blahblah" appending the field, so you can't tell if it is a retweet or not in the final csv.

[altanner]

for NLP, dealt with by the function tweet_or_retweet.

In plain csv output, both fields are sent out to file, otherwise unclear.