UnicodeEncodeError

[twintproject]

UnicodeEncodeError

[haccer]

Ok, I’m going to try reproducing this.

[grepsedawk]

This is a fairly common python issue, first piece on the Unicode docs here I believe https://docs.python.org/2.7/howto/unicode.html#the-unicode-type Essentially instead of calling str, you have to call encode instead since str assumes ascii and it's really utf-8 On mobile and replying in email, I forgot if this was my 2 or 3. Docs should be pretty much the same for either on Unicode I thought though.

…

On Thu, Feb 15, 2018, 9:05 PM Cody Zacharias ***@***.***> wrote: Ok, I’m going to try reproducing this. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#24 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGaPS_uMKOq9gOOveCixenWbA208itzlks5tVPB-gaJpZM4SGmH_> .

[haccer]

Yea it appears when I do a search of ü which is \xfc, tweep won't scrape tweets. I tested this with some other unique unicode characters and got the same result. Going to try and figure out a fix next week.

[tredmill]

Thank you for the script Haccer. We changed the script so that it does not stop, but it is by no means an elegant solution. This might work for those in a hurry until you find a more sustainable alternative.

async def getTweets(init):
    tweets, init = await getFeed(init)
    count = 0
    for tweet in tweets:
        try:
            tweetid = tweet["data-item-id"]
            datestamp="01 jan 1970"
            try:
                datestamp = tweet.find("a", "tweet-timestamp")["title"].rpartition(" - ")[-1]
            except TypeError:
                pass
            d = datetime.datetime.strptime(datestamp, "%d %b %Y")
            date = d.strftime("%Y-%m-%d")
            timestamp="00:00:00"
            try:
                timestamp = str(datetime.timedelta(seconds=int(tweet.find("span", "_timestamp")["data-time"]))).rpartition(", ")[-1]
            except TypeError:
                pass        
            t = datetime.datetime.strptime(timestamp, "%H:%M:%S")
            time = t.strftime("%H:%M:%S")
            username = tweet.find("span", "username").text.replace("@", "")
            timezone = strftime("%Z", gmtime())
            text = tweet.find("p", "tweet-text").text.replace("\n", " ").replace("http"," http").replace("pic.twitter"," pic.twitter")
            hashtags = ",".join(re.findall(r'(?i)\#\w+', text, flags=re.UNICODE))
            replies = tweet.find("span", "ProfileTweet-action--reply u-hiddenVisually").find("span")["data-tweet-stat-count"]
            retweets = tweet.find("span", "ProfileTweet-action--retweet u-hiddenVisually").find("span")["data-tweet-stat-count"]
            likes = tweet.find("span", "ProfileTweet-action--favorite u-hiddenVisually").find("span")["data-tweet-stat-count"]
            try:
                mentions = tweet.find("div", "js-original-tweet")["data-mentions"].split(" ")
                for i in range(len(mentions)):
                    mention = "@{}".format(mentions[i])
                    if mention not in text:
                        text = "{} {}".format(mention, text)
            except:
                pass
            if arg.users:
                output = username
            elif arg.tweets:
                output = tweets
            else:
                output = "{} {} {} {} <{}> {}".format(tweetid, date, time, timezone, username, text)
                if arg.hashtags:
                    output+= " {}".format(hashtags)
                if arg.stats:
                    output+= " | {} replies {} retweets {} likes".format(replies, retweets, likes)

            if arg.o != None:
                if arg.csv:
                    dat = [tweetid, date, time, timezone, username, text, hashtags, replies, retweets, likes]
                    with open(arg.o, "a", newline='') as csv_file:
                        writer = csv.writer(csv_file, delimiter="|")
                        writer.writerow(dat.encode('utf-8'))
                else:
                    print(output.encode('utf-8'), file=open(arg.o, "a"))

            count += 1
            print(output)
        except:
            print("skipped tweet")

[haccer]

Sorry it took until today for me to replicate your environment.

tl;dr
The problem was an issue with your locale. To fix, simply run:

export LC_CTYPE=en_GB.UTF-8

--
This kinda threw me off a little because, Python3 is supposed to be unicode by default.
I tested w/ the default locale against Twitter's that contained special characters and had no issues, but after changing to your specific locale settings I got that error.

After exporting that variable, I confirmed it worked and was not an issue with Tweep itself or connected to the known-issue of searching special characters with Tweep.

[grepsedawk]

Had to be British not us?

…

On Sat, Feb 24, 2018, 4:24 PM Cody Zacharias ***@***.***> wrote: Sorry it took until today for me to replicate your environment. *tl;dr* The problem was an issue with your locale. To fix, simply run: export LC_CTYPE=en_GB.UTF-8 -- I tested w/ the default locale against Twitter's that contained special characters and had no issues, but after changing to your specific locale settings I got that error. This kinda threw me off a little because, Python3 is supposed to be unicode by default. After exporting that variable, I confirmed it worked and was not an issue with Tweep itself or connected to the known-issue of searching special characters with Tweep. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#24 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGaPS04HovISIYBteHjDmwvdUVVMBio9ks5tYIwPgaJpZM4SGmH_> .

[haccer]

@pachonk nah, his locale had: LC_CTYPE=UTF-8

I guess language has to be specified, so for USA it'd be en_US.UTF-8

[grepsedawk]

Okay, I thought so but I was curious if only the local for GB worked. Thanks!

…

On Sat, Feb 24, 2018, 6:10 PM Cody Zacharias ***@***.***> wrote: @pachonk <https://github.com/pachonk> nah, his locale had: LC_CTYPE=UTF-8 I guess language has to be specified, so for USA it'd be en_US.UTF-8 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGaPS-jqaRVNxVWMlX7wlq_WtZ9U2OX4ks5tYKTZgaJpZM4SGmH_> .

[grepsedawk]

Okay, I thought so but I was curious if only the local for GB worked. Thanks!

…

On Sat, Feb 24, 2018, 6:32 PM Alex Piechowski ***@***.***> wrote: Okay, I thought so but I was curious if only the local for GB worked. Thanks! On Sat, Feb 24, 2018, 6:10 PM Cody Zacharias ***@***.***> wrote: > @pachonk <https://github.com/pachonk> nah, his locale had: LC_CTYPE=UTF-8 > > I guess language has to be specified, so for USA it'd be en_US.UTF-8 > > — > You are receiving this because you were mentioned. > > > Reply to this email directly, view it on GitHub > <#24 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AGaPS-jqaRVNxVWMlX7wlq_WtZ9U2OX4ks5tYKTZgaJpZM4SGmH_> > . >