Encoding issue when applying the script to non-English language

[taspinar]

Encoding issue when applying the script to non-English language

[ralfian]

Same here, i need to mining in non-english text also, please help us.
Thanks!

[ralfian]

alternative solution is use location of the non-english city.

[aliyudin21]

You can write a script in your Python intepreter like following :

from twitterscraper import query_tweets

for tweet in query_tweets('your keyword', 10)[:10]:
    text = tweet.text.encode('utf-8')
    user = tweet.user.encode('utf-8')
   
    xfile = 'F:/tweets.txt'    #you can create your own .json in your own path 
    with open(xfile, 'ab+') as f:    #ab+ is to write in the end position of the file (appending) 
        f.seek(0)
        f.write(text+'\n')    #you need '\n' to make new line for incoming tweets 
        f.write(user+'\n')

Hope it helps.

[taspinar]

@zdw-chj @ralfian @aliyudin21
The twitterscraper as it is, should work properly with non-english characters.

The thing is, when the text is saved to file, it is saved as the utf-8 encoded bytes of these characters instead of the actual characters.

So when you scrape for "戦う", the contents of the file will indeed look something like ""\u30b1\u30f3\u3055\u307e\u30fe ..."

so what you should do is open this file with the proper encoding

import codecs, json
filename = 'chinese.json'
with codecs.open(filename, 'r', 'utf-8') as f:
    tweets = json.load(f, encoding='utf-8')

See example below: