fitnr/twitter_markov

Create markov chain ("_ebooks") accounts on Twitter

twitter markov

Create markov chain ("_ebooks") accounts on Twitter. The audience for this library is those with at least basic Python experience. Before you set this up, you'll need:

• A twitter account
• A twitter application (register at dev.twitter.com) with authentication keys for the account (read more)
• A text for the bot to learn from, which can be a text file or a Twitter archive. Several thousand lines are needed to get decent results, with fewer than 100 or so it won't work at all.

Install

Run pip install twitter_markov, or download/clone the package and run python setup.py install. Feel free to use a virtualenv, if you're into that.

Setting up a bot

See the docs for a complete guide to setting up an ebooks bot. Here are the basics:

• Create a app and authenticate it with your new Twitter account
• Create a corpus, a text file with one sentence or text per line
• Create a bots.yaml config file
• Set up a task to tweet and reply

API

See the docs.

License

Copyright 2014-2016, Neil Freeman. This software is available under the GPL 3.0. See LICENSE for more information.

twitter_markov's Issues

Odd behavior: Verbose mode seems to indicate that state is always set to 2 no matter what config says

Config has in it

state_size: 4

But when run, verbose output says

twitter_markov.py 104: setting up models (state_size=2)

what has gone wrong?

Corpus Creation Errors

Python v3.6.5

Trying to create a corpus, getting an error with wordfilter.

Traceback (most recent call last):
  File "/Users/timharrison/.pyenv/versions/3.6.5/bin/twittermarkov", line 11, in <module>
    load_entry_point('twitter-markov==0.5.0', 'console_scripts', 'twittermarkov')()
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pkg_resources/__init__.py", line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2693, in load_entry_point
    return ep.load()
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2324, in load
    return self.resolve()
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/twitter_markov-0.5.0-py3.6.egg/twitter_markov/__init__.py", line 17, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/twitter_markov-0.5.0-py3.6.egg/twitter_markov/twitter_markov.py", line 24, in <module>
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/wordfilter/__init__.py", line 6, in <module>
    from .wordfilter import blacklisted, add_words, clear_list, Wordfilter
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/wordfilter/wordfilter.py", line 52, in <module>
    _module_instance = Wordfilter()
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/wordfilter/wordfilter.py", line 29, in __init__
    self.blacklist = [s.lower() for s in json.loads(data, 'r')]
TypeError: loads() takes 1 positional argument but 2 were given

I saw in another issue that updating wordfilter was the solution to this, but after doing that, i get this:

Traceback (most recent call last):
  File "/Users/timharrison/.pyenv/versions/3.6.5/bin/twittermarkov", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3088, in <module>
    @_call_aside
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3072, in _call_aside
    f(*args, **kwargs)
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3101, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pkg_resources/__init__.py", line 576, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pkg_resources/__init__.py", line 589, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/Users/timharrison/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pkg_resources/__init__.py", line 778, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'wordfilter<0.2.0pyyaml,>=0.1.8' distribution was not found and is required by twitter-markov

Which makes sense as wordfilter is at 0.2.6.2.

Any way to resolve these issues and get a corpus going?

writelines() argument must be a sequence of strings

First of all, thanks for the quick answer in #2 - it's absolutely gotten me one step closer! But now I'm having this problem instead:

pi@raspberrypi:~/ThePostmodern $ twittermarkov corpus --one-per-line ./data/tweets.txt
Reading ./data/tweets.txt
Traceback (most recent call last):
  File "/usr/local/bin/twittermarkov", line 9, in <module>
    load_entry_point('twitter-markov==0.4.3', 'console_scripts', 'twittermarkov')()
  File "/usr/local/lib/python2.7/dist-packages/twitter_markov/cli.py", line 70, in main
    func(argdict)
  File "/usr/local/lib/python2.7/dist-packages/twitter_markov/cli.py", line 98, in learn_func
    sys.stdout.writelines(tweets)
TypeError: writelines() argument must be a sequence of strings

I first assumed there was something problem with my tweets.txt file, but there doesn't seem to be. The linebreaks are all in UNIX format, there's just plain text without any fancy characters that could be breaking anything... Any ideas?

Script doesn't recognize bots.yaml

http://pastebin.com/raw/Dc65uTFx - bots.yaml, in the same directory that I'm running the command from

The command I'm running (I've tried a few variants, but):
twittermarkov -c /usr/local/bin/twitterbots/pastebin_ebooks/bots.yaml tweet
Traceback (most recent call last):
File "/usr/local/bin/twittermarkov", line 11, in
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/twitter_markov/cli.py", line 73, in main
func(**argdict)
File "/usr/local/lib/python2.7/dist-packages/twitter_markov/cli.py", line 80, in tweet_func
tm = TwitterMarkov(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/twitter_markov/twitter_markov.py", line 49, in init
self.api = tbu.API(screen_name=screen_name, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/twitter_bot_utils/api.py", line 83, in init
raise ValueError("Incomplete config. Missing {}".format(missing))
ValueError: Incomplete config. Missing ['consumer_key', 'consumer_secret', 'key', 'secret']

Which I'm taking to mean that it's not understanding my bots.yaml file correctly. Perhaps I'm misunderstanding how to format the command, though.

Thanks!

Limiting length

Is there any way of telling the bot to finish after the first sentence? Just "break on period" or similar? Preferably as a --oneSentence modifier or similar.

Reply To One

Last question, promise. :)

Any way of getting the --reply command to just reply to the one latest unanswered tweet? I'm trying to emulate a "natural" posting schedule using random intervals, post windows, cron jobs, etc, and having a block of multiple replies the same minute looks strange.

Question regarding corpus size

Is there a basic guideline for what size of corpus works best? I'm using your library in a way that could have a corpus of nearly any size (and much bigger than most twitter accounts) and I'm wondering where I should prune it. Thanks!

Corpus updates and learning [questions]

Hi! Two questions:

1. when I want to manually update the corpus, would just plain old adding lines of text work fine "out of the box", or would I need to carry out some re-learn/update process?

2. there is a function that allows to keep learning from another twitter acc

Documentation states:

# If you want your bot to continue to learn, include this
parent: your_screen_name

How does the learning "work"?
Do tweets from that account get gradually added to the txt file specified in the config as corpus?
Or do they live "someplace else"?
How many "parent's tweets" are added "per cron run"?

After upgrading to ubuntu 16.04, no longer recognizes bots.yaml

I upgraded to ubuntu 16.04 and I was wondering if you had any insight as to why my script wasn't working anymore. It seems it just doesn't recognize the config file at all. Here are both the config file I'm using and the command I'm running, as well as the error output:

/usr/local/bin/twittermarkov -c /usr/local/bin/twitterbots/pastebin_ebooks/bots.yaml -u pastebin_ebooks -n -v tweet

Traceback (most recent call last):
  File "/usr/local/bin/twittermarkov", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/twitter_markov/cli.py", line 73, in main
    func(**argdict)
  File "/usr/local/lib/python2.7/dist-packages/twitter_markov/cli.py", line 80, in tweet_func
    tm = TwitterMarkov(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/twitter_markov/twitter_markov.py", line 49, in __init__
    self.api = tbu.API(screen_name=screen_name, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/twitter_bot_utils/api.py", line 83, in __init__
    raise ValueError("Incomplete config. Missing {}".format(missing))
ValueError: Incomplete config. Missing ['consumer_key', 'consumer_secret', 'key', 'secret']

bots.yaml:

apps:
    pastebin_ebooks:
        consumer_key: [c_key]
        consumer_secret: [c_secret]

users:
    pastebin_ebooks:
        key: [key]
        secret: [secret]

        app: pastebin_ebooks

        corpus: /usr/local/bin/twitterbots/pastebin_ebooks/corpus.txt

I've also tried it with a .json file (after seeing the "missing {}" message, to receive the exact same error)

KeyError: 'text'

Any ideas why this is happening?

pi@raspberrypi:~/ThePostmodern $ twittermarkov corpus ./data/tweets.txt
Reading ./data/tweets.txt
Traceback (most recent call last):
  File "/usr/local/bin/twittermarkov", line 9, in <module>
    load_entry_point('twitter-markov==0.4.2', 'console_scripts', 'twittermarkov')()
  File "/usr/local/lib/python2.7/dist-packages/twitter_markov/cli.py", line 77, in main
    func(argdict)
  File "/usr/local/lib/python2.7/dist-packages/twitter_markov/cli.py", line 100, in learn_func
    sys.stdout.writelines(tweets)
  File "/usr/local/lib/python2.7/dist-packages/twitter_markov/cli.py", line 96, in <genexpr>
    tweets = ((tweet.replace('\n', ' ') + '\n') for tweet in gen)
  File "/usr/local/lib/python2.7/dist-packages/twitter_markov/checking.py", line 50, in generator
    for status in tweets:
  File "/usr/local/lib/python2.7/dist-packages/twitter_bot_utils/archive.py", line 34, in read_csv
    tweet['text'] = unicode(tweet['text'], 'utf-8')
KeyError: 'text'

pip install twitter_markov errors out

When I attempt to install twitter_markov via pip install it errors out with the following error message:

Downloading/unpacking wordfilter (from twitter-markov)
  Could not find any downloads that satisfy the requirement wordfilter (from twitter-markov)
  Some externally hosted files were ignored (use --allow-external wordfilter to allow).
Cleaning up...
No distributions at all found for wordfilter (from twitter-markov)
Storing debug log for failure in /Users/nathan/.pip/pip.log

if I --allow-external wordfilter it fails with the same message.

Conflicts between twitter_markov and pyyaml

Hi, users are unable to run twitter_markov due to dependency conflict with pyyaml package. As shown in the following full dependency graph of twitter_markov, twitter_markov requires pyyaml *，while twitter-bot-utils requires pyyaml ==3.11.

According to pip’s “first found wins” installation strategy, pyyaml 5.2 is the actually installed version. However, pyyaml 5.2 does not satisfy ==3.11.

Dependency tree-----------

twitter-markov - 0.5.1
| +- markovify(install version:0.3.1 version range:<0.4,>=0.2.4)
| | +- unidecode(install version:1.1.1 version range:*)
| +- python-levenshtein(install version:0.12.0 version range:<0.13,>=0.12.0)
| | +- setuptools(install version:42.0.2 version range:*)
| +- pyyaml(install version:5.2 version range:*)
| +- six(install version:1.13.0 version range:*)
| +- tweepy(install version:3.8.0 version range:*)
| | +- pysocks(install version:1.7.1 version range:>=1.5.7)
| | +- requests(install version:2.22.0 version range:>=2.11.1)
| | | +- certifi(install version:2019.11.28 version range:>=2017.4.17)
| | | +- chardet(install version:3.0.4 version range:<3.1.0,>=3.0.2)
| | | +- idna(install version:2.8 version range:>=2.5,<2.9)
| | | +- urllib3(install version:1.25.7 version range:<1.26,>=1.21.1)
| | +- requests-oauthlib(install version:1.3.0 version range:>=0.7.0)
| | +- six(install version:1.13.0 version range:>=1.10.0)
| +- twitter-bot-utils(install version:0.11.6.post1 version range:<0.12,>=0.11.6.post1)
| | +- pyyaml(install version:3.11 version range:==3.11)
| | +- tweepy(install version:3.8.0 version range:<4,>=3.5.0)
| | | +- pysocks(install version:1.7.1 version range:>=1.5.7)
| | | +- requests(install version:2.22.0 version range:>=2.11.1)
| | | | +- certifi(install version:2019.11.28 version range:>=2017.4.17)
| | | | +- chardet(install version:3.0.4 version range:<3.1.0,>=3.0.2)
| | | | +- idna(install version:2.8 version range:>=2.5,<2.9)
| | | | +- urllib3(install version:1.25.7 version range:<1.26,>=1.21.1)
| | | +- requests-oauthlib(install version:1.3.0 version range:>=0.7.0)
| | | +- six(install version:1.13.0 version range:>=1.10.0)
| +- wordfilter(install version:0.2.6.2 version range:>=0.1.8)

Thanks for your help.
Best,
Neolith

Issue with non-standard characters?

Is there a potential problem if the corpus contains non-standard characters? I'm thinking, for example, Swedish characters like Å Ä Ö. I'm testing a corpus which contains this kind of character in most rows, and the process seems to hang without outputting anything.

Ending reply chains

Just out of curiosity, have you put any logic into the bot that will end a reply chain after a specific time if two of these bots reply to each other?