If TwtApi.db is empty, everything fails

[cruisedevice]

If TwtApi.db is empty, everything fails

[iminier]

@ediamondscience Okay, this is starting to sound really cool. I agree, my solution only solves the issue for one account and removes the need for the SQL DB. Lines of code aside, I personally don't think a plain text file is very structured.

I would instead vote for a JSON file. You can update JSON files from a running script and then use that same file's new data. Then you can remove setup.py and instead run a setup script from main() in twweet_cli.py automatically if the JSON file has not been created. If it has been created main() checks to see how many accounts are in the JSON file, configures the application and continues running.

editAPI() can be re-written to update the JSON file and when it triggers main() it reconfigures again.

[moyamanuel]

I think there are two options here:

1. If the program detects a system exit, it deletes TwtApi.db and prompts the user to rerun the setup. The downside would be that the user would need to rerun the setup again.

2. The other option would be pre-populating the database with dummy values like you stated. I don't see any downside to this option.

I'd like to hear other ideas or feedback as to how we can solve this issue in a clean, yet an intuitive way for the user.

[ediamondscience]

I think there may be a third option here: removing support for the SQL database entirely and storing the keys in a text file. Currently the SQL db is just a permanent container for the keys, we could just as easily make this a simpler (and possibly more versatile) piece of code by removing the dependency. Unless there's some reason I'm not seeing that makes SQL a better choice to hold onto the user keys, I think it's unnecessary. Python has great built-in tools for reading and writing to text files, so it's fewer lines of code as well.

[iminier]

You can also store the keys in another python file and then

from file import *

file.py

consumer_key = 'string'
consumer_secret = 'string'
access_token = 'string'
access_token_secret = 'string'

[ediamondscience]

True, but can you set and store them between sessions in the python file? That's where I seem to be drawing a blank with this route.

[iminier]

Not sure I follow as Twitter access tokens aren't set to expire unless revoked.

[moyamanuel]

@iminier, what would be the benefit of storing the keys in a python file vs. a regular text file?

[ediamondscience]

@iminier Can you store the key values in the python file even once you close twweet-cli and the python command line? You're correct that the key values don't change frequently, so I figured we would try to save them between sessions so that the user doesn't have to enter them every time they start the program. I know how to do that by reading and writing to a text file, but I don't know how to permanently write to a python file, is there some way to permanently set the variables from the python command line?

[iminier]

I think the benefit would be writing less code or doing less. Using a text file, you need to open the file, loop through the lines, set variables, close file. Instead, you can have setup.py take credentials and store them to the file before running twweet_cli.py

[ediamondscience]

@iminier I see where you're coming from, but then the only way to set the keys is through setup.py. This could hurt us if we want to add some form of modularity to the number of accounts you can post from later on down the road . I think that would not only be better for the interface (as we could then handle api errors by having users validate their keys) but allow us to make a feature where you can post to multiple accounts at once if we stored the keys in separate text files (one for each account).

As for it being less lines of code, I think we're picking nits if we're worried about the difference between four and six lines. Thankfully twweet-cli is pretty lightweight as it currently stands, so I think we have some room to grow and add features.

[ediamondscience]

@iminier I like the idea of having more structure too, but I'd really like a file I can just copy-paste into if I'm being totally honest. The current interface is a pretty big pain because it makes the user type out four long keys into the python console. I want something where I can copy and paste my keys into gedit, save the file and move it into the proper directory so I can continue to be a lazy typer. I'm tempted to add a whole directory where we would store text files with four lines worth of keys there so I could loop through them with python and not have to worry about adding and removing user accounts by doing a lot of typing within the actual program itself.

What we could to is have the program take the keys in text files from a key directory and have it generate a JSON file that describes those accounts in better detail that the program would then read on start up. We can find out everything we need by querying the api with the keys, and storing these values in a file like this would be pretty helpful because you wouldn't need to run those queries every time the program started.

[iminier]

I agree, adding a directory to store the files and leaving the plain text would make it the easiest for the user. Regarding the creation of files, you could leave it "open" in a sense. Modify it so a user can drag and drop files into the specified directory and restart the application or the application can create a file from the user input through the application.

The JSON is simply to make a file with numerous accounts easier to read and update manually. Having multiple text files solves that issue so I don't think JSON would be necessary anymore.

[ediamondscience]

@iminier I was actually thinking we should do both. I really like the idea of having a really nice, structured place to store the account credentials along with other interesting things about the accounts (screen name/follow list/etc) like you recommended is a very good idea and I think a JSON file would fit the bill, as it lets us store a great many more things about the accounts we can post from without having to worry about which line it is on a text file.

While we could accomplish this by just using the text file idea, we'd have to send out a number of queries to Twitter's API each time the application started and then we'd have to worry about handling all sorts of rate limiting exceptions and 429 status returns.

I think we should try to set up a JSON file that stores account information and then set up a python function to read in our input directory and make all the relevant queries to the Twitter API just once per account with checks in place for duplicate accounts and have that modify our JSON file to contain the account keys as well as the extraneous account information we choose to store offline.

[iminier]

@ediamondscience sorry I got busy with work.

Would you be limited by IP or is it only by request from a user? Additionally, in what case would you query the Twitter API rapidly. Will this eventually grow to offer automation on multiple accounts?

[ediamondscience]

@iminier In the API documentation it implies that the limits are on a per user account basis, so that's why I'm curious about whether or not we could automate several accounts to do larger scale data mining. I mostly want to see what protections against this sort of thing that Twitter has in place. I can't imagine what one could do with a ton of Twitter data, but I bet someone could do absolutely insane things with that data if it's sifted through properly.

For example, imagine if you had twweet-cli pull tweets that fulfill the requirement that they are part of a chain of conversations at least x tweets long. If you could filter down that data a bit to exclude people replying to themselves and other weird anomalies, I bet you could throw the tweets into Tensorflow and have a pretty experienced conversational AI. There are loads of things you could do.

As far as twweet-cli goes, I think that automation offers the best value to the end user. We want to give people a reason to use our program, and offering better automation than the standard web browser based twitter interface will help us fill this niche. I think that the automation on one account is probably the hard part in terms of coding a program that automates several users. I imagine we'll need a new function in the program for automation. If we can code that function to properly automate a single account, we can apply it to multiple accounts with just a single for loop.

[0x006E]

Hey guys, The sqlite db was used just to protect editing from text editors and reading using text editors.. I thought of implementing text file but then shifted to sqlitedb.. Anyway good info from you guys. These keys are used to be kept secret I think. And also I couldn't reproduce your issue IDK why.. Correct if I am wrong :)

[mblissmer]

Hey all,

This is literally my first comment in a github conversation, so please let me know if I'm breaking any sort of rules or social norms or whatever. This was listed on Up For Grabs, which is how I got here.

I'm fairly new at python (been messing with it for about a year), but I just this last week wrote a few quick functions that would pull data from a json file (it was a bunch of directory paths, but there's no reason it couldn't be used to store usernames and such), and create a new json file there if none was found.

But @nithinswarrier is right, you can just pop in there and edit it with notepad or whatever... which is both good and bad.

I'd be happy to add that to the project if it would be helpful!

[0x006E]

@mblissmer Good luck with your JSON Pulling.. Tag Akash and send a PR. He will look to it. :) Anyway Good luck

[mblissmer]

@CruiseDevice This is me tagging you! I've sent a PR.

[0x006E]

@andypiper @CruiseDevice Please close this issue.. This has been fixed in PR #22