jfmatth/fishtankclient

Client side of fishtank

Hi there 👋

• ⚡ Excited to start a job at Public Storage as Director, Cloud Operations.
• 🔭 I’m currently working on my Kubernetes clusters using Helm
• Gave a fun talk on PIKU at Scale21x this year.

fishtankclient's Issues

Check for settings.txt.

currently, if settings.txt doesn't exist, the settingsmanager just creates it and moves on, which shouldn't be the case. It should fail if the file doesn't exist.

Task - Restores

need a new task scheduled that looks for restore jobs.

Wrote up design doc in google and waiting for review

Use python requests

requests is a great library, that simplifies HTTP calls (gets and posts).

the code is littered with HTTP GET's and POST's, but the restore class uses requests. Convert our existing code to use requests.

testing scripts

tasks interval from settings

Currently, when the agent is setup either in agent.py or agentservice.py, the tasks for the agent are hard coded times.

These times should come from the settings manager, that way there are no hard coded times in the system.

Consolidate agent code

Currently, agent.py and agentservice.py serve the same purpose.

One is for running from command line (python agent.py) and the other for the agent package as a service (agentservice.exe install / start).

The code base is similar but should be consolidated.

The agentservice.py code already has provisions for running as "python.exe", so we should try to combine the functionality into one file (agentservice.py)

settingsmanager, refresh local settings too.

either refresh the local settings, or don't write local settings to DB, i think the latter is easier.

torrent class runs 100% when peer can't be reached.

when a peer can't be reached, or even the tracker (because of firewall), the torrent class runs 100% on client.

Verify settings

need a routine that brings down all the settings for a client, and checks to make sure each path is valid, and either errors or creates them for us.

Backup least copied torrent

when /getcloud is called, it returns a hash of which torrent needs a backup created, but it's sorted naturally, and always starts at the last record that meets criteria. Need to sort by client count, so those torrents with the fewest copies get spread out first.

Clean up local torrents when needed

currently, the local "cloud" directory is just added to.

A mechanism needs to be called that allows it to be cleaned up from time to time, typically when the system starts and maybe when a trigger is set off.

Several scenarios need to be accounted for:

1. Torrent files exist on the client, that aren't on the tracker for this client.
2. Torrent files exist without a fast-resume file attached.
3. Tracker thinks that files should be on client, but are not, remove those from the tracker.

threading failure on service exit

when the service finishes, it throws a threading error. No rhyme or reason yet.

Improve the way settingsmanager GET's things from manager

right now settingsmanager uses /manager/settings//setting to retrieve info from the manager.

Since this is a HTTP GET, we should use parameters instead of the URL, so the new format would be

HTTP GET /manager/setting?guid=&key=

Sync local backup (fulldb.dbm) with cloud

When backups are removed from the cloud, those changes need to be reflected in the client that they originated from in it's fulldb.dbm file.

When a backup is 'deleted', it gets flagged, a client query is then run to ask which files are being removed from the cloud, those files are removed from the local DB and then tell the tracker to go ahead and remove the backups from the cloud.

this makes sure that stale files get filter out from time to time, but still provide a way to keep a client fully backed up.

Check free disk space before each backup set

I don't check to make sure the system has enough space to create a backup "set".

register.py - static settings.txt usage

register.py should get it's registration key from the command line, not settings.txt.

working directory - not saved during service usage

when testing the agent piece (python agent.py), the environment knows it's local working directory (i.e. C:/fishtankclient/) but when run as a service (agentservice.py) it defaults to c:\windows\system32, and that won't work.

Need a way to save the installation directory.

Fix Date / Time on upload of backup file.

buildclient.cmd - look in build directory for file

buildclient.cmd has evolved over the months, and now that we have the basics for EXE's and services built, when building a client via buildclient.cmd, it should look in the build\agent\ folder under fishtankclient to get its files.

settingsmanager, remove from db when not on /manager/settings

when settingsmanager.settings fires up for the first time, it refreshes it's keys in the local DB copy to make sure it's all good, however, if a setting returns None, it should delete from the DB

Reload Tasks

Need a mechanism to reload the Task list (Tasker queue) upon command or in a timely manner.

Drives to backup

right now, there is a setting called backupdrives, which is LAME.

New code needs to figure out what local drives are available and back them up.

Maximum size file setting

Need to not backup files over a certain size, it just doesn't make sense?

Link torrent to backup record

We need a way that when a torrent is uploaded to the system, it gets linked to it's associated backup record.

If when uploading a torrent, we can't find a torrent record, then we fail somehow. Uploading a torrent without the backup record is a no-no.

Change how settings[] are identified

right now, local .txt file settings are prefixed with . and server field settings are prefixed with _, this makes now sense at all.

Should prefix local settings with local. and server settings with server.

IP Address update

Need the client to update it's IP address to the manager.

This should be a first task on the tasker in the client, that way we don't need to update the info from register.py, and instead, just pass down it's GUID and other registration information.

Create two EXE's for client

using pyinstaller, we need two EXE's for the download client

register.exe (.py) allows us to register the client
agentservice.exe (.py) - whole agent service for backup.

Stop() callback for all functions

Need a way to tell a function to stop by adding a callback function to each sub-process of backup.

so, if the stop() function returns true, we can return quickly.

this revolves around the service aspect of the application

Installer for client

Use one of the many free utilities to make an .EXE installer for the client.

It must keep the same functionality as register.py does today, in fact, it should call register.py during install.

It will prompt for the registration key or allow it to be passed on the command line.

I downloaded Advanced Installer 8.2 to use, but haven't built anything yet.