hastexo/tutor-contrib-backup

This is an experimental Tutor plugin. You should not consider it ready for production use at this point.

Backup plugin for Tutor

This is a plugin for Tutor that provides backup and restore functionality for MySQL, MongoDB, and Caddy services in both local and Kubernetes Tutor deployments.

In a local deployment, you can run the backup from the command line. The backups are stored as a single compressed tar file, named backup.YYYY-MM-DD.tar.xz, in $(tutor config printroot)/env/backup/. You can then copy the Tutor config root folder to a new host and restore your Open edX environment with the restore command.

In Kubernetes, the plugin runs the backup job as a CronJob by default. You can also run the backup job from the command line. In both cases the backup tar file, named backup.YYYY-MM-DD.tar.xz, is stored in an S3 bucket. Then, in a new Kubernetes deployment, you can use the restore command to restore your Open edX environment. You can even schedule the restore as a CronJob to periodically download the latest backup and restore your environment. This can, for example, be useful if you want to maintain a standby site for disaster recovery purposes.

Version compatibility matrix

You must install a supported release of this plugin to match the Open edX and Tutor version you are deploying. If you are installing this plugin from a branch in this Git repository, you must select the appropriate one:

Open edX release	Tutor version	Plugin branch	Plugin release
Lilac	>=12.0, <13	Not supported	Not supported
Maple	>=13.2, <141	maple	0.3.x
Nutmeg	>=14.0, <15	nutmeg	1.x.x
Olive	>=15.0, <16	main	2.x.x
Palm	>=16.0, <17	main	3.x.x
Quince	>=17.0, <18	main	3.x.x

Installation

pip install git+https://github.com/hastexo/[email protected]

Usage

To enable this plugin, run:

tutor plugins enable backup

Then, run the following command to add the plugin's configuration parameters to your Tutor environment:

tutor config save

Before starting Tutor, build the Docker image:

tutor images build backup

In a Tutor local deployment:

To run a backup in a local Tutor deployment:

tutor local backup

This creates a tar file containing a dump of the MySQL database, a dump of the MongoDB database, and a copy of the Caddy data directory.

You can find the tar file under $(tutor config printroot)/env/backup/.

To restore MySQL, MongoDB, and Caddy from a previously-made backup tar file:

tutor local restore

This will look for the tar file under $(tutor config printroot)/env/backup/, extracts the tar file, and restore the services from the backup files.

You can also exclude specific data from the restore. For example, if you want to restore only MySQL data, and leave the state of MongoDB and Caddy untouched, run:

tutor local restore --exclude=caddy --exclude=mongodb

In a Tutor k8s deployment:

By default, the backup job runs as a scheduled CronJobs once a day at midnight. You can change the schedule by changing the BACKUP_K8S_CRONJOB_BACKUP_SCHEDULE configuration parameter. To suspend the scheduled backup job, set BACKUP_K8S_CRONJOB_BACKUP_ENABLE to false. Note that you need to restart your Kubernetes deployment with tutor k8s quickstart for this change to take effect.

If you need to run the backup job outside the schedule, use:

tutor k8s backup

The backup job backs up MySQL, MongoDB, and the Caddy data directory, creates a tar file with a date stamp, and uploads it to an S3 storage bucket as set by the BACKUP_S3_* configuration parameters. Note that if you want to run multiple backups in a day, you might want to enable object versioning in your bucket. Otherwise, only the last backup taken on any day survives.

You might also consider applying a lifecyle expiration rule to your storage bucket if you want to retain your backups for a limited time, and discard backups beyond a certain age.

If you are running Kubernetes in a production environment, you might need to store the dump and tar files in a generic ephemeral volume instead of the default local node storage (requires Kubernetes 1.23 or higher). To accomplish this, set BACKUP_K8S_USE_EPHEMERAL_VOLUMES to True. Optionally you can change the volume size using the BACKUP_K8S_EPHEMERAL_VOLUME_SIZE variable, which is 10Gi by default.

To restore from the latest version of the backup made today:

tutor k8s restore

To restore from the latest version of the backup made on a particular date:

tutor k8s restore --date={YYYY-MM-DD}

To restore from a particular version of the backup, you first need its version ID. You can either look this up by interacting with your S3 bucket through your S3 provider's CLI or web UI; or you can use the command below to list the version ID and the corresponding timestamp of the last 20 backups:

tutor k8s restore --list-versions

You can change the number of versions shown by passing an integer to the list-versions argument. For example, --list-versions=10. This will start a Kubernetes Job. So the output will be in the corresponding pod's log.

Use the ID of the desired backup version to restore services:

tutor k8s restore --version={version-id}

If you run multiple backups each day and want to restore from a specific version of a backup on a particular day, use --version in combination with --date.

The restore command will start a job that downloads the specified version of the backup from the S3 storage bucket and restores MySQL, MongoDB, and Caddy from it.

You can also exclude specific data from the restore. For example, if you want to restore MySQL and MongoDB data, but not certificate data for Caddy, run:

tutor k8s restore --exclude=caddy

If you want to restore your environment periodically, set the BACKUP_K8S_CRONJOB_RESTORE_ENABLE configuration parameter to true and provide the desired schedule by setting the BACKUP_K8S_CRONJOB_RESTORE_SCHEDULE (by default it is set to once a day at 30 mins past midnight). This will always download the latest version of the backup from the S3 bucket. Note that you need to restart your Kubernetes deployment with tutor k8s quickstart for these changes to take effect.

You can also tweak the history limits for the CronJobs using the BACKUP_K8S_CRONJOB_HISTORYLIMIT_* configuration parameters.

A note on manual restores

The plugin does not stop services while restoring them. Before doing a manual (that is, not CronJob scheduled) restore, you might consider stopping the Caddy service or taking your site offline by other means. When the restore process is complete, remember to bring the Caddy service (or external load balancer or proxy) back online. This prevents your users from encountering errors during the restore process.

Configuration

• BACKUP_K8S_CRONJOB_HISTORYLIMIT_FAILURE (default: 1)
• BACKUP_K8S_CRONJOB_HISTORYLIMIT_SUCCESS (default: 3)
• BACKUP_K8S_CRONJOB_STARTING_DEADLINE_SECONDS (default: 900)
• BACKUP_K8S_CRONJOB_BACKUP_ENABLE (default: true, periodic backup is enabled.)
• BACKUP_K8S_CRONJOB_BACKUP_SCHEDULE (default: "0 0 * * *", once a day at midnight)
• BACKUP_K8S_CRONJOB_RESTORE_ENABLE (default: false, periodic restore is disabled.)
• BACKUP_K8S_CRONJOB_RESTORE_SCHEDULE (default: "30 0 * * *", once a day at 30 mins past midnight)
• BACKUP_K8S_CRONJOB_CONCURRENCYPOLICY (default: "Forbid", see the Kubernetes documentation for other available options)
• BACKUP_K8S_USE_EPHEMERAL_VOLUMES (default: false)
• BACKUP_K8S_EPHEMERAL_VOLUME_SIZE (default: 10Gi)

Make sure the periodic backup job always runs before the restore job during the day.

The following parameters will be pre-populated if the tutor-contrib-s3 plugin is enabled in your Tutor deployment. If you don't have that plugin enabled, or you prefer to use a different S3 service or setting for your backup storage, change these configuration parameters:

• BACKUP_S3_HOST (default: None) - set only if using any other service than AWS S3
• BACKUP_S3_PORT (default: None) - set only if using any other service than AWS S3
• BACKUP_S3_REGION (default: "")
• BACKUP_S3_SIGNATURE_VERSION (default: "s3v4")
• BACKUP_S3_ADDRESSING_STYLE (default: "auto")
• BACKUP_S3_USE_SSL (default: true)
• BACKUP_S3_ACCESS_KEY (default: "{{ OPENEDX_AWS_ACCESS_KEY }}")
• BACKUP_S3_SECRET_ACCESS_KEY (default: "{{ OPENEDX_AWS_SECRET_ACCESS_KEY }}")
• BACKUP_S3_BUCKET_NAME (default: "backups")

These values can be modified with tutor config save --set PARAM_NAME=VALUE commands.

Depending on the nature and configuration of your S3-compatible service, some of these values may be required to set.

• If using AWS S3, you will need to set BACKUP_S3_REGION to a non-empty value. And make sure BACKUP_S3_ADDRESSING_STYLE is set to "auto".
• If you want to use an alternative S3-compatible service, you need to set the BACKUP_S3_HOST and BACKUP_S3_PORT parameters.
• For a Ceph Object Gateway that doesn’t set rgw_dns_name, you will need BACKUP_S3_ADDRESSING_STYLE: path.
• When dealing with very large databases, restoring MongoDB may consume more memory than available and the process may break. In this situation, you can set BACKUP_MONGORESTORE_ADDITIONAL_OPTIONS to -j 1 to limit the number of collections that are restored in parallel in order to save memory.

Selecting databases to backup

By default, all MySQL and MongoDB databases will be included in the backup. This is the desired behavior in most cases.

However, in some situations it may be necessary to explicitly choose which databases must be included in the backup. For example, when the same database cluster is used for other purposes and holds logical databases for other services, you might not want to backup all databases. In addition, if you are using a cloud provider database as a service, the cluster may include internal databases that the LMS user might not be allowed to access, thus throwing an error during the backup process.²

In such cases, you can specify the MySQL databases you would like to back up, using the BACKUP_MYSQL_DATABASES option. This option takes a list of strings with the names of the databases, such as:

BACKUP_MYSQL_DATABASES:
  - openedx
  - notes
  - ecommerce
  - discovery

Remember to include all databases used by edx-platform, as well as those created by any plugins installed.

You may or may not choose to include the internal mysql database in this list. If you have made sure that all your database users and privileges are managed by Tutor (as you should), and none have been created manually, then it should be safe to not include mysql in the BACKUP_MYSQL_DATABASES list. However, if you omit BACKUP_MYSQL_DATABASES from your configuration altogether, then the plugin does include the mysql database in the backup.

For MongoDB databases, the corresponding configuration option is:

BACKUP_MONGODB_DATABASES:
  - openedx
  - cs_comments_service

If your MongoDB instance uses an authentication database name other than admin, make sure you provide that with BACKUP_MONGODB_AUTHENTICATION_DATABASE.

Using this plugin with service version upgrades

In general, you should be able to use this plugin even when you are upgrading to a new MySQL, MongoDB, or Caddy version. This is something that commonly happens as a result of a new Open edX/Tutor major release.

However, you need to make sure that after restoring a backup from a prior version, you run tutor local init or tutor k8s init in order to ensure that any required in-place data modifications also rerun.

Changelog

For a detailed breakdown of features and fixes in each release, please see the changelog.

License

This software is licensed under the terms of the AGPLv3.

Footnotes

1. For Open edX Maple and Tutor 13, you must run version 13.2.0 or later. That is because this plugin uses the Tutor v1 plugin API, which was introduced with that release. ↩

2. There is a known limitation in certain configurations of AWS Aurora in which an attempt to back up the internal mysql database will result in an error: mysqldump: Couldn't execute 'SHOW CREATE PROCEDURE rds_import_binlog_ssl_material': Failed to load routine mysql.rds_import_binlog_ssl_material. The table mysql.proc is missing, corrupt, or contains bad data. ↩

tutor-contrib-backup's Issues

I created two new Github Actions for this plugin

Summary

**This is not a bug report. This is the only place i can make a general announcement since discussion menu is disabled in this repo. **
I created a Github Action that greatly simplifies using this plugin in Kubernetes:

• https://github.com/marketplace/actions/open-edx-tutor-k8s-build-backup-plugin
• https://github.com/marketplace/actions/open-edx-tutor-k8s-enable-backup-plugin

I sent invitations to Foad Lind and and Florian Haas for membership into the new Github organization openedx-actions but both invitations remain pending. it would be great to have both of you involved in any ongoing development of these new Github Actions.

Cannot use plugin when MySQL is not managed by Tutor

Summary

When running a local deployment of tutor where MySQL is not managed by tutor, the plugin fails.

What I did

When running

tutor local backup

I got the following error:

ERROR: Service 'backup-job' depends on service 'mysql' which is undefined.

What I expected to happen

I would expect this dependency to be contingent on RUN_MYSQL to be set to True in the tutor config. I'm happy to open a PR with a fix if that's ok.

Issues backing up `--all-databases`

Summary

When backing up MySQL databases, the plugin dumps all databases.
Specially in Kubernetes deployments, there are some problems with this approach:

• I may be using many databases for different LMS, when deploying many LMS in the same cluster sharing the same MySQL engine (only in Kubernetes). I would like each instance of the plugin to backup only the databases of the corresponding LMS.
• I may be using more databases in the same MySQL cluster that I don't want to backup here
• When using AWS Aurora, it will throw an error mysqldump: Couldn't execute 'SHOW CREATE PROCEDURE rds_import_binlog_ssl_material': Failed to load routine mysql.rds_import_binlog_ssl_material. The table mysql.proc is missing, corrupt, or contains bad data. This is caused by a limitation of AWS Aurora for MySQL 5.7 (works only in 5.6)
• Additionally, using --all-databases may cause permissions issues with the internal tables, as the master user is not allowed to access them.

What I did

This impacts mostly Kubernets with AWS Aurora for MySQL 5.7.

What I expected to happen

I would like to have an option to select the databases I want to backup. This will have to include the edxapp db of course, but also discovery, ecommerce, notes, and any other db that may be used by other plugins. It might be a challenge, as some of them may not even exist.

My environment

• Tutor version: 14.0.1
• tutor-contrib-backup version: 1.0.0
• AWS Aurora RDS for MySQL 5.7

Additional context

Dump only this instance mongo databases

Summary

Hi! It's me again 😊
Something similar to MySQL happens in MongoDB. All databases are dumped, which is a problem when using the same Mongo cluster for many LMS.

What I did

tutor k8s backup

What I expected to happen

Only the two databases related to this LMS should be backed up: one for the courses structures and another for forum

What actually happened

All databases are backed up, including others not related to this LMS.

My environment

Please add some information about the environment that you’re working in. At a minimum, include these items:

• Tutor version: 14.0.1
• tutor-contrib-backup version: 1.1.0

Additional context

Multiple LMS's share the same MongoDB cluster.
There is a plan to deprecate MongoDB in the short term. I wouldn't expect the number of Mongo databases to increase in the future, so I wouldn't spend a lot of effort in a flexible structure.
If you don't disagree, I would just change the mongodump command and dump only these two databases.

Restore job killed in K8s

Summary

Hi there,
I am testing the restore function in my K8s with a very large backup file (5.8G compressed, ~20G plain). It suddenly stops as the pod gets killed.

What I did

tutor k8s restore

What I expected to happen

With smaller files it works like a charm.

What actually happened

My environment

• Tutor version: 14.0.5
• tutor-contrib-backup version: 1.2.0
• Output of the restore command:

2022-10-20T15:04:21.429+0000    finished restoring edxmongo_mynamespace.users (8689 documents, 0 failures)
2022-10-20T15:04:21.561+0000    reading metadata for edxmongo_mynamespace.subscriptions from /data/mongodb_dump/edxmongo_mynamespace/subscriptions.metadata.json
2022-10-20T15:04:21.683+0000    [........................]   edxmongo_mynamespace.modulestore.structures  291MB/10.7GB   (2.7%)
2022-10-20T15:04:21.685+0000    [#.......................]                edxmongo_mynamespace.fs.chunks  343MB/6.60GB   (5.1%)
2022-10-20T15:04:21.685+0000    [#####...................]  edxmongo_mynamespace.modulestore.definitions   139MB/575MB  (24.1%)
2022-10-20T15:04:21.685+0000
2022-10-20T15:04:21.937+0000    restoring edxmongo_mynamespace.subscriptions from /data/mongodb_dump/edxmongo_mynamespace/subscriptions.bson
Killed
Traceback (most recent call last):
  File "restore_services.py", line 247, in <module>
    main()
  File "/s3/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/s3/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/s3/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/s3/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "restore_services.py", line 241, in main
    restore_mongodb()
  File "restore_services.py", line 77, in restore_mongodb
    check_call(cmd,
  File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'mongorestore --stopOnError --drop --host=ip-10-0-9-66.ec2.internal --port=27017 /data/mongodb_dump --username=xxxx --password=xxxx' returned non-zero exit status 137.

Additionally, you might also want to add any of the items below, if you think they might be relevant to the bug:

• Kubernetes versions: 1.21

Additional context

Looks like the pod is running out of memory while restoring MongoDB. I think we could add a memory request to the pod, but this should be an optional parameter. What do you think?

No disk space in K8s

Summary

In K8s, backup is failing because there is not enough disk space in the node to store the db dumps and temporary files

What I did

Just running tutor k8s backup on an instance with large databases, or letting the cronjob run. The job fails and there are disk pressure alarms reported.

What I expected to happen

Backup should happen normally

What actually happened

Backup job fails.

My environment

• Tutor version: 14.0.1

• tutor-contrib-backup version: 1.0.0

• Output of the backup/restore command/CronJob:
  container "backup" in pod "backup-restore-job-20220809082445-265rt" is not available
  The node had condition: [DiskPressure].
  The node was low on resource: ephemeral-storage. Container backup was using 7282412Ki, which exceeds its request of 0.

• Kubernetes versions: Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:17:11Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"darwin/amd64"}
  Kustomize Version: v4.5.4
  Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.13-eks-84b4fe6", GitCommit:"e1318dce57b3e319a2e3fecf343677d1c4d4aa75", GitTreeState:"clean", BuildDate:"2022-06-09T18:22:07Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Additional context

My mongodb dump is about 20Gi.

Restore mysql fails

I have a local openedx installation (Palm edition) and using your plugin I am able to do a backup of Mysql/Mongodb and Caddy.
When I then try to restore it following your instructions, the mysql won't restore.

docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.jobs.yml run --rm backup-job sh -e -c 'python restore_services.py --date=2023-09-13'
[+] Creating 4/0
 ✔ Container tutor_local-caddy-1        Running                                                                                                                                                                                                                        0.0s 
 ✔ Container tutor_local-mysql-1        Running                                                                                                                                                                                                                        0.0s 
 ✔ Container tutor_local-permissions-1  Created                                                                                                                                                                                                                        0.0s 
 ✔ Container tutor_local-mongodb-1      Running                                                                                                                                                                                                                        0.0s 
[+] Running 1/1
 ✔ Container tutor_local-permissions-1  Started                                                                                                                                                                                                                        0.4s 
2023-09-13 09:54:30,295 INFO Extracting archive /data/backup/backup.2023-09-13.tar.xz to /data
2023-09-13 09:54:32,250 INFO Complete. /data is 190286838 bytes.
2023-09-13 09:54:32,250 INFO Restoring MySQL databases on mysql:3306 from /data/mysql_dump.sql
ERROR 3552 (HY000) at line 25: Access to system schema 'mysql' is rejected.
Traceback (most recent call last):
  File "restore_services.py", line 248, in <module>
    main()
  File "/s3/venv/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/s3/venv/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/s3/venv/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/s3/venv/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "restore_services.py", line 240, in main
    restore_mysql()
  File "restore_services.py", line 50, in restore_mysql
    check_call(cmd,
  File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'mysql --host=mysql --port=3306 --user=root --password="xxxx"' returned non-zero exit status 1.
Error: Command failed with status 1: docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.jobs.yml run --rm backup-job sh -e -c python restore_services.py --date=2023-09-13

When investigating further, problems seems related to the mysqldump itself that can't be restored. When trying to restore the mysqldump manually, I get this error.

ERROR 3552 (HY000) at line 25: Access to system schema 'mysql' is rejected.
--> this I could solve by following instructions found here but I wonder if that is really ok ?

But then I get next error:

ERROR 3105 (HY000) at line 218: The value specified for generated column 'default_value' in table 'engine_cost' is not allowed.

So there is an error when trying to recover generated columns but this is now getting out of my league and worrying that the mysqldump needs to be altered to be able to restore it again. I guess that can't be the intention of this plugin who should be plug&play right.
The instructions for using your plugin seems straightforward so no idea what I might be doing wrong.

Need to list minimum required tutor version

I wanted to duplicate a site running tutor 12.0.4 / Lilac. But when I went to install the plugin it upgraded me to tutor 13.3.1. That seems like a very bad thing to happen automatically, as it will break all sorts of stuff. So please list the version requirements on the site (apparently currently tutor >= 13.2.0, in which case I would have known I couldn't use this plugin.) This also somehow reactivated a vestigial Cairn plugin

ubuntu@ip-172-31-94-95:~$ tutor --version
tutor, version 12.0.4
ubuntu@ip-172-31-94-95:~$ sudo su tutor
tutor@ip-172-31-94-95:/home/ubuntu$ pip install git+https://github.com/hastexo/[email protected]
Collecting git+https://github.com/hastexo/[email protected]
  Cloning https://github.com/hastexo/tutor-contrib-backup (to revision v0.1.0) to /tmp/pip-req-build-69wule_a
  Running command git clone -q https://github.com/hastexo/tutor-contrib-backup /tmp/pip-req-build-69wule_a
  Running command git checkout -q 3d3ea6fc13b69338d15b74adc30dfb8d14f34cd5
Collecting tutor<14,>=13.2.0
  Downloading tutor-13.3.1.tar.gz (75 kB)
     |████████████████████████████████| 75 kB 5.0 MB/s 
Requirement already satisfied: appdirs in /home/tutor/.local/lib/python3.8/site-packages (from tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (1.4.4)
Requirement already satisfied: click>=7.0 in /usr/lib/python3/dist-packages (from tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (7.0)
Requirement already satisfied: jinja2 in /usr/lib/python3/dist-packages (from tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (2.10.1)
Requirement already satisfied: kubernetes in /home/tutor/.local/lib/python3.8/site-packages (from tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (12.0.1)
Requirement already satisfied: mypy in /home/tutor/.local/lib/python3.8/site-packages (from tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (0.910)
Requirement already satisfied: pycryptodome in /home/tutor/.local/lib/python3.8/site-packages (from tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (3.10.1)
Requirement already satisfied: pyyaml>=4.2b1 in /usr/lib/python3/dist-packages (from tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (5.3.1)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (2.22.0)
Requirement already satisfied: urllib3>=1.24.2 in /usr/lib/python3/dist-packages (from kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (1.25.8)
Requirement already satisfied: google-auth>=1.0.1 in /home/tutor/.local/lib/python3.8/site-packages (from kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (1.27.1)
Requirement already satisfied: python-dateutil>=2.5.3 in /home/tutor/.local/lib/python3.8/site-packages (from kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (2.8.1)
Requirement already satisfied: six>=1.9.0 in /usr/lib/python3/dist-packages (from kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (1.14.0)
Requirement already satisfied: certifi>=14.05.14 in /usr/lib/python3/dist-packages (from kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (2019.11.28)
Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /home/tutor/.local/lib/python3.8/site-packages (from kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (0.58.0)
Requirement already satisfied: setuptools>=21.0.0 in /usr/lib/python3/dist-packages (from kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (45.2.0)
Requirement already satisfied: requests-oauthlib in /home/tutor/.local/lib/python3.8/site-packages (from kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (1.3.0)
Requirement already satisfied: typing-extensions>=3.7.4 in /home/tutor/.local/lib/python3.8/site-packages (from mypy->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (3.10.0.0)
Requirement already satisfied: mypy-extensions<0.5.0,>=0.4.3 in /home/tutor/.local/lib/python3.8/site-packages (from mypy->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (0.4.3)
Requirement already satisfied: toml in /home/tutor/.local/lib/python3.8/site-packages (from mypy->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (0.10.2)
Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3.6" in /home/tutor/.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (4.7.2)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/tutor/.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (4.2.1)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/lib/python3/dist-packages (from google-auth>=1.0.1->kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (0.2.1)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/lib/python3/dist-packages (from requests-oauthlib->kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (3.1.0)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/lib/python3/dist-packages (from rsa<5,>=3.1.4; python_version >= "3.6"->google-auth>=1.0.1->kubernetes->tutor<14,>=13.2.0->tutor-contrib-backup==0.1.0) (0.4.2)
Building wheels for collected packages: tutor-contrib-backup, tutor
  Building wheel for tutor-contrib-backup (setup.py) ... done
  Created wheel for tutor-contrib-backup: filename=tutor_contrib_backup-0.1.0-py3-none-any.whl size=15813 sha256=ee68ef932805c88fa7364acc9285807cd62ba096da6367907df3dd7abee39453
  Stored in directory: /tmp/pip-ephem-wheel-cache-bero_sgi/wheels/95/12/5f/4939537c5fe3aa4bc00211776e7c2ecdf550ef1a6dc03e5845
  Building wheel for tutor (setup.py) ... done
  Created wheel for tutor: filename=tutor-13.3.1-py3-none-any.whl size=106377 sha256=5f93bbdd23fe93f712748bcac23c7010134a3ff78207afc73a91573aeadf2b3b
  Stored in directory: /home/tutor/.cache/pip/wheels/39/cd/42/70272cdb7d98c305d81dd6432f795f601b017d6748722415de
Successfully built tutor-contrib-backup tutor
ERROR: tutor-cairn 12.0.8 has requirement tutor<13.0.0,>=12.0.0, but you'll have tutor 13.3.1 which is incompatible.
Installing collected packages: tutor, tutor-contrib-backup
  Attempting uninstall: tutor
    Found existing installation: tutor 12.0.4
    Uninstalling tutor-12.0.4:
      Successfully uninstalled tutor-12.0.4
Successfully installed tutor-13.3.1 tutor-contrib-backup-0.1.0
tutor@ip-172-31-94-95:~$ tutor --version
tutor, version 13.3.1