tsileo/bakthat

Bakthat is a MIT licensed backup framework written in Python, it's both a command line tool and a Python module that helps you manage backups on Amazon S3/Glacier and OpenStack Swift. It automatically compress, encrypt (symmetric encryption) and upload your files.

Bakthat

I stopped working on bakthat, I'm now working on another backup-related project: blobstash/blobsnap. If somebody want to be a collaborator and continue development, please open an issue.

Bakthat is a MIT licensed backup framework written in Python, it's both a command line tool and a Python module that helps you manage backups on Amazon S3/Glacier and OpenStack Swift. It automatically compress, encrypt (symmetric encryption) and upload your files.

Here are some features:

• Compress with tarfile
• Encrypt with beefish (optional)
• Upload/download to S3 or Glacier with boto
• Local backups inventory stored in a SQLite database with peewee
• Delete older than, and Grandfather-father-son backup rotation supported
• Possibility to sync backups database between multiple clients via a centralized server
• Exclude files using .gitignore like file
• Extendable with plugins

You can restore backups with or without bakthat, you just have to download the backup, decrypt it with Beefish command-line tool and untar it.

Check out the documentation to get started.

Overview

Bakthat command line tool

$ pip install bakthat

$ bakthat configure

$ bakthat backup mydir
Backing up mydir
Password (blank to disable encryption): 
Password confirmation: 
Compressing...
Encrypting...
Uploading...
Upload completion: 0%
Upload completion: 100%

or

$ cd mydir
$ bakthat backup

$ bakthat show
2013-03-05T19:36:15 s3  3.1 KB  mydir.20130305193615.tgz.enc

$ bakthat restore mydir
Restoring mydir.20130305193615.tgz.enc
Password: 
Downloading...
Decrypting...
Uncompressing...

$ bakthat delete mydir.20130305193615.tgz.enc
Deleting mydir.20130305193615.tgz.enc

Bakthat Python API

import logging
import sh
logging.basicConfig(level=logging.INFO)

from bakthat.helper import BakHelper

BACKUP_NAME = "myhost_mysql"
BACKUP_PASSWORD = "mypassword"
MYSQL_USER = "root"
MYSQL_PASSWORD = "mypassword"

with BakHelper(BACKUP_NAME, password=BACKUP_PASSWORD, tags=["mysql"]) as bh:
    sh.mysqldump("-p{0}".format(MYSQL_PASSWORD),
                u=MYSQL_USER,
                all_databases=True,
                _out="dump.sql")
    bh.backup()
    bh.rotate()

Changelog

0.7.0

Not released yet, developed in the develop branch.

• Incremental backups support, with Incremental-Backups-Tools.
• Revamped configuration handling
• Stronger unit tests
• Plugin architecture improved
• Switch from aaargh to cliff for the CLI handling

0.6.0

June 5 2013

• Event hooks handling
• Support for plugin

0.5.5

May 26 2013

• Support for excluding files, using .bakthatexclude/.gitignore file, or a custom file.
• Added support for reduced redundancy when using S3

0.5.4

May 8 2013

• Better log handling
• Allow more complex rotation scheme

0.5.3

May 6 2013

• Bugfix config

0.5.2

May 6 2013

• new BAKTHAT_PASSWORD environment variable to set password from command line.

0.5.1

May 5 2013

• New -c/--config argument.
• New periodic_backups command tied to BakManager API.

0.5.0

April 21 2013

• New Swift backend, thanks to @yoyama
• ls command removed in favor of the show command
• Compression can now be disabled with the compress setting
• Bugfix default destination

0.4.5

Mars 20 2013

• bugfix configure (cancel of configure cmd cause empty yml), thanks to @yoyama
• new bakthat.helper.KeyValue
• BakSyncer improvement

0.4.4

Mars 10 2013

• bugfix (forgot to remove a dumptruck import)

0.4.3

Mars 10 2013

• bakthat show bugfix

0.4.2

Mars 10 2013

• Using peewee instead of dumptruck, should be Python2.6 compatible again.

0.4.1

Mars 8 2013

• small bugfix when restoring from glacier
• bakhelper now support custom configuration and profiles
• aligned date in show command

0.4.0

If you come from bakthat 0.3.x, you need to run:

$ bakthat upgrade_to_dump_truck

And you also need to run again bakthat configure.

$ cat ~/.bakthat.conf
$ bakthat configure

Changes:

• The filename is now a positional argument for all command
• Using DumpTruck instead of shelve
• Save backups metadata for both backends
• BakHelper to help build backup scripts
• BakSyncer to help keep you list sync over a custom REST API
• Now adding a dot between the original filename and the date component
• Tags support (-t/--tags argument)
• Profiles support (-p/--profile argument)
• New show command, with search support (tags/filename/destination)
• Hosted documentation

0.3.10 ~~~~~

• bug fix glacier upload

0.3.9

• small bug fixes (when updating an existing configuration)

0.3.8

• Added delete_older_than command
• Added rotate_backups command (Grandfather-father-son backup rotation scheme)

Contributors

• Eric Chamberlain
• Darius Braziunas
• Sławomir Żak
• Andreyev Dias de Melo
• Jake McGraw
• You Yamagata
• Jordi Funollet

License (MIT)

Copyright (c) 2012 Thomas Sileo

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

bakthat's Issues

Restoring a file dies with this message

Hey again,

Went to test the restore of a tar.gz file,now using a long name without the double tar.gz. Can you actually add a period back after the last part of the filename that way the date code doesnt get jumbled together.

Here is what I get:

2013-02-26 16:33:13,182 INFO: Restoring gridlock.XXXX.org.23420130221100557.tgz.enc <*_**-- note the 234 is the last part of the file name and would like that separate from the date
Password:
2013-02-26 16:33:43,045 INFO: Downloading...
Traceback (most recent call last):
File "/usr/local/bin/bakthat", line 9, in
load_entry_point('bakthat==0.3.7', 'console_scripts', 'bakthat')()
File "/usr/local/bin/bakthat.py", line 598, in main
app.run()
File "/usr/local/lib/python2.6/dist-packages/aaargh.py", line 157, in run
return func(__kwargs)
File "/usr/local/bin/bakthat.py", line 506, in restore
out = storage_backend.download(key_name, *_download_kwargs)
File "/usr/local/bin/bakthat.py", line 267, in download
job = self.vault.retrieve_archive(archive_id)
File "/usr/local/lib/python2.6/dist-packages/boto/glacier/vault.py", line 293, in retrieve_archive
response = self.layer1.initiate_job(self.name, job_data)
File "/usr/local/lib/python2.6/dist-packages/boto/glacier/layer1.py", line 351, in initiate_job
response_headers=response_headers)
File "/usr/local/lib/python2.6/dist-packages/boto/glacier/layer1.py", line 84, in make_request
raise UnexpectedHTTPResponseError(ok_responses, response)
boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 202, got (400, code=InvalidParameterValueException, message=Invalid vault name: arn:aws:glacier:us-east-1:917509789911:vaults/backups-east)

store archive contents in sqlite

I am new to Amazon Glacier and bakthat. Bakthat seems to have a lot of nice features and I thank you for writing this software. I did one test backup and reviewed many of the options and I have a recommendation for an improvement. When backing up directories it is great that they are tar.gz'ed automatically, yet the contents of the archive are unknown. It would be useful if the files associated with an archive were stored in the sqlite database. It could be as simple as a json blob with the files (possibly with timestamps, size, perms, etc.) added to the archive or in another table associated to the archive. Right now this is my only reservation with using Glacier for a bulk backup because I want to be sure I know that the version of the files I want are in the archive I restore. I am a Python developer and am willing to try to incorporate this feature, but I figured I may have overlooked something obvious or this improvement may be trivial to add by someone who has more familiarity with the code.

Thanks!

reduced redundancy on S3

It will be great to be able to choose to store on S3 reduced redundancy from the command line.

Restore dies with "Killed" and no other info

Hey there,

The last python fixes worked fine, I am able to upload fine but when I go to restore a large file (16GB) I get the following:

root@dagobah:/home/ecarter# bakthat restore -f dantooine.253.tar.gz -d glacier
2013-02-13 11:06:11,890 INFO: Restoring dantooine.253.tar.gz20130212120602.tgz.enc
Password:
2013-02-13 11:06:22,036 INFO: Downloading...
2013-02-13 11:06:22,132 INFO: Job ArchiveRetrieval: Succeeded (2013-02-12T20:47:54.042Z/2013-02-13T00:47:58.206Z)
2013-02-13 11:06:22,132 INFO: Downloading...
Killed

There is no other information, I have looked at the source code and didnt see anything that stuck out at me. Is the file being loaded into memory first (could it be running out of memory while downloading?) I also didnt see any files in /tmp that appeared to be growing while running.

Let me know what I can do to debug. Thanks.

first_week_day isn't respected

I'm not 100% sure about this one, however:

 > bakthat configure_backups_rotation
 first week day (entered 'Monday')

in /xxxxx/.bakthat.yml

      first_week_day: 0

Monthlies seem to default to 1st of the month (which is of course fine), but weeklies on the other hand seems to keep Saturdays rather than Mondays.

Problem in rotate_backups function

import bakthat

bakthat_conf = {
    "default": {
        'access_key': 'my_access_key',
        'secret_key': 'my_secret_key',
        'glacier_vault': 'my_vault',
        's3_bucket': '',
        'compress': False,
        'region_name': 'my_region_name'',
         'rotation': {
             'days': 1,
             'first_week_day': 5,
             'months': 0,
             'weeks': 0
         }
    }

bakthat.backup("/home/user/my_log.log", config=bakthat_conf, destination="glacier", prompt="no")  # this guy works

bakthat.rotate_backups("log", config=bakthat_conf, destination="glacier")  # but this guy won't :(

Traceback:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-72-bcaeca070a0e> in <module>()
----> 1 bakthat.rotate_backups("log", config=bakthat_conf, destination="glacier")

/home/tomek/.virtualenv/double/local/lib/python2.7/site-packages/bakthat/__init__.pyc in rotate_backups(filename, destination, profile, config, **kwargs)
    150     deleted = []
    151 
--> 152     backups = Backups.search(filename, destination, profile=profile, config=config)
    153     backups_date = [datetime.fromtimestamp(float(backup.backup_date)) for backup in backups]
    154 

/home/tomek/.virtualenv/double/local/lib/python2.7/site-packages/bakthat/models.pyc in search(cls, query, destination, **kwargs)
     92         conf = config
     93         if kwargs.get("config"):
---> 94             conf = load_config(kwargs.get("config"))
     95 
     96         if not destination:

/home/tomek/.virtualenv/double/local/lib/python2.7/site-packages/bakthat/conf.pyc in load_config(config_file)
     21     """ Try to load a yaml config file. """
     22     config = {}
---> 23     if os.path.isfile(config_file):
     24         log.debug("Try loading config file: {0}".format(config_file))
     25         config = yaml.load(open(config_file))

/home/tomek/.virtualenv/double/lib/python2.7/genericpath.pyc in isfile(path)
     27     """Test whether a path is a regular file"""
     28     try:
---> 29         st = os.stat(path)
     30     except os.error:
     31         return False

TypeError: coercing to Unicode: need string or buffer, dict found

Does not work with the 'eu-west-1' region

I run it:

import bakthat
bakthat_conf = {'access_key': '%KEY1%',
                'secret_key': '%KEY2%',
                's3_bucket': '%my_backet_in_europe%',
                'region_name': 'eu-west-1'}
bakthat.backup("%/home/my/dir/to/backup%", destination='s3', conf=bakthat_conf)

and script print that:

Backing up %/home/my/dir/to/backup%
Password (blank to disable encryption):
Compressing...
Uploading...
Upload completion: 0%
Upload completion: 0%
Upload completion: 0%
Upload completion: 0%
Upload completion: 0%
Upload completion: 0%
Upload completion: 0%
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "bakthat/__init__.py", line 311, in backup
    storage_backend.upload(stored_filename, outname)
  File "bakthat/backends.py", line 109, in upload
    k.set_contents_from_filename(filename, **upload_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1187, in set_contents_from_filename
    encrypt_key=encrypt_key)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1121, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 827, in send_file
    query_args=query_args)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 490, in make_request
    override_num_retries=override_num_retries)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 932, in make_request
    return self._mexe(http_request, sender, override_num_retries)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 894, in _mexe
    raise e
socket.error: [Errno 104] Connection reset by peer

But! If I run it:

import bakthat
bakthat_conf = {'access_key': '%KEY1%',
                'secret_key': '%KEY2%',
                's3_bucket': '%my_backet_in_us_east%',
                'region_name': 'us-east-1'}
bakthat.backup("%/home/my/dir/to/backup%", destination='s3', conf=bakthat_conf)

and script print that (and create backup at s3):

Backing up %/home/my/dir/to/backup%
Password (blank to disable encryption):
Compressing...
Uploading...
Upload completion: 0%
Upload completion: 11%
Upload completion: 22%
Upload completion: 33%
Upload completion: 44%
Upload completion: 55%
Upload completion: 66%
Upload completion: 77%
Upload completion: 88%
Upload completion: 100%
{%some_data_about_created_backup%}

Swift auth docs aren't clear

It took me a while to figure out how to upload stuff to a swift server. The docs say access_key: ACCOUNT:USER which wasn't clear to me. I couldn't get v2 (aka Keystone) auth working. For v1, these are example settings that work for Rackspace CloudFiles:

  access_key: <RACKSPACE_USERNAME>
  auth_url: https://auth.api.rackspacecloud.com/v1.0
  auth_version: '1'
  default_destination: swift
  glacier_vault: ''
  region_name: ''
  s3_bucket: bakthat_test
  secret_key: <RACKSPACE_SECRET_KEY>

Is there something in bakthat that would prevent logging from preserving its format string?

Hi, am using:

logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s', datefmt='%d/%m/%Y %H:%M:%S', level=getattr(logging, options.loglevel.upper()))

logging.info('Starting %s v%s, environment: %s, loglevel: %s' % (__file__, __version__, STRENV, options.loglevel.upper()))

.... with good success in all my scripts, but within the ones that leverage bakthat for some reason the format is completely ignored.

I suspect it's to do with a conflict in logger name, however I'm afraid I'm not yet enough of an expert to be 100% of this.

No distributions matching the version for byteformat

This error appear when installing bakthat.

pip install bakthat
.....
Downloading/unpacking byteformat
Could not find a version that satisfies the requirement byteformat (from versions: 0.1a, 0.2a)
Cleaning up...
No distributions matching the version for byteformat

bakthat dumps glacier backups to /tmp

after I run

$ bakthat backup -d glacier /data/backuppc/cpool/ --prompt no
bakthat does its job of backing up the necessary directory. but it dumps backups in /tmp

I havent made any adjustments to the bakthat code so I'm not quite sure why this is happening or where to start troubleshooting this error..

please help

Thanks

Possible to use bakthat for similar files from multiple servers?

Say I've got 3 pairs of redis servers and I'd like to backup their dump.rdb files to the same s3 bucket, is there anyway to prefix the s3 key to disambiguate the three sources? ( eg "pair1.dumprdb.{timestamp}.gz" or prefix them by / "pair1/dumprdb.{timestamp}.gz" from the configuration file?

Upgrade errors (unable to run bakthat upgrade_to_dump_truck)

Hey there I ran pip install bakthat -U

got this back:

root@dagobah:/home/xxxx# pip install bakthat --upgrade
Downloading/unpacking bakthat from http://pypi.python.org/packages/source/b/bakthat/bakthat-0.4.0.tar.gz#md5=40ce177b27f0b0407b9ce761e38434a3
Downloading bakthat-0.4.0.tar.gz
Running setup.py egg_info for package bakthat
Requirement already up-to-date: aaargh in /usr/local/lib/python2.6/dist-packages (from bakthat)
Requirement already up-to-date: boto in /usr/local/lib/python2.6/dist-packages (from bakthat)
Requirement already up-to-date: pycrypto in /usr/local/lib/python2.6/dist-packages (from bakthat)
Requirement already up-to-date: beefish in /usr/local/lib/python2.6/dist-packages (from bakthat)
Downloading/unpacking grandfatherson (from bakthat)
Downloading GrandFatherSon-1.1.tar.gz
Running setup.py egg_info for package grandfatherson
Downloading/unpacking dumptruck (from bakthat)
Downloading dumptruck-0.1.4.tar.gz
Running setup.py egg_info for package dumptruck
Downloading/unpacking byteformat (from bakthat)
Downloading byteformat-0.2a.tar.gz
Running setup.py egg_info for package byteformat
Downloading/unpacking pyyaml (from bakthat)
Downloading PyYAML-3.10.zip (364kB): 364kB downloaded
Running setup.py egg_info for package pyyaml
Downloading/unpacking sh (from bakthat)
Downloading sh-1.08.tar.gz
Running setup.py egg_info for package sh
Requirement already up-to-date: argparse in /usr/local/lib/python2.6/dist-packages (from aaargh->bakthat)
Installing collected packages: bakthat, grandfatherson, dumptruck, byteformat, pyyaml, sh
Found existing installation: bakthat 0.3.7
Uninstalling bakthat:
Successfully uninstalled bakthat
Running setup.py install for bakthat
Installing bakthat script to /usr/local/bin
Running setup.py install for grandfatherson
Running setup.py install for dumptruck
SyntaxError: ('invalid syntax', ('/usr/local/lib/python2.6/dist-packages/dumptruck/adapters_and_converters.py', 65, 24, ' d = {k: None for k in val}\n'))

SyntaxError: ('invalid syntax', ('/usr/local/lib/python2.6/dist-packages/dumptruck/convert.py', 99, 84, '    elif type(value) == dict and not set(map(type, value.keys())).issubset({unicode, str}):\n'))

SyntaxError: ('invalid syntax', ('/usr/local/lib/python2.6/dist-packages/dumptruck/dumptruck.py', 126, 35, "    column_names_expected = {'key', 'type', 'value'}\n"))

Running setup.py install for byteformat
Running setup.py install for pyyaml
checking if libyaml is compilable
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c build/temp.linux-x86_64-2.6/check_libyaml.c -o build/temp.linux-x86_64-2.6/check_libyaml.o
build/temp.linux-x86_64-2.6/check_libyaml.c:2:18: error: yaml.h: No such file or directory
build/temp.linux-x86_64-2.6/check_libyaml.c: In function ‘main’:
build/temp.linux-x86_64-2.6/check_libyaml.c:5: error: ‘yaml_parser_t’ undeclared (first use in this function)
build/temp.linux-x86_64-2.6/check_libyaml.c:5: error: (Each undeclared identifier is reported only once
build/temp.linux-x86_64-2.6/check_libyaml.c:5: error: for each function it appears in.)
build/temp.linux-x86_64-2.6/check_libyaml.c:5: error: expected ‘;’ before ‘parser’
build/temp.linux-x86_64-2.6/check_libyaml.c:6: error: ‘yaml_emitter_t’ undeclared (first use in this function)
build/temp.linux-x86_64-2.6/check_libyaml.c:6: error: expected ‘;’ before ‘emitter’
build/temp.linux-x86_64-2.6/check_libyaml.c:8: warning: implicit declaration of function ‘yaml_parser_initialize’
build/temp.linux-x86_64-2.6/check_libyaml.c:8: error: ‘parser’ undeclared (first use in this function)
build/temp.linux-x86_64-2.6/check_libyaml.c:9: warning: implicit declaration of function ‘yaml_parser_delete’
build/temp.linux-x86_64-2.6/check_libyaml.c:11: warning: implicit declaration of function ‘yaml_emitter_initialize’
build/temp.linux-x86_64-2.6/check_libyaml.c:11: error: ‘emitter’ undeclared (first use in this function)
build/temp.linux-x86_64-2.6/check_libyaml.c:12: warning: implicit declaration of function ‘yaml_emitter_delete’

libyaml is not found or a compiler error: forcing --without-libyaml
(if libyaml is installed correctly, you may need to
 specify the option --include-dirs or uncomment and
 modify the parameter include_dirs in setup.cfg)

Running setup.py install for sh
Successfully installed bakthat grandfatherson dumptruck byteformat pyyaml sh
Cleaning up...
root@dagobah:/home/xxxx# bakthat upgrade_to_dump_truck
usage: bakthat [-h]

           {info,restore,configure,show_glacier_inventory,backup_glacier_inventory,ls,restore_glacier_inventory,show_local_glacier_inventory,backup,delete}
           ...

bakthat: error: invalid choice: 'upgrade_to_dump_truck' (choose from 'info', 'restore', 'configure', 'show_glacier_inventory', 'backup_glacier_inventory', 'ls', 'restore_glacier_inventory', 'show_local_glacier_inventory', 'backup', 'delete')

configuration bakthat problem

How i can install as mysql?

This is what i do.

python setup.py install

running install
running bdist_egg
running egg_info
writing requirements to bakthat.egg-info/requires.txt
writing bakthat.egg-info/PKG-INFO
writing top-level names to bakthat.egg-info/top_level.txt
writing dependency_links to bakthat.egg-info/dependency_links.txt
writing entry points to bakthat.egg-info/entry_points.txt
reading manifest file 'bakthat.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'test'
writing manifest file 'bakthat.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/bakthat
copying build/lib/bakthat/models.py -> build/bdist.linux-x86_64/egg/bakthat
copying build/lib/bakthat/sync_wip.py -> build/bdist.linux-x86_64/egg/bakthat
copying build/lib/bakthat/utils.py -> build/bdist.linux-x86_64/egg/bakthat
copying build/lib/bakthat/backends.py -> build/bdist.linux-x86_64/egg/bakthat
copying build/lib/bakthat/helper.py -> build/bdist.linux-x86_64/egg/bakthat
copying build/lib/bakthat/sync.py -> build/bdist.linux-x86_64/egg/bakthat
copying build/lib/bakthat/plugin.py -> build/bdist.linux-x86_64/egg/bakthat
copying build/lib/bakthat/init.py -> build/bdist.linux-x86_64/egg/bakthat
copying build/lib/bakthat/conf.py -> build/bdist.linux-x86_64/egg/bakthat
byte-compiling build/bdist.linux-x86_64/egg/bakthat/models.py to models.pyc
byte-compiling build/bdist.linux-x86_64/egg/bakthat/sync_wip.py to sync_wip.pyc
byte-compiling build/bdist.linux-x86_64/egg/bakthat/utils.py to utils.pyc
byte-compiling build/bdist.linux-x86_64/egg/bakthat/backends.py to backends.pyc
byte-compiling build/bdist.linux-x86_64/egg/bakthat/helper.py to helper.pyc
byte-compiling build/bdist.linux-x86_64/egg/bakthat/sync.py to sync.pyc
byte-compiling build/bdist.linux-x86_64/egg/bakthat/plugin.py to plugin.pyc
byte-compiling build/bdist.linux-x86_64/egg/bakthat/init.py to init.pyc
byte-compiling build/bdist.linux-x86_64/egg/bakthat/conf.py to conf.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying bakthat.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bakthat.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bakthat.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bakthat.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bakthat.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bakthat.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bakthat.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
creating 'dist/bakthat-0.6.0-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing bakthat-0.6.0-py2.7.egg
removing '/usr/lib64/python2.7/site-packages/bakthat-0.6.0-py2.7.egg' (and everything under it)
creating /usr/lib64/python2.7/site-packages/bakthat-0.6.0-py2.7.egg
Extracting bakthat-0.6.0-py2.7.egg to /usr/lib64/python2.7/site-packages
bakthat 0.6.0 is already the active version in easy-install.pth
Installing bakthat script to /usr/bin

Installed /usr/lib64/python2.7/site-packages/bakthat-0.6.0-py2.7.egg
Processing dependencies for bakthat==0.6.0
Searching for Events==0.1.0
Best match: Events 0.1.0
Processing Events-0.1.0-py2.7.egg
Events 0.1.0 is already the active version in easy-install.pth

Using /usr/lib64/python2.7/site-packages/Events-0.1.0-py2.7.egg
Searching for requests==1.2.3
Best match: requests 1.2.3
Processing requests-1.2.3-py2.7.egg
requests 1.2.3 is already the active version in easy-install.pth

Using /usr/lib64/python2.7/site-packages/requests-1.2.3-py2.7.egg
Searching for sh==1.08
Best match: sh 1.08
Processing sh-1.08-py2.7.egg
sh 1.08 is already the active version in easy-install.pth

Using /usr/lib64/python2.7/site-packages/sh-1.08-py2.7.egg
Searching for PyYAML==3.10
Best match: PyYAML 3.10
Processing PyYAML-3.10-py2.7-linux-x86_64.egg
PyYAML 3.10 is already the active version in easy-install.pth

Using /usr/lib64/python2.7/site-packages/PyYAML-3.10-py2.7-linux-x86_64.egg
Searching for byteformat==0.2a
Best match: byteformat 0.2a
Processing byteformat-0.2a-py2.7.egg
byteformat 0.2a is already the active version in easy-install.pth

Using /usr/lib64/python2.7/site-packages/byteformat-0.2a-py2.7.egg
Searching for peewee==2.1.2
Best match: peewee 2.1.2
Processing peewee-2.1.2-py2.7.egg
peewee 2.1.2 is already the active version in easy-install.pth
Installing pwiz.py script to /usr/bin

Using /usr/lib64/python2.7/site-packages/peewee-2.1.2-py2.7.egg
Searching for GrandFatherSon==1.1
Best match: GrandFatherSon 1.1
Processing GrandFatherSon-1.1-py2.7.egg
GrandFatherSon 1.1 is already the active version in easy-install.pth

Using /usr/lib64/python2.7/site-packages/GrandFatherSon-1.1-py2.7.egg
Searching for beefish==0.1.1
Best match: beefish 0.1.1
Processing beefish-0.1.1-py2.7.egg
beefish 0.1.1 is already the active version in easy-install.pth
Installing beefish.py script to /usr/bin

Using /usr/lib64/python2.7/site-packages/beefish-0.1.1-py2.7.egg
Searching for pycrypto==2.6
Best match: pycrypto 2.6
Processing pycrypto-2.6-py2.7-linux-x86_64.egg
pycrypto 2.6 is already the active version in easy-install.pth

Using /usr/lib64/python2.7/site-packages/pycrypto-2.6-py2.7-linux-x86_64.egg
Searching for boto==2.9.6
Best match: boto 2.9.6
Processing boto-2.9.6-py2.7.egg
boto 2.9.6 is already the active version in easy-install.pth
Installing mturk script to /usr/bin
Installing cwutil script to /usr/bin
Installing cfadmin script to /usr/bin
Installing sdbadmin script to /usr/bin
Installing elbadmin script to /usr/bin
Installing cq script to /usr/bin
Installing bundle_image script to /usr/bin
Installing lss3 script to /usr/bin
Installing dynamodb_dump script to /usr/bin
Installing fetch_file script to /usr/bin
Installing asadmin script to /usr/bin
Installing list_instances script to /usr/bin
Installing launch_instance script to /usr/bin
Installing glacier script to /usr/bin
Installing kill_instance script to /usr/bin
Installing instance_events script to /usr/bin
Installing taskadmin script to /usr/bin
Installing s3put script to /usr/bin
Installing route53 script to /usr/bin
Installing dynamodb_load script to /usr/bin
Installing pyami_sendmail script to /usr/bin

Using /usr/lib64/python2.7/site-packages/boto-2.9.6-py2.7.egg
Searching for aaargh==0.4
Best match: aaargh 0.4
Processing aaargh-0.4-py2.7.egg
aaargh 0.4 is already the active version in easy-install.pth

Using /usr/lib64/python2.7/site-packages/aaargh-0.4-py2.7.egg
Searching for argparse==1.2.1
Best match: argparse 1.2.1
Processing argparse-1.2.1-py2.7.egg
argparse 1.2.1 is already the active version in easy-install.pth

Using /usr/lib64/python2.7/site-packages/argparse-1.2.1-py2.7.egg
Finished processing dependencies for bakthat==0.6.0

$ bakthat configure
Traceback (most recent call last):
File "/usr/bin/bakthat", line 9, in
load_entry_point('bakthat==0.6.0', 'console_scripts', 'bakthat')()
File "/usr/lib64/python2.7/site-packages/pkg_resources.py", line 346, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib64/python2.7/site-packages/pkg_resources.py", line 2516, in load_entry_point
return ep.load()
File "/usr/lib64/python2.7/site-packages/pkg_resources.py", line 2210, in load
entry = import(self.module_name, globals(),globals(), ['name'])
File "/usr/lib64/python2.7/site-packages/bakthat-0.6.0-py2.7.egg/bakthat/init.py", line 25, in
from bakthat.backends import GlacierBackend, S3Backend, RotationConfig, SwiftBackend
File "/usr/lib64/python2.7/site-packages/bakthat-0.6.0-py2.7.egg/bakthat/backends.py", line 17, in
from bakthat.models import Inventory, Jobs
File "/usr/lib64/python2.7/site-packages/bakthat-0.6.0-py2.7.egg/bakthat/models.py", line 1, in
import peewee
File "build/bdist.linux-x86_64/egg/peewee.py", line 98, in
peewee.ImproperlyConfigured: Either sqlite3, psycopg2 or MySQLdb must be installed

File "build/bdist.linux-x86_64/egg/peewee.py", line 98, in
peewee.ImproperlyConfigured: Either sqlite3, psycopg2 or MySQLdb must be installed <-- how mysql?

Cant install bakthat due to "Could not find a version that satisfies the requirement byteformat"

Hi there,

It seems bakthat is a great backup framework and would love to test it but getting it up and running is kind of a pain till now.

I ran pip install -U bakthat

Downloading/unpacking bakthat
  Downloading bakthat-0.6.0.tar.gz
  Running setup.py egg_info for package bakthat

    warning: no files found matching 'test'
Requirement already up-to-date: aaargh in /usr/local/lib/python2.7/dist-packages (from bakthat)
Requirement already up-to-date: boto in /usr/local/lib/python2.7/dist-packages (from bakthat)
Downloading/unpacking pycrypto (from bakthat)
  Downloading pycrypto-2.6.tar.gz (443kB): 443kB downloaded
  Running setup.py egg_info for package pycrypto

Downloading/unpacking beefish (from bakthat)
  Downloading beefish-0.1.1.tar.gz
  Running setup.py egg_info for package beefish

Downloading/unpacking grandfatherson (from bakthat)
  Downloading GrandFatherSon-1.2.tar.gz
  Running setup.py egg_info for package grandfatherson

Downloading/unpacking peewee (from bakthat)
  Downloading peewee-2.1.4.tar.gz (819kB): 819kB downloaded
  Running setup.py egg_info for package peewee

Downloading/unpacking byteformat (from bakthat)
  Could not find a version that satisfies the requirement byteformat (from bakthat) (from versions: 0.1a, 0.2a)
Cleaning up...
No distributions matching the version for byteformat (from bakthat)
Storing complete log in /root/.pip/pip.log

I followed #26 which got me further but now I am stuck here.

Any idea's?

trying to automate bakthat

So I'm trying to write a script (for cron) that will call bakthat. so the obvious issue that I am having is having to input the password. IS there a way to save the password to a config file somehow?

Thanks

Oh I tested bakthat out on a few files and it works great ... I'm waiting to use it on my office's backups.

How much control do we have over the GFS scheme

Hi, now that the GFS issue has been resolved, I find myself in a bit of a pickle, with rotate() deleting all my backups for the current day except for the first one.

So for example, if I backup 24 times a day on the hour, it seems to only keep the one that took place at 1am that day, and discards everything that came afterwards, INCLUDING the current backup :(

Would a better solution in my case be perhaps to somehow use delete_older_than , at least I'd keep my 48 backups for the last 2 days, and nevermind using GFS?

Is it possible to use delete_older_than with a tag? That way I could keep some backups for longer than others.

Cheers!

Ability to specify the location of the sqlite file

The system creates a .bakthat.sqlite in the users home directory.

It would be nice if the path could be specified in the config file or passed in as an arg.

Time in filename is wrong

Hi there, the time in the timestamp of the created backup file is always 2 hours behind the server time (and the time reported to me in python). What could that be?

Bakthat fails if compression is disabled and Directory is selected for backup.

Bakthat fails if compression is disabled and Directory is selected for backup.

There is no version argument for Bakthat but from the /usr/local/bin/bakthat file it looks like I am running 0.6. Nevertheless here is the output

$ bakthat backup vms/

Compression disabled
Traceback (most recent call last):
File "/usr/local/bin/bakthat", line 9, in
load_entry_point('bakthat==0.6.0', 'console_scripts', 'bakthat')()
File "/usr/local/lib/python2.7/dist-packages/bakthat/init.py", line 649, in main
app.run()
File "/usr/local/lib/python2.7/dist-packages/aaargh.py", line 157, in run
return func(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/bakthat/init.py", line 301, in backup
with open(outname) as outfile:

"--delete-source" arg?

The rotating backup stuff is great, however Glacier can also be treated as a cost-effective upload-once-download-maybe safety backup for some data. E.G. raw data whose derived products are of actual interest, but need to be preserved for reference in case something in the derived product(s) needs to be reevaluated down the road.

So, while I can add this to my own scripts, and/or use the API, it might be nice to be able to delete the source data, directly in bakthat's command-line, once its successfully been copied to S3 or Glacier. E.G. bakthat backup -d glacier --delete-source myfile. Some sort of stern warning could accompany bakthat backup -h.

By similar logic, since the data saved on Glacier may need to be download, manipulated, then re-uploaded -- and having multiple copies may be expensive/confusing -- it may be useful to delete-upon-restore, too. E.G. backthat restore -d glacier --delete-source myfile.20130401235446.tgz

Show does not show every destinations by default

The docs say this:

$ bakthat show -h
usage: bakthat show [-h] [-d DESTINATION] [-t TAGS] [-p PROFILE] [-c CONFIG]
                    [query]

positional arguments:
  query                 search filename for query

optional arguments:
  -h, --help            show this help message and exit
  -d DESTINATION, --destination DESTINATION
                        glacier|s3|swift, show every destination by default

But I see this:

$ bakthat show mybackup
$ bakthat show -d swift mybackup
2013-07-03T16:20:04 swift       5.0 MB      mybackup_qFfadH.20130703162004.tgz (mytag)

Not possible to specify custom conf file

According to

https://github.com/tsileo/bakthat/blob/master/bakthat/conf.py#L8

path to conf file is hard coded. Was it design decision?

Additional archiving/compression options

I use xz compression for most of my backups (I can afford the extra time and save myself a few bucks on storage). I guess other people might do the same with it or bz2.

So it would be very useful to have archiving/compression options in bakthat to choose .tar, .tbz2, .tar.xz...

Use stronger encryption instead of Blowfish

Bakthat currently encrypts using beefish, which is essentially a wrapper around pycrypto's Blowfish. Although not terribly broken, like DES, Blowfish is a suboptimal choice of cipher. The author of the cipher, Bruce Schneier, already said in 2007 he is surprised it is still being used at all.

Therefore, it would be good to at least support a stronger cipher, like AES-128. For backwards compatibility, Blowfish could be default, AES configurable. AES is already supported by pycrypto.

trying to configure bakthat

root@riker:/data# bakthat configure
Traceback (most recent call last):
File "/usr/local/bin/bakthat", line 9, in
load_entry_point('bakthat==0.5.0', 'console_scripts', 'bakthat')()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 337, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2279, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1989, in load
entry = import(self.module_name, globals(),globals(), ['name'])
File "/usr/local/bin/bakthat.py", line 5, in
pkg_resources.run_script('bakthat==0.2.1', 'bakthat.py')
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 499, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 686, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 588, in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (bakthat 0.5.0 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('bakthat==0.2.1'))

THanks

bakthat configure errors out

Hey there I was running 'bakthat configure' and got the following:

AWS Access Key:
AWS Secret Key:
S3 Bucket Name: test
Traceback (most recent call last):
File "/usr/local/bin/bakthat", line 9, in
load_entry_point('bakthat==0.3.0', 'console_scripts', 'bakthat')()
File "/usr/local/bin/bakthat.py", line 528, in main
app.run()
File "/usr/local/lib/python2.6/dist-packages/aaargh.py", line 157, in run
return func(**kwargs)
File "/usr/local/bin/bakthat.py", line 400, in configure
default_destination = raw_input("Default destination ({}): ".format(DEFAULT_DESTINATION))
ValueError: zero length field name in format

It seems to happen no matter what i enter for s3 bucket name (which ironically I am not using because I want to use this for glacier)

Missing support for Tenant Name in Swift

When attempting to use the Swift with Dreamobjects I get the following error:

Traceback (most recent call last):
  File "/usr/local/bin/bakthat", line 9, in <module>
    load_entry_point('bakthat==0.6.0', 'console_scripts', 'bakthat')()
  File "/usr/local/lib/python2.7/dist-packages/bakthat/__init__.py", line 649, in main
    app.run()
  File "/usr/local/lib/python2.7/dist-packages/aaargh/app.py", line 176, in run
    return func(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/bakthat/__init__.py", line 245, in backup
    storage_backend, destination, conf = _get_store_backend(config, destination, profile)
  File "/usr/local/lib/python2.7/dist-packages/bakthat/__init__.py", line 57, in _get_store_backend
    return STORAGE_BACKEND[destination](conf, profile), destination, conf
  File "/usr/local/lib/python2.7/dist-packages/bakthat/backends.py", line 311, in __init__
    self.con.put_container(self.conf["s3_bucket"])
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1182, in put_container
    response_dict=response_dict)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1103, in _retry
    self.url, self.token = self.get_auth()
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1079, in get_auth
    insecure=self.insecure)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 295, in get_auth
    raise ClientException('No tenant specified')
swiftclient.exceptions.ClientException: No tenant specified

This is also observed by another user here: http://thomassileo.com/blog/2013/04/24/bakthat-0-dot-5-0-released-with-openstack-swift-support-and-bakmanager-integration/

Bakthat Python API (MySQL)

I have a problem with this script. I have configurate this without a problem but when try to upload give an "Errno 104".

bakthat backup -p mysql -d glacier --prompt no
New current working directory: /tmp/myhost_mysql_Ljhg7N.
INFO:bakthat.helper:New current working directory: /tmp/myhost_mysql_Ljhg7N.
before_backup 5b61f799-238b-4351-8467-8a84cc35e52c
INFO:bakthat.plugin:before_backup 5b61f799-238b-4351-8467-8a84cc35e52c
before_backup 5b61f799-238b-4351-8467-8a84cc35e52c
INFO:bakthat.plugin:before_backup 5b61f799-238b-4351-8467-8a84cc35e52c
Backing up /tmp/myhost_mysql_Ljhg7N
INFO:bakthat:Backing up /tmp/myhost_mysql_Ljhg7N
Compressing...
INFO:bakthat:Compressing...
Encrypting...
INFO:bakthat:Encrypting...
Uploading...
INFO:bakthat:Uploading...
Upload completion: 0%
INFO:bakthat.backends:Upload completion: 0%
Upload completion: 11%
INFO:bakthat.backends:Upload completion: 11%
Upload completion: 22%
INFO:bakthat.backends:Upload completion: 22%
Upload completion: 0%
INFO:bakthat.backends:Upload completion: 0%
Upload completion: 11%
INFO:bakthat.backends:Upload completion: 11%
Upload completion: 0%
INFO:bakthat.backends:Upload completion: 0%
Upload completion: 0%
INFO:bakthat.backends:Upload completion: 0%
Upload completion: 11%
INFO:bakthat.backends:Upload completion: 11%
Upload completion: 22%
INFO:bakthat.backends:Upload completion: 22%
Upload completion: 33%
INFO:bakthat.backends:Upload completion: 33%
Upload completion: 0%
INFO:bakthat.backends:Upload completion: 0%
Upload completion: 11%
INFO:bakthat.backends:Upload completion: 11%
Upload completion: 0%
INFO:bakthat.backends:Upload completion: 0%
Upload completion: 0%
INFO:bakthat.backends:Upload completion: 0%
Upload completion: 11%
INFO:bakthat.backends:Upload completion: 11%
Back to /home/username
INFO:bakthat.helper:Back to /home/username
Traceback (most recent call last):
File "/usr/bin/bakthat", line 9, in
load_entry_point('bakthat==0.6.0', 'console_scripts', 'bakthat')()
File "/usr/lib64/python2.7/site-packages/bakthat/init.py", line 649, in main
app.run()
File "/usr/lib64/python2.7/site-packages/aaargh.py", line 157, in run
return func(*_kwargs)
File "/usr/lib64/python2.7/site-packages/bakthat/init.py", line 245, in backup
storage_backend, destination, conf = _get_store_backend(config, destination, profile)
File "/usr/lib64/python2.7/site-packages/bakthat/init.py", line 54, in _get_store_backend
setup_plugins(conf)
File "/usr/lib64/python2.7/site-packages/bakthat/plugin.py", line 27, in setup_plugins
p = load_class(plugin)
File "/usr/lib64/python2.7/site-packages/bakthat/plugin.py", line 41, in load_class
module = importlib.import_module(module_path)
File "/usr/lib64/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/home/username/.bakthat_plugins/mysql.py", line 17, in
bh.backup()
File "/usr/lib64/python2.7/site-packages/bakthat/helper.py", line 271, in backup
custom_filename=self.backup_name)
File "/usr/lib64/python2.7/site-packages/bakthat/init.py", line 362, in backup
storage_backend.upload(stored_filename, outname, s3_reduced_redundancy=s3_reduced_redundancy)
File "/usr/lib64/python2.7/site-packages/bakthat/backends.py", line 110, in upload
k.set_contents_from_filename(filename, *_upload_kwargs)
File "/usr/lib64/python2.7/site-packages/boto/s3/key.py", line 1280, in set_contents_from_filename
encrypt_key=encrypt_key)
File "/usr/lib64/python2.7/site-packages/boto/s3/key.py", line 1211, in set_contents_from_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/lib64/python2.7/site-packages/boto/s3/key.py", line 711, in send_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/lib64/python2.7/site-packages/boto/s3/key.py", line 879, in _send_file_internal
query_args=query_args
File "/usr/lib64/python2.7/site-packages/boto/s3/connection.py", line 547, in make_request
retry_handler=retry_handler
File "/usr/lib64/python2.7/site-packages/boto/connection.py", line 947, in make_request
retry_handler=retry_handler)
File "/usr/lib64/python2.7/site-packages/boto/connection.py", line 908, in _mexe
raise e
socket.error: [Errno 104] Connection reset by peer

i have to point out that i can send individual files perfectly fine, but when i use the script it doesn't work i can't upload anything because of the error i mentioned previously

missing python.h

FYI, I couldn't install with: pip install bakthat it failed with a gcc error on missing python.h

after installing python dev (apt-get install python-dev), it continued the install but gcc bailed looking for yaml.h

osx 10.7.5 supported?

running python 2.6.8 on 10.7.5... installs fine but get an error when trying to run configure.

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/dumptruck-0.1.4-py2.6.egg/dumptruck/dumptruck.py", line 126
column_names_expected = {'key', 'type', 'value'}

GlusterFS backend

Hello,
I've been working on adding a GlusterFS backend to bakthat and was wondering if you would have interest in accepting a pull request.

Thanks,

Thiago

Support for DreamObjects

DreamObjects supports the S3 API but it appears that bakthat is hardcoded for Amazon S3. Is it possible to configure bakthat to point toward DreamObjects instead?

bh.rotate() doesn't seem to delete anything on s3

Hi, got a little issue with rotate():

with BakHelper("mongobak",
           destination=DESTINATION,
           password=PASSWORD,
           tags=["mongodb"]) as bh:
logging.info(sh.mongodump(**MONGODUMP_PARAMS))

bh.backup()
bh.rotate()

The backup work, upload to s3 encrypted and tarred as expected, however rotate() never rotates them. As per the conf file below I was expecting 2 full days of backup and one copy kept a month.

Conf file info:

  [xxxx@xxx xxx_scripts]$ sudo cat /xxxxxx/.bakthat.yml
  default:
    access_key: xxxxxxxxxxx
    default_destination: s3
    glacier_vault: xxxxxxxx-ice
    region_name: us-east-1
    rotation:
      days: 2
      first_week_day: 0
      months: 12
      weeks: 4
    s3_bucket: xxxx-backups-xxx
    secret_key: xxxxxxxxxxxxxxxxxxxxxx

Note: my backup script runs several backups a day, ie once an hour.

KeyError: 'default_destination'

after installing bakthat and running configure i get the following on some commands (ls)

$ bakthat ls
Traceback (most recent call last):
File "/usr/local/bin/bakthat", line 9, in
load_entry_point('bakthat==0.4.5', 'console_scripts', 'bakthat')()
File "/usr/local/lib/python2.7/dist-packages/bakthat/init.py", line 690, in main
app.run()
File "/usr/local/lib/python2.7/dist-packages/aaargh.py", line 157, in run
return func(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/bakthat/init.py", line 580, in ls
storage_backend = _get_store_backend(conf, destination, profile)
File "/usr/local/lib/python2.7/dist-packages/bakthat/init.py", line 44, in _get_store_backend
return STORAGE_BACKEND[destination](conf, profile)
KeyError: 'default_destination'

the contents of my ~/.bakthat.yml are:

default:
access_key: [removed]
default_destination: glacier
glacier_vault: test
region_name: ap-northeast-1
s3_bucket: test
secret_key: [removed]

note: backing up seems to work OK

Support for prefixing a key name in S3 backend

Would be great to add support for a prefix to add to uploading files to s3. This would allow organizing backups within an existing bucket nested in directories (in S3 it's just a key).

Could add a configuration variable called prefix and check for a prefix any time you generate a key name (upload, download). How would this effect the restore behavior? Seems like changing upload/download/ls in backends would cover it. Would be happy to add work on this for S3.

Configuration should not be world readable

After running bakthat configure, I see that ~/.bakthat.yml is configured with permissions 644. I would expect 600 given that it contains secrets.

Incremental backups?

Does bakthat support incremental backups (e.g. duplicity)? Sorry if I missed this in the docs.

Specify multiple files to be uploaded

The system right now only supports uploading a single file at a time or a directory.

Are you planning on allowing users to upload multiple files at a time eg:
bakthat -d glacier --prompt no mylog-2013-05-10.log.gz mylog-2013-05-11.gz

bakthat takes 100% cpu

I ran bakthat to backup about 2TB of data to glacier. unfortunately my machine
$uname -a
Linux riker 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
ops@riker:$ cat /proc/meminfo
MemTotal: 16177716 kB
MemFree: 6782320 kB
Buffers: 4694440 kB
Cached: 3941100 kB
SwapCached: 3280 kB
Active: 4695988 kB
Inactive: 3963408 kB
Active(anon): 452 kB
Inactive(anon): 23432 kB
Active(file): 4695536 kB
Inactive(file): 3939976 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 16510972 kB
SwapFree: 16469940 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 22620 kB
Mapped: 7520 kB
Shmem: 28 kB
Slab: 503792 kB
SReclaimable: 483756 kB
SUnreclaim: 20036 kB
KernelStack: 1152 kB
PageTables: 3296 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 24599828 kB
Committed_AS: 163364 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 321772 kB
VmallocChunk: 34359409396 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 476736 kB
DirectMap2M: 13940736 kB
DirectMap1G: 2097152 kB

ops@riker:~$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 4
model name : AMD Phenom(tm) II X4 965 Processor
stepping : 3
microcode : 0x10000b6
cpu MHz : 800.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 5

not quite sure why or how to stop bakthat from eating 100% cpu please help...

not deleting .tgz files

Using CentOS 6, python 2.6. Installed like so:

$ sudo yum groupinstall "Development Tools"
$ sudo yum install python-pip python-devel PyYAML
$ sudo pip-python install bakthat boto requests

$ uname -a
Linux ***** 2.6.32-279.1.1.el6.x86_64 #1 SMP Tue Jul 10 13:47:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Python info:

Name : python
Arch : x86_64
Version : 2.6.6
Release : 36.el6
Size : 20 M
Repo : installed
From repo : base

Set $TMP per the docs, e.g. export TMP=/scr/day.
... but when I upload to Glacier, my temporary file is not purged once the transfer is complete. I aim to move ~500 archives, each 4-12GB. So, my staging disk will fill up if I do this programmatically. Workaround is to blindly delete $TMP/tmp* in my loop.

E.G. of my issue.

$ ls -l $TMP/tmp*

$ bakthat backup -p TA -t test -d glacier --prompt no /root/baler14.log
Backing up /root/baler14.log
Compressing...
Uploading...

$ echo $?
0

$ ls -l $TMP/tmp*
-rw-------. 1 user group 276 Apr 1 16:43 tmpjEV6ZM

BakHelper and svndump

svn repository

I try to use BakHelper with svn repository.
The process works fine(i.e in the log), but the command bakthat show displays:
2013-11-14T15:16:27 glacier 160 B SVN_HERACLES_wYUoBA.20131114151627.tgz.enc (svn)

The repository is around 4GB..

with BakHelper(SVN_HERACLES_BACKUP_NAME, password=HERACLES_BACKUP_PWD,destination="glacier", tags=["svn"]) as bhsvn:
logging.info(sh.svnadmin("dump",
q="/mnt/data/svnrepo/",
_out="svnrepo.dmp"))

    bhsvn.backup()
    bhsvn.rotate()

Error running bakthat on freebsd9.1

Hi, first of all I wanted to thak you for your great job on bakthat.
I think I found a bug on a default install on FreeBSD, after installing and testing that everything worked on a Fedora box, I installed on FreeBSD 9.1 using pip install bakthat and i get this error:

Traceback (most recent call last):
File "/usr/local/bin/bakthat", line 8, in
load_entry_point('bakthat==0.4.5', 'console_scripts', 'bakthat')()
File "/usr/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 318, in load_entry_point
File "/usr/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 2221, in load_entry_point
File "/usr/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1954, in load
File "/usr/local/lib/python2.7/site-packages/bakthat/init.py", line 23, in
from bakthat.backends import GlacierBackend, S3Backend, RotationConfig
File "/usr/local/lib/python2.7/site-packages/bakthat/backends.py", line 16, in
from bakthat.models import Inventory, Jobs
File "/usr/local/lib/python2.7/site-packages/bakthat/models.py", line 1, in
import peewee
File "/usr/local/lib/python2.7/site-packages/peewee.py", line 97, in
raise ImproperlyConfigured('Either sqlite3, psycopg2 or MySQLdb must be installed')
peewee.ImproperlyConfigured: Either sqlite3, psycopg2 or MySQLdb must be installed

Thanks

Problems with api

https://github.com/alexsilva/smartbackup

The link above is the repo I work, implementing some plugins.

The api plugin was working fine until I get to work with a local backend.
https://github.com/alexsilva/smartbackup/blob/master/backuptools/backends.py#L98

This part of the model "Backups", forces me to reset the entire function!

if kwargs.get("config"):
   conf = load_config(kwargs.get("config"))

    profile = conf.get(kwargs.get("profile", "default"))

     s3_key = hashlib.sha512(profile.get("access_key") +
                                profile.get("s3_bucket")).hexdigest()
     glacier_key = hashlib.sha512(profile.get("access_key") +
                                 profile.get("glacier_vault")).hexdigest()

All this so I can use the api functions already set in lib "bakthat".
https://github.com/alexsilva/smartbackup/blob/master/backuptools/models.py#L22

requirement.txt specify the version for each package

It is recommended to specify the version for each package in requirement.txt to prevent conflict in packages and its dependencies and to be sure it is working in this version.

Another issue with python 2.6

Get this when actually doing a glacier backup:

013-02-07 13:36:12,158 INFO: Backing up /xxx/xxxx/
Password (blank to disable encryption):
Password confirmation:
2013-02-07 13:36:15,826 INFO: Compressing...
Traceback (most recent call last):
File "/usr/local/bin/bakthat", line 9, in
load_entry_point('bakthat==0.3.1', 'console_scripts', 'bakthat')()
File "/usr/local/bin/bakthat.py", line 528, in main
app.run()
File "/usr/local/lib/python2.6/dist-packages/aaargh.py", line 157, in run
return func(**kwargs)
File "/usr/local/bin/bakthat.py", line 357, in backup
with tarfile.open(fileobj=out, mode="w:gz") as tar:
AttributeError: 'TarFile' object has no attribute 'exit'

Looks like another 2.6 issue http://stackoverflow.com/questions/6086603/statement-with-and-tarfile

ImportError: no module named glacier.exceptions

I'm trying to install and configure Bakthat and am receiving the following error:

ubuntu@ip-10-168-78-253:~$ bakthat configure
Traceback (most recent call last):
  File "/usr/local/bin/bakthat", line 9, in <module>
    load_entry_point('bakthat==0.6.0', 'console_scripts', 'bakthat')()
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 346, in load_entry_point

  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2516, in load_entry_point

  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2210, in load
    for ext in extras:
  File "/usr/local/lib/python2.7/dist-packages/bakthat-0.6.0-py2.7.egg/bakthat/__init__.py", line 25, in <module>
    from bakthat.backends import GlacierBackend, S3Backend, RotationConfig, SwiftBackend
  File "/usr/local/lib/python2.7/dist-packages/bakthat-0.6.0-py2.7.egg/bakthat/backends.py", line 13, in <module>
    from boto.glacier.exceptions import UnexpectedHTTPResponseError
ImportError: No module named glacier.exceptions

System is Ubuntu 12.04 (on EC2), running Python 2.7.3. I just did a fresh install of setuptools following these instructions (https://pypi.python.org/pypi/setuptools/0.7.2#unix-based-systems-including-mac-os-x), and pip following these instructions (http://www.pip-installer.org/en/latest/installing.html#using-get-pip).

Any ideas on what I can do to get this working?

Helper and conf passing confusion

Since you mentioned in in #43

don't hesitate if you have any suggestions!

I found BakHelper to be difficult to work with. I started to dig into the code, but eventually gave up. I wanted to create a helper script and have it read a config from a different file path, but ran into several stumbling blocks, including:

• Some methods take a conf as a file name, others take a conf with profiles as top-level keys, and others take a conf for an individual profile. This was confusing and some standardization would help.
• Incorrect defaults were set in the helper and then used to override the conf. For example, I wanted to use swift which was in my default profile, but if I didn't specify it directly in BakHelper, it would get set to DEFAULT_DESTINATION.

It's likely that I'm missing something here that would make this all easier, but that was my experience as a first time user.

Support `snet` for swiftclient

Rackspace gives you free and fast bandwidth to cloudfiles over their internal network. This can be passed in when instantiating the Connection class as the kwarg, snet. More details here, http://docs.openstack.org/developer/python-swiftclient/swiftclient.html#swiftclient.client.get_auth

Bakthat 060 not Python <2.7 compatible?

Upgraded, getting 'ImportError: No module named importlib' on any command.

pip install importlib fixes it.