Giter Site home page Giter Site logo

sinnwerkstatt / runrestic Goto Github PK

View Code? Open in Web Editor NEW
113.0 6.0 30.0 456 KB

A wrapper script for Restic backup software that inits, creates, prunes and checks backups

License: GNU General Public License v3.0

Python 98.38% Dockerfile 1.62%
restic restic-backups automation prometheus-metrics backups

runrestic's Introduction

python version Code style: black Travis (.com) PyPI Stackshare: runrestic PyPI - Downloads

Runrestic

runrestic is a simple Python wrapper script for the Restic backup software that initiates a backup, prunes any old backups according to a retention policy, and validates backups for consistency. The script supports specifying your settings in a declarative configuration file rather than having to put them all on the command-line, and handles common errors.

Example config

repositories = [
    "/tmp/restic-repo",
    "sftp:user@host:/srv/restic-repo",
    "s3:s3.amazonaws.com/bucket_name"
    ]

[environment]
RESTIC_PASSWORD = "CHANGEME"

[backup]
sources = [
    "/home",
    "/var"
    ]

[prune]
keep-last =  3
keep-hourly =  5

Alternatively you can also just use JSON. For a more comprehensive example see the example.toml or check the schema.json

Getting started

Installing runrestic and restic

To install runrestic, run the following command to download and install it:

sudo pip3 install --upgrade runrestic

You can either manually download and install [Restic](https://restic.net/) or you can just run `runrestic` and it'll try to download it for you.

Initializing and running

Once you have restic and runrestic ready, you should put a config file in on of the scanned locations, namely:

  • /etc/runrestic.toml
  • /etc/runrestic/example.toml
  • ~/.config/runrestic/example.toml
  • /etc/runrestic.json
  • /etc/runrestic/example.json
  • ~/.config/runrestic/example.json

Afterwards, run

runrestic init # to initialize all the repos in `repositories`

runrestic  # without actions will do: runrestic backup prune check
# or
runrestic [action]

Certain `restic` flags like `--dry-run/-n` are built into `runrestic` as well and will be passed to restic where applicable.

If, however, you need to pass along arbitrary other flags you can now add them to the end of your runrestic call like so:

runrestic backup -- --one-file-system

Logs for restic and hooks

The output of restic and the configured pre/post-hooks is added to the runrestic logs at the level defined in [execution] proc_log_level (default: DEBUG), which can be overwritten with the CLI option -p/--proc-log-level.

For process log levels greater than INFO the output of file names is suppressed and for log levels greater than WARNING restic is executed with the --quiet option. If the process log level is set to DEBUG, then restic is executed with the --verbose option.

It is also possible to add restic progress messages to the logs by using the CLI option --show-progress INTERVAL where the INTERVAL is the number of seconds between the progress messages.

Restic shell

To use the options defined in runrestic with restic (e.g. for a backup restore), you can use the shell action:

runrestic shell

If you are using multiple repositories or configurations, you can select one now.

Prometheus / Grafana metrics

@d-matt created a nice dashboard for Grafana here: https://grafana.com/grafana/dashboards/11064/revisions

systemd timer or cron

If you want to run runrestic automatically, say once a day, the you can configure a job runner to invoke it periodically.

systemd

If you're using systemd instead of cron to run jobs, download the sample systemd service file and the sample systemd timer file. Then, from the directory where you downloaded them:

sudo mv runrestic.service runrestic.timer /etc/systemd/system/
sudo systemctl enable runrestic.timer
sudo systemctl start runrestic.timer

cron

If you're using cron, download the sample cron file. Then, from the directory where you downloaded it:

sudo mv runrestic /etc/cron.d/runrestic
sudo chmod +x /etc/cron.d/runrestic

Changelog

  • v0.5.28

    • Allow jsonschema >= 4.0
  • v0.5.27

    • Fix output parsing for new restic version 0.14.0
    • Introduce failsafe output parser which supports default values
  • v0.5.26

    • Add output messages from restic and pre/post-hook commands to runrestic logs.
    • New CLI argument --show-progress INTERVAL for the restic progress update interval in seconds (default None)
  • v0.5.25

    • Drop support for Python 3.6, add support for Python 3.9 and 3.10, update dependencies
  • v0.5.24

    • Exit the script with returncode = 1 if there was an error in any of the tasks
  • v0.5.23

    • support JSON config files.
  • v0.5.21

    • fix issue where "check" does not count towards overall "errors"-metric
  • v0.5! Expect breaking changes.

    • metrics output is a bit different
    • see new parallel and retry_* options.

Ansible

@tabic wrote an ansible role, you can find it here: https://github.com/outwire/ansible-role-restic . (I have neither checked nor tested it.)

Development

This project is managed with poetry

Install it if not already present:

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
# or
pip install --user poetry

Installing dependencies

poetry install

Running Tests

poetry run pytest

Using VScode devcontainer

The project contains a .devcontainer folder with the settings for VScode to develop inside container. The Python virtual environment created by poetry is stored outside the container in the projects path .virtualenvs so that it survives container rebuilds.

The Ubuntu 22.04 based container uses Python 3.10 as system version and includes minimal Python 3.7, 3.8 and 3.9 versions for creating virtual environments in any of those versions.

It is possible to switch the Python version used by poetry with the command poetry use <version>, see poetry managing environments for more details.

Thanks

This project was initially based on borgmatic but has since evolved into something else.

runrestic's People

Contributors

andreasnuesslein avatar d-matt avatar dev-zero avatar fdw avatar gdhordain avatar major avatar mborgelt avatar opie4624 avatar palto42 avatar saimonn avatar tabic avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

runrestic's Issues

Runrestic on Windows

Hi!
After a few test I can conclude that runrestic is not runnable on Windows.

Is it possible to make it work on it?
It will be possible on the future? Is it planed?

Thank you in advance

Local vs Remote Repos

I've been using Runrestic for the incredibly long time of almost 72hrs now. (Quite liking it!) I do quite a lot of work without a connection, so restic + runrestic has been me slowly building up a time-machine like backup strategy. Time Machine backs up locally to a hidden partition, then copies over backups to the network storage when you're back within the network. With this background in mind, does it make sense to try to expand runrestic to have the concept of "backup" repos vs "copy-to" repos? prune and check would still need to be done against these copy-to repos, but backing up would always happen to a local destination first, with a second phase of copying out to these other repos if reachable, next.

Support Python 3.10 - EOL of Python 3.6

Hi @andreasnuesslein,
What do you think of a general version update for this module since Python 3.6 is end of life already and latest Python is meanwhile 3.10?

The main changes in my view would be:

  • Change min supported version to 3.8 (or 3.7?)
  • Update the dependencies (poetry.lock)
  • Change versions tested in Travis to 3.8, 3.9 and 3.10

I'm not expecting any direct impact on the code, it would be mainly to catch-up with Python release cycle.
Since my main laptop uses Python 3.10 I've already using an update version of runrestic, including updated dependencies via Poetry.

Broken prune output parsing with Restic 0.13.0

Just updated to Restic 0.13.0 and faced a traceback in the prune stage, which is likely caused by changed output

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/runrestic/restic/runner.py", line 204, in prune
    process_infos
  File "/usr/local/lib/python3.7/dist-packages/runrestic/restic/output_parsing.py", line 64, in parse_prune
    )[0]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/runrestic", line 10, in <module>
    sys.exit(runrestic())
  File "/usr/local/lib/python3.7/dist-packages/runrestic/runrestic/runrestic.py", line 75, in runrestic
    runner.run()
  File "/usr/local/lib/python3.7/dist-packages/runrestic/restic/runner.py", line 57, in run
    self.prune()
  File "/usr/local/lib/python3.7/dist-packages/runrestic/restic/runner.py", line 210, in prune
    ] = parse_new_prune(process_infos)
  File "/usr/local/lib/python3.7/dist-packages/runrestic/restic/output_parsing.py", line 105, in parse_new_prune
    )[0]
IndexError: list index out of range

add RESTIC_PASSWORD_COMMAND

Hello!

First of all, thank you for this program.

I'm using keyring to store password for restic repo https://pypi.org/project/keyring/

in restic, it's used as RESTIC_PASSWORD_COMMAND env var

but the problem is, in runrestic json schema command is not defined as a valid value

https://github.com/sinnwerkstatt/runrestic/blob/6c10fedc1814e760e8ec41c5d167463d3f9a16e1/runrestic/runrestic/schema.json#L46C1-L49C8

so, I'm forced to specify at least this RESTIC_PASSWORD_FILE=""

runrestic fails to parse valid `keep-within` values

runrestic -n
Problem parsing /etc/runrestic.toml: invalid literal for int() with base 0: '48h' (line 19 column 1 char 360)

restic assumes the parameter keep-within should be integer, which is not the case.

It's a duration, like 3w2d8h

--keep-within duration keep all snapshots which have been made within the duration of the latest snapshot. duration needs to be a number of years, months, days, and hours, e.g. 2y5m7d3h will keep all snapshots made in the two years, five months, seven days, and three hours before the latest snapshot.

drop fastjsonschema for jsonschema

I'm not a Python pro, but it seems to be feasible and simple to drop fastjsonschema for jsonschema to simplify packaging (jsonschema is mostly already available as dist package).

AttributeError: 'ResticRepository' object has no attribute 'log'

hi all, everything working fine, just getting this error after backup:

Traceback (most recent call last):
  File "/usr/local/bin/runrestic", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/runrestic/commands/runrestic.py", line 127, in main
    run_configuration(config, args)
  File "/usr/local/lib/python3.7/site-packages/runrestic/commands/runrestic.py", line 96, in run_configuration
    logs['repositories'][repository] = repo.log
AttributeError: 'ResticRepository' object has no attribute 'log'

any ideas on that? Thanks

Support (arbitrary) restic options

There doesn't seem to be a way to use arbitrary restic options, like --one-file-system or --no-lock, especially the former is very useful to me.

I suggest to either add a list, like CMD_ARGS to the config file, or add support the specific options like ONE-FILE-SYSTEM=True.

Support of S3 backend

Hi 👋,

I'm trying to setup runrestic to push my backup in S3.
By reading the sample config file, it's not clear to me in you support all environments variables or only RESTIC_PASSWORD and RESTIC_PASSWORD_FILE` ?

The schema.json file into your code let me think it's not supported by I prefer to ask.
If it's not supported, do you plan to do so ? 

Somewhere the return code is still called "rc"

Hey @palto42 ,

I think you missed a rc somewhere - do you have time to check real quick?

Aug 16 02:51:34 bkp1 runrestic[1530969]:       "total_file_count": 250361,
Aug 16 02:51:34 bkp1 runrestic[1530969]:       "total_size_bytes": 1324666550,
Aug 16 02:51:34 bkp1 runrestic[1530969]:       "duration_seconds": 7.828546524047852,
Aug 16 02:51:34 bkp1 runrestic[1530969]:       "rc": 0
Aug 16 02:51:34 bkp1 runrestic[1530969]:     }
Aug 16 02:51:34 bkp1 runrestic[1530969]:   },
Aug 16 02:51:34 bkp1 runrestic[1530969]:   "last_run": 1660611094.014,
Aug 16 02:51:34 bkp1 runrestic[1530969]:   "total_duration_seconds": 56.1098952293396
Aug 16 02:51:34 bkp1 runrestic[1530969]: }
Aug 16 02:51:34 bkp1 runrestic[1530969]: Traceback (most recent call last):
Aug 16 02:51:34 bkp1 runrestic[1530969]:   File "/usr/local/bin/runrestic", line 8, in <module>
Aug 16 02:51:34 bkp1 runrestic[1530969]:     sys.exit(runrestic())
Aug 16 02:51:34 bkp1 runrestic[1530969]:   File "/usr/local/lib/python3.10/dist-packages/runrestic/runrestic/runrestic.py", line 81, in runrestic
Aug 16 02:51:34 bkp1 runrestic[1530969]:     result.append(runner.run())
Aug 16 02:51:34 bkp1 runrestic[1530969]:   File "/usr/local/lib/python3.10/dist-packages/runrestic/restic/runner.py", line 73, in run
Aug 16 02:51:34 bkp1 runrestic[1530969]:     write_metrics(self.metrics, self.config)
Aug 16 02:51:34 bkp1 runrestic[1530969]:   File "/usr/local/lib/python3.10/dist-packages/runrestic/metrics/__init__.py", line 12, in write_metrics
Aug 16 02:51:34 bkp1 runrestic[1530969]:     file.writelines("".join(lines))
Aug 16 02:51:34 bkp1 runrestic[1530969]:   File "/usr/local/lib/python3.10/dist-packages/runrestic/metrics/prometheus.py", line 205, in generate_lines
Aug 16 02:51:34 bkp1 runrestic[1530969]:     yield backup_metrics(metrics["backup"], name)
Aug 16 02:51:34 bkp1 runrestic[1530969]:   File "/usr/local/lib/python3.10/dist-packages/runrestic/metrics/prometheus.py", line 222, in backup_metrics
Aug 16 02:51:34 bkp1 runrestic[1530969]:     retval += _restic_pre_hooks.format(name=name, **mtrx)
Aug 16 02:51:34 bkp1 runrestic[1530969]: KeyError: 'rc'

cheers

Partial duplication of Backupninja?

Some of the core features of this project appear to be duplicating Backupninja. We are very close to adding support for Restic in the merge request Add restic support.

Would it be possible to:

  • Test Backupninja with Restic support, and help move that initiative forward, and also
  • Keep this module for Restic-specific things that can't logically be added to Backupninja?

Bump version on PyPi

Hi,
The latest version on PyPi is still 0.5.23 and doesn't include fix #54 for restic >= 0.13.0.
Can it be bumped?

Broken backup output parsing with Restic 0.14.0

I guess Restic output parsing broken again after Restic update to 0.14.0. In this case after backup operation.

Before something similar hapened with prune as #51

This happens with Restic 0.14.0 and RunRestic 0.5.26.1 from pip.

$ restic version
restic 0.14.0 compiled with go1.19 on linux/arm64
$ runrestic -v
runrestic
0.5.26.1
Traceback (most recent call last):
  File "/usr/local/bin/runrestic", line 8, in <module>
    sys.exit(runrestic())
  File "/usr/local/lib/python3.8/dist-packages/runrestic/runrestic/runrestic.py", line 81, in runrestic
    result.append(runner.run())
  File "/usr/local/lib/python3.8/dist-packages/runrestic/restic/runner.py", line 56, in run
    self.backup()
  File "/usr/local/lib/python3.8/dist-packages/runrestic/restic/runner.py", line 143, in backup
    metrics[redact_password(repo, self.pw_replacement)] = parse_backup(
  File "/usr/local/lib/python3.8/dist-packages/runrestic/restic/output_parsing.py", line 19, in parse_backup
    added_to_the_repo = re.findall(
IndexError: list index out of range

Of course, this is fixed by downgrading Restic to 0.13.1: https://github.com/restic/restic/releases/tag/v0.13.1

So not an urgent issue

runrestic exit with status code 0 when parsing fails

This makes, for example, systemd thinks everything is okay, when it is not.

# systemctl  status runrestic
● runrestic.service - runrestic backup
bash-4.2# systemctl status runrestic.service
● runrestic.service - runrestic backup
   Loaded: loaded (/etc/systemd/system/runrestic.service; static; vendor preset: disabled)
   Active: inactive (dead) since Tue 2020-03-10 12:42:04 CET; 2h 5min ago
  Process: 17443 ExecStart=/usr/local/bin/runrestic (code=exited, status=0/SUCCESS)
 Main PID: 17443 (code=exited, status=0/SUCCESS)

Mar 10 12:42:04 myhostname systemd[1]: Starting runrestic backup...
Mar 10 12:42:04 myhostname runrestic[17443]: Problem parsing /etc/runrestic.toml: invalid literal for int() with base 0: '4...r 360)
Mar 10 12:42:04 myhostname systemd[1]: Started runrestic backup.
Hint: Some lines were ellipsized, use -l to show in full.

Cache handling

Optimized cache management is important to keep cloud storage costs under control.

I would like to use --cache-dir for all operations. Having this built-in makes sure I don't forget it for any command.

The other options like --with-cache/--no-cache can be specified via -- --option if you don't want to add official support.


Furthermore, behavior different from restic default should be documented:

if os.geteuid() == 0: # pragma: no cover; if user is root, we just use system cache
os.environ["XDG_CACHE_HOME"] = "/var/cache"
elif not (os.environ.get("HOME") or os.environ.get("XDG_CACHE_HOME")):
os.environ["XDG_CACHE_HOME"] = "/var/cache"

jsonschema 3.2.0 does not satisfy "jsonschema==3.0.*,>=3.0.0"

Shouldn't a jsonschema version of 3.2.0 satisfy the >=3.0.0 portion of this from requirements.txt?

community/python-jsonschema 3.2.0-1 (93.4 KiB 582.6 KiB) (Installed)
    An implementation of JSON Schema validation for Python
0 ✓ fryfrog@apollo ~/aur/runrestic $[master] runrestic
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (jsonschema 3.2.0 (/usr/lib/python3.8/site-packages), Requirement.parse('jsonschema==3.0.*,>=3.0.0'), {'runrestic'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/runrestic", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3252, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jsonschema==3.0.*,>=3.0.0' distribution was not found and is required by runrestic

Expose logging to user

I see you use logging library already. It would be nice, if the user could add additional logging.Formatter or logging.Handler (e.g. FileHandler with custom path and level, SMTPHandler).

Config file could consist of

  • import (or can this be determined based from handlerType automatically?)
  • handlerType, handlerArgs for constructor
  • level
  • format string

runrestic hangs when restic unexpectedly exits

# cat /etc/runrestic/home.toml
name = "home directories"
repositories = [ "rclone:test",]

[execution]
parallel = true
retry_count = 10
retry_backoff = "1:00 exponential"

[environment]
RESTIC_PASSWORD_FILE = "/etc/secrets/restic.password"
RCLONE_CONFIG = "/etc/secrets/rclone.conf"

[backup]
sources = [ "/home",]
exclude_patterns = [ "/home/_sysupgrade",]

[prune]
keep-last = 2
group-by = "host,paths"

[check]
checks = [ "check-unused", "read-data",]

[metrics.prometheus]
path = "/tmp/runrestic_home.prom"
# time /usr/local/virtualenvs/runrestic/bin/runrestic -c /etc/runrestic/home.toml -l debug -- --tag home
Parsing configuration file: /etc/runrestic/home.toml
[Environment] RESTIC_PASSWORD_FILE=/etc/secrets/restic.password
[Environment] RCLONE_CONFIG=/etc/secrets/rclone.conf
Spawning "['restic', '-r', 'rclone:test', 'backup', '--tag', 'home', '--exclude', '/home/_sysupgrade', '/home']"
Spawning "['restic', '-r', 'rclone:test', 'forget', '--tag', 'home', '--keep-last', '2', '--group-by', 'host,paths']"
Spawning "['restic', '-r', 'rclone:test', 'prune', '--tag', 'home']"

and just hangs there until I ^C out of it. Meanwhile

# RESTIC_PASSWORD_FILE=/etc/secrets/restic.password RCLONE_CONFIG=/etc/secrets/rclone.conf restic -r rclone:test prune --tag home ; echo $?
unknown flag: --tag
1

Support Docker

Hi,
Thank you for the wonderful work.
I am trying to run this in docker container. But unable to do so.

Can anyone can help please?

Recursion error

I'm getting a lot of similar errors in different circumstances. They all look more or less like this:

Exception ignored in: <Finalize object, dead>
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/util.py", line 201, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 726, in _terminate_pool
    p.join()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 47, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 1 more time]
... snip ...
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 31, in kill_the_group
    def kill_the_group(signal_number: signal.Signals, frame: Any) -> None:
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 32, in kill_the_group
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 15 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

Importantly, I have set

[execution]
parallel = false

I'm running 0.5.2 from the Arch AUR. What else do you need?

Do you have any idea what the problem could be?

Github release for 0.5.4?

Hi, I see you guys have released a new version on pypi, could you also do a release here? I use it for my Arch Linux AUR package because it has your systemd sample files in it. :)

Thanks!

Fix the setup.py in pypi

The setup.py that comes out of your pypi package uses a handful of setuptools things, but has from distutils.core import setup which means they don't work. Maybe something about poetry change could fix that?

Error with large number of files

Hello

I have a Ubuntu 14.04.5 LTS system with a large number of files, doing a backup to OVH swift open stack, the backup ends well but when runrestic is computing the files added to the repo, runrestic crashes, this is the output of the process:

Files:           0 new,    10 changed, 421764 unmodified
Dirs:            0 new,     5 changed,     0 unmodified
Added:      2.698 MiB

processed 421774 files, 16.520 GiB in 6:22
snapshot fd661ba0 saved

Traceback (most recent call last):
  File "/usr/local/bin/runrestic", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.4/dist-packages/runrestic/commands/runrestic.py", line 131, in main
    run_configuration(config, args)
  File "/usr/local/lib/python3.4/dist-packages/runrestic/commands/runrestic.py", line 88, in run_configuration
    rcs += repo.backup(config.get('backup'))
  File "/usr/local/lib/python3.4/dist-packages/runrestic/restic/__init__.py", line 68, in backup
    self.log['restic_backup'] = parse_backup(output)
  File "/usr/local/lib/python3.4/dist-packages/runrestic/restic/output_parser.py", line 9, in parse_backup
    added_to_the_repo = re.findall('Added to the repo:\s+(-?[0-9.]+ [a-zA-Z]*B)', output)[0]
IndexError: list index out of range

I think the problem is the large number of files but I don't know for sure. I tried to run runrestic in a Ubuntu 14.04.5 LTS with a few files and works fine but I tried in the other system and fails.

Greetings.

config file command line argument

It will be good if runrestic will have command line argument for set config file path. a.g.: runrestic -c /mypath/my-restic.conf
It will be useful when have 2 different repositories.

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Add "includes" in config file

Allow including other files.

Modify parse_configuration() to get a base config as parameter:

-def parse_configuration(config_filename: str) -> Dict[str, Any]:
+def parse_configuration(config_filename: str, config_base: Dict[str, Any]) -> Dict[str, Any]:
     logger.debug(f"Parsing configuration file: {config_filename}")
     with open(config_filename) as file:
         config = toml.load(file)
 
-    config = deep_update(config_base, dict(config))
+    config = deep_update(CONFIG_DEFAULTS, dict(config))
 
     if "name" not in config:
         config["name"] = os.path.basename(config_filename)
 
     jsonschema.validate(instance=config, schema=SCHEMA)
     return config

Replace

parsed_cfg = parse_configuration(c)
with

parsed_cfg = parse_configuration(c, CONFIG_DEFAULTS)
for include in reversed(config['includes']):
    parsed_cfg = parse_configuration(include, parsed_cfg)

Use return code from restic

When using runrestic within a cron job it would be helpful to look at the return code to notify the admin if the backup failed. But Runrestic always return zero.

[12:03:53 root@host] runrestic -l debug
Parsing configuration file: /etc/runrestic.toml
[Environment] RESTIC_PASSWORD=**********
Spawning "['restic', '-r', 'rest:xxx', 'backup', '/data', '/root']"
{'current_try': 1, 'tries_total': 1, 'output': [(1, 'Fatal: wrong password or no key found\n')], 'time': 0.5132162570953369}
Spawning "['restic', '-r', 'rest:xxx', 'forget', '--keep-last', '3', '--keep-hourly', '5', '--keep-daily', '7', '--keep-weekly', '8', '--keep-monthly', '6', '--keep-yearly', '11']"
[(1, 'Fatal: wrong password or no key found\n')]
Spawning "['restic', '-r', 'rest:xxx', 'prune']"
[(1, 'Fatal: wrong password or no key found\n')]
Spawning "['restic', '-r', 'rest:xxx', 'check']"
[(1, 'using temporary cache in /tmp/restic-check-cache-545556970\nFatal: wrong password or no key found\n')]
{
"errors": 4,
"backup": {
"rest:xxx": {
"rc": 1
}
},
"forget": {
"rest:xxx": {
"rc": 1
}
},
"prune": {
"rest:xxx": {
"rc": 1
}
},
"check": {
"rest:xxx": {
"errors": 0,
"errors_data": 0,
"errors_snapshots": 0,
"read_data": 0,
"check_unused": 0,
"duration_seconds": 0.5457494258880615,
"rc": 1
}
},
"last_run": 1628157841.395445,
"total_duration_seconds": 2.183987855911255
}
[12:04:01 root@host] echo $?
0

TOML config file in XDG_CONFIG_HOME is being parsed as JSON

When trying to run runrestic 0.5.23 the follow error comes up:

$ runrestic
Traceback (most recent call last):
  File "/home/connzen/.local/bin/runrestic", line 8, in <module>
    sys.exit(runrestic())
  File "/home/connzen/.local/pipx/venvs/runrestic/lib/python3.8/site-packages/runrestic/runrestic/runrestic.py", line 65, in runrestic
    parsed_cfg = parse_configuration(c)
  File "/home/connzen/.local/pipx/venvs/runrestic/lib/python3.8/site-packages/runrestic/runrestic/configuration.py", line 131, in parse_configuration
    else json.load(file)
  File "/usr/lib/python3.8/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I suspect the if conditional in

if str(config_filename).endswith(".toml")

prevents the config file from being parsed as TOML. Sadly, neither .json nor .toml work because the config file's name in XDG_CONFIG_HOME is expected to have no suffix.

Include restic logs

I tried runrestic and it seems to work in principle, but I don't get any log messages from restic itself, even if I specify log level "debug".
Is this intended behaviour or an issue with my installation?

I installed runrestic in Python 3.9 virtual environment and got no errors or warnings, for "backup" I only get the "spawning" message and no further putput.

Example output for "stats" command, which at least provides the result output:

$ sudo /opt/runrestic/bin/runrestic stats -l debug
Parsing configuration file: /etc/runrestic/user_backup.toml
[Environment] RESTIC_PASSWORD=**********
Spawning "['restic', '-r', 'rclone:qnap:my_pc/restic/users', 'stats', '-q', '--json']"
{
  "errors": 0,
  "stats": {
    "rclone:qnap:my_pc/restic/users": {
      "total_file_count": 3636345,
      "total_size_bytes": 52699188029,
      "duration_seconds": 156.62893509864807,
      "rc": 0
    }
  },
  "last_run": 1636301210.36535,
  "total_duration_seconds": 156.64174032211304
}

Use native bindings instead of string parsing

Use gopy to create the bindings to restic/cmd/cmd_backup.go#runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Terminal, args []string). The same should be done for the other commands of course as well.

Structs and string arrays should work out of the box, for termstatus.Terminal.New(wr io.Writer, errWriter io.Writer, disableStatus bool) one needs to implement io.Writer interface which only needs a simple Write(p []byte) (n int, err error) which should call the python logger callback.

Arch Linux package builds, but errors when run

If I'm lucky, you'll recognize exactly what I'm doing wrong. Otherwise, I'll continue to dig into what is going on... probably something silly on my part. ;)

0 ✓ fryfrog@apollo ~ $ runrestic
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2451, in resolve
    return functools.reduce(getattr, self.attrs, module)
AttributeError: module 'runrestic.runrestic.runrestic' has no attribute 'main'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/runrestic", line 11, in <module>
    load_entry_point('runrestic==0.5.0', 'console_scripts', 'runrestic')()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2453, in resolve
    raise ImportError(str(exc))
ImportError: module 'runrestic.runrestic.runrestic' has no attribute 'main'

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.