kenkundert/borg-space

Accessory for Emborg used to report and track the size of your Borg repositories

borg-space's Issues

New version stopped working

When I try to use the new version, I'm getting an error:

xcp-alpine-test:~# emborg --version
1.27.0 (2021-09-21)
xcp-alpine-test:~# borg-space luned
Traceback (most recent call last):
  File "/usr/bin/borg-space", line 173, in <module>
    generate_report(
  File "/usr/bin/borg-space", line 108, in generate_report
    if emborg.version < (1,27) or configs is None:
TypeError: '<' not supported between instances of 'NoneType' and 'tuple'

Make 'official' release

There are currently no releases tagged. I'm about to package this up for Alpine, so it'd be good to have an 'official' release to build against, and use its version number.

Formatting of axis labels

While looking at getting the new version working, I've noticed that some of the axis labels aren't really readable:

A couple of examples:

Y axis:

X axis:

borg-space doesn't handle config containing '-'

It appears that borg-space can't handle an emborg config containing a '-':

xcp-alpine-emborg:~# borg-space syno-docker
borg-space error: syno-docker: expected identifier

Depends on emborg still?

Just wondering if this should still depend on emborg? The docs say it reads files created by emborg 1.36 to determine the space used, so surely this should still be a requirement?

v2 seems to require a config file

Since upgrading to borg-space v2, it now seems to require a config file to be present:

xcp-alpine-emborg:~/.config/emborg# borg-space -r downloads
borg-space error: /root/.config/borg-space/settings.nt:
    /root/.config/borg-space/settings.nt: no such file or directory.

What is the minimum content of this file? If I create an empty one it complains about a missing 'repositories' entry. If I create one, with a single repository name in it, I get:

xcp-alpine-emborg:~/.config/emborg# borg-space -r downloads
Traceback (most recent call last):
  File "/usr/bin/borg-space", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/python3.11/site-packages/borg_space/main.py", line 329, in main
    repos = collect_repos(requests, cmdline['--record'])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg_space/main.py", line 69, in collect_repos
    repos = get_repos(request)
            ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg_space/config.py", line 216, in get_repos
    to_process.extend(gather(child))
                      ^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/borg_space/config.py", line 192, in gather
    if name.isidentifier() and name in repositories:
       ^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'isidentifier'

This is the contents of the settings file:

xcp-alpine-emborg:~/.config/emborg# cat ~/.config/borg-space/settings.nt
repositories:
    downloads:

tests fails with pytest 8

Tests are failing after pytest upgraded to 8.1.1 on Alpine linux edge x86_64:

======================================= test session starts ========================================
platform linux -- Python 3.11.8, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/ncopa/aports/community/borg-space/src/borg-space-2.2
configfile: pyproject.toml
plugins: cov-4.1.0, xdist-3.5.0, pytest_tmp_files-0.0.2, mock-3.10.0, requests-mock-1.11.0
32 workers [60 items]     
..........................F..........F......................                                 [100%]
============================================= FAILURES =============================================
__________________________________________ test_main[59] ___________________________________________
[gw26] linux -- Python 3.11.8 /home/ncopa/aports/community/borg-space/src/borg-space-2.2/.testenv/bin/python3

tmp_files = TestPath('/tmp/pytest-of-ncopa/pytest-69/popen-gw26/test_main_59_0'), name = 'misdo'
args = 'home', env = {}
stdout = 'bs error:\\s+❬HOME❭/.config/borg-space/settings.nt, repositories, home\\+!primary:\n    unknown key.'
stderr = '^$', status = 1

    @parametrize(
        schema = borg_space_schema,
        indirect = ['tmp_files'],
    )
    def test_main(tmp_files, name, args, env, stdout, stderr, status):
>       check_command(name, args, env, stdout, stderr, status, home=tmp_files)

tests/test_cli.py:101: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_cli.py:91: in check_command
    Matches(stdout, flags=re.DOTALL).assert_matches(borg_space.stdout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Matches('bs error:\\s+/tmp/pytest-of-ncopa/pytest-69/popen-gw26/test_main_59_0/.config/borg-space/settings.nt, reposit...ontains invalid characters: ‘!+’.
    #      ^
    #>            2 ❬    home+!primary: home❭
    #>                   ▲
s = 'bs error: /tmp/pytest-of-ncopa/pytest-69/popen-gw26/test_main_59_0/.config/borg-space/settings.nt, repositories, home+!primary:\n    key contains invalid characters: ‘!+’.\n           2 ❬    home+!primary: home❭\n                  ▲\n'

    def assert_matches(self, s: str) -> None:
>       assert self == s, self._fail
E       AssertionError:  regex failed to match at:
E       
E       > bs error: /tmp/pytest-of-ncopa/pytest-69/popen-gw26/test_main_59_0/.config/borg-space/settings.nt, repositories, home+!primary:
E       >     key contains invalid characters: ‘!+’.
E             ^
E       >            2 ❬    home+!primary: home❭
E       >                   ▲

/usr/lib/python3.11/site-packages/re_assert.py:63: AssertionError
--------------------------------------- Captured stdout call ---------------------------------------
Test name: misdo
__________________________________________ test_main[58] ___________________________________________
[gw25] linux -- Python 3.11.8 /home/ncopa/aports/community/borg-space/src/borg-space-2.2/.testenv/bin/python3

tmp_files = TestPath('/tmp/pytest-of-ncopa/pytest-69/popen-gw25/test_main_58_0'), name = 'prelude'
args = 'home', env = {}
stdout = 'bs error:\\s+❬HOME❭/.config/borg-space/settings.nt, repositories, home\\+primary:\n    unknown key.'
stderr = '^$', status = 1

    @parametrize(
        schema = borg_space_schema,
        indirect = ['tmp_files'],
    )
    def test_main(tmp_files, name, args, env, stdout, stderr, status):
>       check_command(name, args, env, stdout, stderr, status, home=tmp_files)

tests/test_cli.py:101: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_cli.py:91: in check_command
    Matches(stdout, flags=re.DOTALL).assert_matches(borg_space.stdout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Matches('bs error:\\s+/tmp/pytest-of-ncopa/pytest-69/popen-gw25/test_main_58_0/.config/borg-space/settings.nt, reposit...ontains an invalid character: ‘+’.
    #      ^
    #>            2 ❬    home+primary: home❭
    #>                   ▲
s = 'bs error: /tmp/pytest-of-ncopa/pytest-69/popen-gw25/test_main_58_0/.config/borg-space/settings.nt, repositories, home+primary:\n    key contains an invalid character: ‘+’.\n           2 ❬    home+primary: home❭\n                  ▲\n'

    def assert_matches(self, s: str) -> None:
>       assert self == s, self._fail
E       AssertionError:  regex failed to match at:
E       
E       > bs error: /tmp/pytest-of-ncopa/pytest-69/popen-gw25/test_main_58_0/.config/borg-space/settings.nt, repositories, home+primary:
E       >     key contains an invalid character: ‘+’.
E             ^
E       >            2 ❬    home+primary: home❭
E       >                   ▲

/usr/lib/python3.11/site-packages/re_assert.py:63: AssertionError
--------------------------------------- Captured stdout call ---------------------------------------
Test name: prelude
===================================== short test summary info ======================================
FAILED tests/test_cli.py::test_main[59] - AssertionError:  regex failed to match at:
FAILED tests/test_cli.py::test_main[58] - AssertionError:  regex failed to match at:
=================================== 2 failed, 58 passed in 3.87s ===================================

Can't run borg-space in the 'run_after_backup' as it causes emborg to fail, meaning size isn't written

If borg-space is run from the 'run_after_backup' setting, it looks like emborg hasn't yet written the size:

Running Borg create command ...
------------------------------------------------------------------------------
Repository: [ssh://[email protected]/./repo](mailto:ssh://[email protected]/./repo)
Archive name: 2023-06-05-21-05-58
Archive fingerprint: f3130428a6764cc0bab96ac5f51c77a997e41e950c874baefc1ee24fe085badd
Time (start): Mon, 2023-06-05 21:06:00
Time (end):   Mon, 2023-06-05 21:06:18
Duration: 17.98 seconds
Number of files: 15248
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               96.58 GB             94.74 GB              1.70 kB
All archives:                2.32 TB              2.27 TB             90.54 GB

                       Unique chunks         Total chunks
Chunk index:                   47437              1197568
------------------------------------------------------------------------------
emborg error: run_after_backup, 0, borg-space:
    Traceback (most recent call last):
      File "/usr/bin/borg-space", line 8, in <module>
        sys.exit(main())
                 ^^^^^^
      File "/usr/lib/python3.11/site-packages/borg_space/main.py", line 329, in main
        repos = collect_repos(requests, cmdline['--record'])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/site-packages/borg_space/main.py", line 84, in collect_repos
        data[now] = repo['size'].fixed()
                    ~~~~^^^^^^^^
    KeyError: 'size'
emborg returns 2

I have both 'prune_after_create' and 'check_after_create' set to False in the config. This is what I have for 'run_after_backup':

#run_after_backup = [
#       'borg-space -r {config_name} --style compact',
#       'borg-space -g -S /var/www/localhost/htdocs/{config_name}.svg {config_name}'
#]

Unable to get working on Alpine

I'm trying to package this up for Alpine, and on initial testing it's not working:

/ # emborg -c pcbackup list
2021-03-31-03-56-49
2021-04-30-14-39-50
2021-05-29-23-31-43
2021-06-30-02-56-15
2021-07-31-04-21-42
2021-08-22-17-11-08
2021-08-29-16-54-33
2021-08-31-23-34-57
2021-09-05-03-19-15
2021-09-12-23-26-38
2021-09-16-03-21-03
2021-09-17-03-18-56
2021-09-21-09-59-20
2021-09-22-03-31-47
2021-09-23-23-21-09
2021-09-24-03-04-32
2021-09-25-03-47-29
/ # borg-space pcbackup
Traceback (most recent call last):
  File "/usr/bin/borg-space", line 161, in <module>
    generate_report(
  File "/usr/bin/borg-space", line 103, in generate_report
    for config in configs:
TypeError: 'NoneType' object is not iterable
/ # emborg -c pcbackup info
              config: pcbackup
               roots: /backups/pcbackup
         destination: [email protected]:nas/pcbackup
  settings directory: /root/.config/emborg
             logfile: /root/.local/share/emborg/pcbackup.log
      last backed up: 2021-09-25T08:58:29.612608+01:00, 5 hours ago

Repository ID: b61faf103d5c2cd9c4da2b7baef556e22107e3bcf954799e6bf15d318dde445b
Location: ssh://#redacted#@#redacted#/./nas/pcbackup
Encrypted: Yes (repokey BLAKE2b)
Cache: /root/.cache/borg/b61faf103d5c2cd9c4da2b7baef556e22107e3bcf954799e6bf15d318dde445b
Security dir: /root/.config/borg/security/b61faf103d5c2cd9c4da2b7baef556e22107e3bcf954799e6bf15d318dde445b
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:               17.57 TB             16.93 TB              1.46 TB

                       Unique chunks         Total chunks
Chunk index:                  603889              7141189

Could it be that it can't find the Emborg API? Where does the Emborg API need to be installed to be accessible by borg-space?

Add option to write plot to a file

If running borg-space in a console, using '-g' doesn't seem to do anything.

Seemingly random test failures

Getting ready for the next release, trying to package up the latest changes for Alpine to make sure I'm good to go once you release it. I seem to be getting semi-random test failures. Sometimes the tests will fail, and others they'll work just fine:

All the failures look like this:

_________________________________ test_main[3] _________________________________

tmp_files = TestPath('/tmp/pytest-of-buildozer/pytest-1/test_main_3_0')
name = 'remedy', args = 'home@❬HOST❭~❬USER❭', env = {}
stdout = 'home@❬HOST❭~❬USER❭: 4.15 GiB', stderr = '^$', status = 0

    @parametrize(
        schema = borg_space_schema,
        indirect = ['tmp_files'],
    )
    def test_main(tmp_files, name, args, env, stdout, stderr, status):
>       check_command(name, args, env, stdout, stderr, status, home=tmp_files)

tests/test_cli.py:94:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_cli.py:83: in check_command
    Matches(stderr, flags=re.DOTALL).assert_matches(process.stderr)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Matches('^$', flags=regex.S | regex.V0)
    # regex failed to match at:
    #
    #> Traceback (most recent call last)...name
    #>     raise Invalid(f"{arg}: expected a name")
    #> voluptuous.error.Invalid: 6f24e76175ac: expected a name
s = 'Traceback (most recent call last):\n  File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2....03, in a_name\n    raise Invalid(f"{arg}: expected a name")\nvoluptuous.error.Invalid: 6f24e76175ac: expected a name\n'

    def assert_matches(self, s: str) -> None:
>       assert self == s, self._fail
E       AssertionError:  regex failed to match at:
E
E       > Traceback (most recent call last):
E         ^
E       >   File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/borg_space/config.py", line 154, in get_repos
E       >     children = repositories[spec]
E       >                ~~~~~~~~~~~~^^^^^^
E       > KeyError: 'home@6f24e76175ac~buildozer'
E       >
E       > During handling of the above exception, another exception occurred:
E       >
E       > Traceback (most recent call last):
E       >   File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/./bs", line 5, in <module>
E       >     main()
E       >   File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/borg_space/main.py", line 332, in main
E       >     repos = collect_repos(requests, cmdline['--record'])
E       >             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       >   File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/borg_space/main.py", line 64, in collect_repos
E       >     new_repos = get_repos(request)
E       >                 ^^^^^^^^^^^^^^^^^^
E       >   File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/borg_space/config.py", line 158, in get_repos
E       >     children = [Repository(spec)]
E       >                 ^^^^^^^^^^^^^^^^
E       >   File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/borg_space/config.py", line 69, in __init__
E       >     self.host = a_name(host) or hostname
E       >                 ^^^^^^^^^^^^
E       >   File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/borg_space/config.py", line 203, in a_name
E       >     raise Invalid(f"{arg}: expected a name")
E       > voluptuous.error.Invalid: 6f24e76175ac: expected a name

/usr/lib/python3.11/site-packages/re_assert.py:63: AssertionError
----------------------------- Captured stdout call -----------------------------
Running: remedy

New version readme needs updating

It looks like the {repo} replacement has been changed to {name}, but the original version is still present in the readme (in the example configuration file).