kenkundert / borg-space Goto Github PK
View Code? Open in Web Editor NEWAccessory for Emborg used to report and track the size of your Borg repositories
License: GNU General Public License v3.0
Accessory for Emborg used to report and track the size of your Borg repositories
License: GNU General Public License v3.0
Since upgrading to borg-space v2, it now seems to require a config file to be present:
xcp-alpine-emborg:~/.config/emborg# borg-space -r downloads
borg-space error: /root/.config/borg-space/settings.nt:
/root/.config/borg-space/settings.nt: no such file or directory.
What is the minimum content of this file? If I create an empty one it complains about a missing 'repositories' entry. If I create one, with a single repository name in it, I get:
xcp-alpine-emborg:~/.config/emborg# borg-space -r downloads
Traceback (most recent call last):
File "/usr/bin/borg-space", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/lib/python3.11/site-packages/borg_space/main.py", line 329, in main
repos = collect_repos(requests, cmdline['--record'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg_space/main.py", line 69, in collect_repos
repos = get_repos(request)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg_space/config.py", line 216, in get_repos
to_process.extend(gather(child))
^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg_space/config.py", line 192, in gather
if name.isidentifier() and name in repositories:
^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'isidentifier'
This is the contents of the settings file:
xcp-alpine-emborg:~/.config/emborg# cat ~/.config/borg-space/settings.nt
repositories:
downloads:
Just wondering if this should still depend on emborg? The docs say it reads files created by emborg 1.36 to determine the space used, so surely this should still be a requirement?
I'm trying to package this up for Alpine, and on initial testing it's not working:
/ # emborg -c pcbackup list
2021-03-31-03-56-49
2021-04-30-14-39-50
2021-05-29-23-31-43
2021-06-30-02-56-15
2021-07-31-04-21-42
2021-08-22-17-11-08
2021-08-29-16-54-33
2021-08-31-23-34-57
2021-09-05-03-19-15
2021-09-12-23-26-38
2021-09-16-03-21-03
2021-09-17-03-18-56
2021-09-21-09-59-20
2021-09-22-03-31-47
2021-09-23-23-21-09
2021-09-24-03-04-32
2021-09-25-03-47-29
/ # borg-space pcbackup
Traceback (most recent call last):
File "/usr/bin/borg-space", line 161, in <module>
generate_report(
File "/usr/bin/borg-space", line 103, in generate_report
for config in configs:
TypeError: 'NoneType' object is not iterable
/ # emborg -c pcbackup info
config: pcbackup
roots: /backups/pcbackup
destination: [email protected]:nas/pcbackup
settings directory: /root/.config/emborg
logfile: /root/.local/share/emborg/pcbackup.log
last backed up: 2021-09-25T08:58:29.612608+01:00, 5 hours ago
Repository ID: b61faf103d5c2cd9c4da2b7baef556e22107e3bcf954799e6bf15d318dde445b
Location: ssh://#redacted#@#redacted#/./nas/pcbackup
Encrypted: Yes (repokey BLAKE2b)
Cache: /root/.cache/borg/b61faf103d5c2cd9c4da2b7baef556e22107e3bcf954799e6bf15d318dde445b
Security dir: /root/.config/borg/security/b61faf103d5c2cd9c4da2b7baef556e22107e3bcf954799e6bf15d318dde445b
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
All archives: 17.57 TB 16.93 TB 1.46 TB
Unique chunks Total chunks
Chunk index: 603889 7141189
Could it be that it can't find the Emborg API? Where does the Emborg API need to be installed to be accessible by borg-space
?
It looks like the {repo}
replacement has been changed to {name}
, but the original version is still present in the readme (in the example configuration file).
Getting ready for the next release, trying to package up the latest changes for Alpine to make sure I'm good to go once you release it. I seem to be getting semi-random test failures. Sometimes the tests will fail, and others they'll work just fine:
All the failures look like this:
_________________________________ test_main[3] _________________________________
tmp_files = TestPath('/tmp/pytest-of-buildozer/pytest-1/test_main_3_0')
name = 'remedy', args = 'home@❬HOST❭~❬USER❭', env = {}
stdout = 'home@❬HOST❭~❬USER❭: 4.15 GiB', stderr = '^$', status = 0
@parametrize(
schema = borg_space_schema,
indirect = ['tmp_files'],
)
def test_main(tmp_files, name, args, env, stdout, stderr, status):
> check_command(name, args, env, stdout, stderr, status, home=tmp_files)
tests/test_cli.py:94:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_cli.py:83: in check_command
Matches(stderr, flags=re.DOTALL).assert_matches(process.stderr)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Matches('^$', flags=regex.S | regex.V0)
# regex failed to match at:
#
#> Traceback (most recent call last)...name
#> raise Invalid(f"{arg}: expected a name")
#> voluptuous.error.Invalid: 6f24e76175ac: expected a name
s = 'Traceback (most recent call last):\n File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2....03, in a_name\n raise Invalid(f"{arg}: expected a name")\nvoluptuous.error.Invalid: 6f24e76175ac: expected a name\n'
def assert_matches(self, s: str) -> None:
> assert self == s, self._fail
E AssertionError: regex failed to match at:
E
E > Traceback (most recent call last):
E ^
E > File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/borg_space/config.py", line 154, in get_repos
E > children = repositories[spec]
E > ~~~~~~~~~~~~^^^^^^
E > KeyError: 'home@6f24e76175ac~buildozer'
E >
E > During handling of the above exception, another exception occurred:
E >
E > Traceback (most recent call last):
E > File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/./bs", line 5, in <module>
E > main()
E > File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/borg_space/main.py", line 332, in main
E > repos = collect_repos(requests, cmdline['--record'])
E > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E > File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/borg_space/main.py", line 64, in collect_repos
E > new_repos = get_repos(request)
E > ^^^^^^^^^^^^^^^^^^
E > File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/borg_space/config.py", line 158, in get_repos
E > children = [Repository(spec)]
E > ^^^^^^^^^^^^^^^^
E > File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/borg_space/config.py", line 69, in __init__
E > self.host = a_name(host) or hostname
E > ^^^^^^^^^^^^
E > File "/home/andy/alpine-packaging/aports/community/borg-space/src/borg-space-2.1/borg_space/config.py", line 203, in a_name
E > raise Invalid(f"{arg}: expected a name")
E > voluptuous.error.Invalid: 6f24e76175ac: expected a name
/usr/lib/python3.11/site-packages/re_assert.py:63: AssertionError
----------------------------- Captured stdout call -----------------------------
Running: remedy
It appears that borg-space can't handle an emborg config containing a '-':
xcp-alpine-emborg:~# borg-space syno-docker
borg-space error: syno-docker: expected identifier
Tests are failing after pytest upgraded to 8.1.1 on Alpine linux edge x86_64:
======================================= test session starts ========================================
platform linux -- Python 3.11.8, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/ncopa/aports/community/borg-space/src/borg-space-2.2
configfile: pyproject.toml
plugins: cov-4.1.0, xdist-3.5.0, pytest_tmp_files-0.0.2, mock-3.10.0, requests-mock-1.11.0
32 workers [60 items]
..........................F..........F...................... [100%]
============================================= FAILURES =============================================
__________________________________________ test_main[59] ___________________________________________
[gw26] linux -- Python 3.11.8 /home/ncopa/aports/community/borg-space/src/borg-space-2.2/.testenv/bin/python3
tmp_files = TestPath('/tmp/pytest-of-ncopa/pytest-69/popen-gw26/test_main_59_0'), name = 'misdo'
args = 'home', env = {}
stdout = 'bs error:\\s+❬HOME❭/.config/borg-space/settings.nt, repositories, home\\+!primary:\n unknown key.'
stderr = '^$', status = 1
@parametrize(
schema = borg_space_schema,
indirect = ['tmp_files'],
)
def test_main(tmp_files, name, args, env, stdout, stderr, status):
> check_command(name, args, env, stdout, stderr, status, home=tmp_files)
tests/test_cli.py:101:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_cli.py:91: in check_command
Matches(stdout, flags=re.DOTALL).assert_matches(borg_space.stdout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Matches('bs error:\\s+/tmp/pytest-of-ncopa/pytest-69/popen-gw26/test_main_59_0/.config/borg-space/settings.nt, reposit...ontains invalid characters: ‘!+’.
# ^
#> 2 ❬ home+!primary: home❭
#> ▲
s = 'bs error: /tmp/pytest-of-ncopa/pytest-69/popen-gw26/test_main_59_0/.config/borg-space/settings.nt, repositories, home+!primary:\n key contains invalid characters: ‘!+’.\n 2 ❬ home+!primary: home❭\n ▲\n'
def assert_matches(self, s: str) -> None:
> assert self == s, self._fail
E AssertionError: regex failed to match at:
E
E > bs error: /tmp/pytest-of-ncopa/pytest-69/popen-gw26/test_main_59_0/.config/borg-space/settings.nt, repositories, home+!primary:
E > key contains invalid characters: ‘!+’.
E ^
E > 2 ❬ home+!primary: home❭
E > ▲
/usr/lib/python3.11/site-packages/re_assert.py:63: AssertionError
--------------------------------------- Captured stdout call ---------------------------------------
Test name: misdo
__________________________________________ test_main[58] ___________________________________________
[gw25] linux -- Python 3.11.8 /home/ncopa/aports/community/borg-space/src/borg-space-2.2/.testenv/bin/python3
tmp_files = TestPath('/tmp/pytest-of-ncopa/pytest-69/popen-gw25/test_main_58_0'), name = 'prelude'
args = 'home', env = {}
stdout = 'bs error:\\s+❬HOME❭/.config/borg-space/settings.nt, repositories, home\\+primary:\n unknown key.'
stderr = '^$', status = 1
@parametrize(
schema = borg_space_schema,
indirect = ['tmp_files'],
)
def test_main(tmp_files, name, args, env, stdout, stderr, status):
> check_command(name, args, env, stdout, stderr, status, home=tmp_files)
tests/test_cli.py:101:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_cli.py:91: in check_command
Matches(stdout, flags=re.DOTALL).assert_matches(borg_space.stdout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Matches('bs error:\\s+/tmp/pytest-of-ncopa/pytest-69/popen-gw25/test_main_58_0/.config/borg-space/settings.nt, reposit...ontains an invalid character: ‘+’.
# ^
#> 2 ❬ home+primary: home❭
#> ▲
s = 'bs error: /tmp/pytest-of-ncopa/pytest-69/popen-gw25/test_main_58_0/.config/borg-space/settings.nt, repositories, home+primary:\n key contains an invalid character: ‘+’.\n 2 ❬ home+primary: home❭\n ▲\n'
def assert_matches(self, s: str) -> None:
> assert self == s, self._fail
E AssertionError: regex failed to match at:
E
E > bs error: /tmp/pytest-of-ncopa/pytest-69/popen-gw25/test_main_58_0/.config/borg-space/settings.nt, repositories, home+primary:
E > key contains an invalid character: ‘+’.
E ^
E > 2 ❬ home+primary: home❭
E > ▲
/usr/lib/python3.11/site-packages/re_assert.py:63: AssertionError
--------------------------------------- Captured stdout call ---------------------------------------
Test name: prelude
===================================== short test summary info ======================================
FAILED tests/test_cli.py::test_main[59] - AssertionError: regex failed to match at:
FAILED tests/test_cli.py::test_main[58] - AssertionError: regex failed to match at:
=================================== 2 failed, 58 passed in 3.87s ===================================
There are currently no releases tagged. I'm about to package this up for Alpine, so it'd be good to have an 'official' release to build against, and use its version number.
If borg-space is run from the 'run_after_backup' setting, it looks like emborg hasn't yet written the size:
Running Borg create command ...
------------------------------------------------------------------------------
Repository: [ssh://[email protected]/./repo](mailto:ssh://[email protected]/./repo)
Archive name: 2023-06-05-21-05-58
Archive fingerprint: f3130428a6764cc0bab96ac5f51c77a997e41e950c874baefc1ee24fe085badd
Time (start): Mon, 2023-06-05 21:06:00
Time (end): Mon, 2023-06-05 21:06:18
Duration: 17.98 seconds
Number of files: 15248
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 96.58 GB 94.74 GB 1.70 kB
All archives: 2.32 TB 2.27 TB 90.54 GB
Unique chunks Total chunks
Chunk index: 47437 1197568
------------------------------------------------------------------------------
emborg error: run_after_backup, 0, borg-space:
Traceback (most recent call last):
File "/usr/bin/borg-space", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/lib/python3.11/site-packages/borg_space/main.py", line 329, in main
repos = collect_repos(requests, cmdline['--record'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg_space/main.py", line 84, in collect_repos
data[now] = repo['size'].fixed()
~~~~^^^^^^^^
KeyError: 'size'
emborg returns 2
I have both 'prune_after_create' and 'check_after_create' set to False in the config. This is what I have for 'run_after_backup':
#run_after_backup = [
# 'borg-space -r {config_name} --style compact',
# 'borg-space -g -S /var/www/localhost/htdocs/{config_name}.svg {config_name}'
#]
When I try to use the new version, I'm getting an error:
xcp-alpine-test:~# emborg --version
1.27.0 (2021-09-21)
xcp-alpine-test:~# borg-space luned
Traceback (most recent call last):
File "/usr/bin/borg-space", line 173, in <module>
generate_report(
File "/usr/bin/borg-space", line 108, in generate_report
if emborg.version < (1,27) or configs is None:
TypeError: '<' not supported between instances of 'NoneType' and 'tuple'
If running borg-space in a console, using '-g' doesn't seem to do anything.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.