netinvent/npbackup

A secure and efficient file backup solution that fits both system administrators (CLI) and end users (GUI)

NetInvent SASU

Committed to open source Development

Happy to code our projects and help improve others.

npbackup's Issues

comments - can we have something for mere mortals?

Now when I can finally try it on macOS I allow myself to have few comments:

1. It is very cool and well designed solution - kudos to you
2. It seems a bit geeky (at least for me) on the surface - I really struggle how to make it work e.g. with restic rclone back end. What I mean is hat maybe I can make it to work but I see very steep learning curve for average user to start using it. Maybe it is just documentation and few straight in your eyes examples missing which would be mega handy.
3. I have read your laptop backup approach - nice - but for example me I would love to be able not to run backup only when it is too old. I know how to make my laptop to wake up at given time. I want to use snapshots specific to my OS - some pre/post backup hooks to run custom scripts would be great - then I can handle it.
4. What about having multiple repos? I do backup to few. I would love to have an option to switch between them.

These are just comments - or wish list - I do not expect that all can happen. But I can still wish:)

Allow pipe in data in CLI mode

See N.L

Housekeeping server zero knowledge

This would be a killer feature for backends.
In the meantime, we might create a proxy service that receives housekeep commands
Investigate how to protect from cryptos... Does housekeep server have a "bare minimum to keep" policy ?

NPBACKUP cannot connect repo when it has already initialized

I have been using restic for a long time and I would like to attache npbackup to restic's repository. I configured the program, but it run into an error: Not connected to repo.
Here is debug information:

2023-12-29 20:53:14,741 :: INFO :: Runner took 1.027485 seconds
2023-12-29 20:53:14,742 :: INFO :: Searching for a backup newer than 1 day, 0:00:00 ago.
2023-12-29 20:53:14,742 :: DEBUG :: Running command: ["C:\Users\xxxx\AppData\Local\Temp\2\onefile_4416_133483531920371038\npbackup\restic.exe" init --repository-version 2 --compression auto]
2023-12-29 20:53:14,744 :: DEBUG :: Setting envrionment variable "AWS_ACCESS_KEY_ID"
2023-12-29 20:53:14,744 :: DEBUG :: Setting envrionment variable "AWS_SECRET_ACCESS_KEY"
2023-12-29 20:53:14,746 :: DEBUG :: Setting GOMAXPROCS to 6
2023-12-29 20:53:15,117 :: DEBUG :: Command ""C:\Users\xxxxxxx\AppData\Local\Temp\2\onefile_4416_133483531920371038\npbackup\restic.exe" init --repository-version 2 --compression auto" returned with exit code "1". Command output was:
2023-12-29 20:53:15,117 :: DEBUG :: STDOUT: Fatal: create key in repository at s3:https://xxxxx.wasabisys.com/xxxxxxxxx failed: repository master key and config already initialized
2023-12-29 20:53:15,124 :: DEBUG :: Running on_exit callable.
2023-12-29 20:53:15,125 :: ERROR :: Cannot connect to repository or repository empty.
2023-12-29 20:53:15,126 :: INFO :: Runner took 0.383868 seconds

It seems to me the npbackup tries to initialize the repo but it run into an error. Is it possible to disable this function?

Bad return code when trying to create remote rest repo

When remote rest-server returns 500 error (cannot create repo dir), npbackup won't alert about the error.

Allow post-script to be executed on interruption

Add an option to catch CTRL+C calls so post exec scripts get exectuted by atexit

Make prometheus metrics configurable

Should a warning (example: no VSS support / missing exclude files) trigger a metrics failure ?
If so, add an opt-out option so warnings don't generate error reports.

Investigate usage of alternative GUI

Since PySimpleGUI v5 now needs a license, we'll need to check whether we should go for another GUI framework.
So far, https://github.com/flet-dev/flet seems a nice alternative.

pre and post-script execution errors should be prometheus metrics

Latest restic version

Discussed in #49

Add --external-backend-binary

Include cube backup plugin

Building on macOS

Following https://github.com/netinvent/npbackup/blob/main/COMPILE_INSTRUCTIONS.md

So far so good - I can install all dependencies but it fails when compiling

python3 bin/compile.py --audience public
INFO: Building with default secret key
Cannot find restic source file.

The reason is that it does not recognize restic_0.15.2_darwin_amd64 as valid restic executable. I am guessing only linux/windows are in the code for arch?

restic_0.15.2_darwin_arm64 and restic_0.15.2_darwin_amd64 should be valid options here

Make sure cmd line --list and --ls returns proper codes

When rest server is offline, --list and --ls should return corresponding error messages instead of empty snapshot lists.

Implement "recover", "migrate" and "rewrite" commands

Can't change hostname

I tried to make Machine identifier to b a hard coded value different than hostname but it is still backed up with the machine's hostname on windows.

I was backing up through restic's rest-server, not sure if it makes any difference.

Shift tests to automatic

We need at least:

• backup test
• check test (with different time)
• restore test
• comparaison test
• ls test
• list test

QUESTION - npbackup log file location

hello and first, thank you for the creation of npbackup!

I have questions on the npbackup logfile and the location of it.

On my workstation i have put npbackup in /usr/local/bin/ as suggested. Then I created an application "icon" (.desktop file) that starts npbackup via bash as follows:
/usr/local/bin/npbackup --config-file=/home/linux/npbackup/npbackup.conf

This is for a non priviliged user backing up some paths in the home folder. With this i get repeated messages about the logfile:

Cannot create logfile. Trying to obtain temporary log file.
Message: [Errno 13] Permission denied: '/usr/local/bin/npbackup.log'
Trying temporary log file in /tmp/ofunctions.logger_utils.log
[Errno 13] Permission denied: '/usr/local/bin/npbackup.log'
Using [/tmp/ofunctions.logger_utils.log]
2024-01-13 16:18:38,172 :: WARNING :: Failed to use log file "/usr/local/bin/npbackup.log", [Errno 13] Permission denied: '/usr/local/bin/npbackup.log'
Using [/tmp/ofunctions.logger_utils.log].
2024-01-13 16:18:38,173 :: INFO :: npbackup v2.2.1-P7-x64 2023082801 - GUI enabled - Copyright (C) 2022-2023 NetInvent
...

Now when i run
npbackup --config-file=/home/linux/npbackup/npbackup.conf
instead then the logfile is created in whatever path i launch this command, i.e. /home/linux/npbackup/. I would prefer to keep the logfile for future reference so /tmp/ofunctions.logger_utils.log is not really working.
My workaround is to do a cd /path/for/my/npbackup/logfile/ before calling npbackup without leading path.

My questions:

1. the logfile location is not fixed or configurable, correct?
2. is there a reason or rationale for this behavior?

Secondary repository

I try to create an initial backup but always get the following error:

Fatal: Secondary repository must only be specified when copying the chunker parameters

In the tab "Backup destination" I've set the path to my repository folder: $HOME/Backups/NPBackup and set the password.

Add a --api output parameter to CLI

This will help use the CLI in an API way.
Actually, this should be the default core output, which is then interpreted by the CLI.

Suggestion to improve Backup in progress bar

I'm sure its current version is temporary, but on Windows 11, its always on top which is a bit of a problem. I just move it down to the right of the screen and that is really cool.

Which gave me the idea that it might be really nice as a Windows 11 notification. To me, this would be the first good use of notifications. Of course, Linux also has notifications. That's extra work though.

Might be even better with less work, just keeping it as it is, but showing it at the corner of the screen and in a little to get past the task bar (or whatever) and have a button on it to get rid of it. Still have it draggable and/or setting to choose the corner. It is actually really handy to have visible like this.

Something to think about :)

Error in opening rest-server repository

When I try to open repository from rest-server npbackup process gets killed both x32 and x64 version

2023-10-12 20:43:20,405 :: INFO :: npbackup v2.2.0-P10-x64 2023060301 - GUI enabled - Copyright (C) 2022-2023 NetInvent
2023-10-12 20:43:21,017 :: INFO :: Backend: restic 0.15.2 compiled with go1.20.3 on windows/amd64
2023-10-12 20:43:21,033 :: INFO :: Running GUI
2023-10-12 20:43:21,277 :: INFO :: Listing snapshots
2023-10-12 20:43:21,378 :: INFO :: Repo already initialized.
2023-10-12 20:43:21,866 :: INFO :: Runner took 0.589341 seconds
2023-10-12 20:43:21,866 :: INFO :: Searching for a backup newer than 1 day, 0:00:00 ago.
2023-10-12 20:43:22,323 :: INFO :: No recent backup found.
2023-10-12 20:43:22,325 :: INFO :: Runner took 0.459628 seconds
2023-10-12 20:43:22,325 :: ERROR :: Program interrupted by error. 'username'
2023-10-12 20:43:22,325 :: INFO :: Trace:
Traceback (most recent call last):
 File "C:\Users\asd\AppData\Local\Temp\ONEFIL~1\npbackup\__main__.py", line 529, in main
 File "C:\Users\asd\AppData\Local\Temp\ONEFIL~1\npbackup\__main__.py", line 507, in interface
 File "C:\Users\asd\AppData\Local\Temp\ONEFIL~1\npbackup\gui\main.py", line 551, in main_gui
 File "C:\Users\asd\AppData\Local\Temp\ONEFIL~1\npbackup\gui\main.py", line 171, in get_gui_data
 File "C:\Users\asd\AppData\Local\Temp\ONEFIL~1\concurrent\futures\_base.py", line 451, in result
 File "C:\Users\asd\AppData\Local\Temp\ONEFIL~1\concurrent\futures\_base.py", line 403, in __get_result
 File "C:\Users\asd\AppData\Local\Temp\ONEFIL~1\ofunctions\threading\__init__.py", line 81, in call_with_future
 File "C:\Users\asd\AppData\Local\Temp\ONEFIL~1\npbackup\gui\main.py", line 120, in _get_gui_data
KeyError: 'username'
2023-10-12 20:49:41,693 :: INFO :: npbackup v2.2.1-P11-x64 2023082801 - GUI enabled - Copyright (C) 2022-2023 NetInvent
2023-10-12 20:49:42,301 :: INFO :: Backend: restic 0.16.0 compiled with go1.20.6 on windows/amd64
2023-10-12 20:49:42,324 :: INFO :: Running GUI
2023-10-12 20:49:42,553 :: INFO :: Listing snapshots
2023-10-12 20:49:42,790 :: INFO :: Repo already initialized.
2023-10-12 20:49:43,229 :: INFO :: Runner took 0.676515 seconds
2023-10-12 20:49:43,229 :: INFO :: Searching for a backup newer than 1 day, 0:00:00 ago.
2023-10-12 20:49:43,675 :: INFO :: No recent backup found.
2023-10-12 20:49:43,676 :: INFO :: Runner took 0.446841 seconds
2023-10-12 20:49:43,681 :: ERROR :: Program interrupted by error. 'username'
2023-10-12 20:49:43,682 :: INFO :: Trace:
Traceback (most recent call last):
 File "C:\Users\asd\AppData\Local\Temp\ON5969~1\npbackup\__main__.py", line 529, in main
 File "C:\Users\asd\AppData\Local\Temp\ON5969~1\npbackup\__main__.py", line 507, in interface
 File "C:\Users\asd\AppData\Local\Temp\ON5969~1\npbackup\gui\main.py", line 589, in main_gui
 File "C:\Users\asd\AppData\Local\Temp\ON5969~1\npbackup\gui\main.py", line 171, in get_gui_data
 File "C:\Users\asd\AppData\Local\Temp\ON5969~1\concurrent\futures\_base.py", line 449, in result
 File "C:\Users\asd\AppData\Local\Temp\ON5969~1\concurrent\futures\_base.py", line 401, in __get_result
 File "C:\Users\asd\AppData\Local\Temp\ON5969~1\ofunctions\threading\__init__.py", line 81, in call_with_future
 File "C:\Users\asd\AppData\Local\Temp\ON5969~1\npbackup\gui\main.py", line 120, in _get_gui_data
KeyError: 'username'

Suggestion about separate "see content" module.

Are you consider to release "see content" module only? For browsing existing Restic archives (non npbackup ones). Your solution is elegant and fast, would be nice to have such a tool available. Just a thought.

rest-server returns bogus result when remote connection sends html error page

2023-12-08 14:32:49,351 :: DEBUG :: Running command: ["/usr/local/bin/restic" init --repository-version 2 --compression auto --limit-upload 500000]
2023-12-08 14:32:49,352 :: DEBUG :: Setting GOMAXPROCS to 1
2023-12-08 14:32:49,712 :: DEBUG :: Command "['/usr/local/bin/restic', 'init', '--repository-version', '2', '--compression', 'auto', '--limit-upload', '500000']" returned with exit code "1". Command output was:
2023-12-08 14:32:49,713 :: DEBUG :: STDOUT: Fatal: create repository at rest:https://USER:***@myhost.tld/USER/ failed: config file already exists


2023-12-08 14:32:49,714 :: DEBUG :: Running on_exit callable.
2023-12-08 14:32:49,715 :: INFO :: Repo already initialized.
2023-12-08 14:32:49,716 :: DEBUG :: Running command: ["/usr/local/bin/restic" snapshots --json --limit-upload 500000]
2023-12-08 14:32:49,717 :: DEBUG :: Setting GOMAXPROCS to 1
2023-12-08 14:33:42,181 :: DEBUG :: Command "['/usr/local/bin/restic', 'snapshots', '--json', '--limit-upload', '500000']" returned with exit code "1". Command output was:
2023-12-08 14:33:42,183 :: DEBUG :: STDOUT: List(key) returned error, retrying after 693.009248ms: Decode: invalid character '<' looking for beginning of value
List(key) returned error, retrying after 760.178904ms: Decode: invalid character '<' looking for beginning of value
List(key) returned error, retrying after 1.010456495s: Decode: invalid character '<' looking for beginning of value
List(key) returned error, retrying after 1.348040655s: Decode: invalid character '<' looking for beginning of value
List(key) returned error, retrying after 2.001668959s: Decode: invalid character '<' looking for beginning of value
List(key) returned error, retrying after 2.321936782s: Decode: invalid character '<' looking for beginning of value
List(key) returned error, retrying after 4.001640537s: Decode: invalid character '<' looking for beginning of value
List(key) returned error, retrying after 7.088563909s: Decode: invalid character '<' looking for beginning of value
List(key) returned error, retrying after 12.024584913s: Decode: invalid character '<' looking for beginning of value
List(key) returned error, retrying after 20.724782293s: Decode: invalid character '<' looking for beginning of value
Fatal: Decode: invalid character '<' looking for beginning of value

Investigate if this is a rest-server issue or a npbackup one

Windows install doesn't install -- Errno 22 Invalid argument

Running the installer NPBackupInstaller.x64.exe yields this log file.

2023-11-08 21:26:44,289 :: INFO :: Running npbackup.installer 1.1.7
2023-11-08 21:26:44,387 :: ERROR :: Could not terminate currant scheduled task Client Sauvegarde NetPerfect:\ERROR: The system cannot find the file specified.


2023-11-08 21:26:44,388 :: INFO :: Copying C:\Users\<user>\AppData\Local\Temp\onefile_14732_133439776035659634\npbackup.exe to C:\Program Files\NetPerfect Backup Client\npbackup.exe
2023-11-08 21:26:46,894 :: ERROR :: Could not copy file C:\Users\<user>\AppData\Local\Temp\onefile_14732_133439776035659634\npbackup.exe to C:\Program Files\NetPerfect Backup Client\npbackup.exe: [Errno 22] Invalid argument: 'C:\\Users\\<user>\\AppData\\Local\\Temp\\onefile_14732_133439776035659634\\npbackup.exe'

Is there a hard coded path in the installer? When I look in the temp folder there is no folder onefile_14732_133439776035659634

Add encrypted field for rclone config password

I use restic with rclone backend, and set this on the Environment variables tab:
RCLONE_CONFIG_PASS=

Could we make a specific encrypted field for that?

Allow loading alternative encryption keys

So we're pretty close to a v3 release.
Would be nice to have a --key-location to load the AES key from a separate file instead of using the baked in one.
Another way would be a --key-command which is supposed to give the AES key when run.
None of these options could be encrypted by said AES key, but only obfuscated locally.

This needs to be discussed in SECURITY.md file too.

Make GUI for cron entry

Identical to windows capabilities
Need to know where cron files are, and what user we're running as.

Improve exclusion system

• Add ignore missing exclude files option > generates warning instead of backend failure
• Add inline ignore (one per line) so we don't have to deal with exclusion files

Remote usage on headless Linux NAS server?

Hi there,
Thanks for providing this software to easier cope with Restic.
I have a question towards a possible usage scenario.
Is it possible, and if so how would I achieve this, to use npbackup on a headless Linux NAS server operating with the npbackup GUI from a remote PC?
Thanks in advance for a hint.

building macOS application bundle

Now when I think that macOS compiled application works it is time to make it proper macOS native application bundle.

At the moment binary is generated but when running with GUI it results with opening console (terminal) window - it is ok for debugging but not for normal use.

I have found the way to produce proper macOS style application bundle but in order to make it work I need macOS compatible icon file.

Could you please provide npbackup_icon.ico as ideally svg image format or at least 512x512 bitmap? This is what is required to create icns file. When I have svg or 512x512 bitmap I will create icns and finalize apple app build.

Allow multiple pre and post exec commands

This will allow to run freeze/thaw as well as SQL dump commands.
Also easens #9
Make selected commands not fatal on failure (by selecting line number ?)

Add "Tag" names to the list of backups

I have a few different, unique sources in the single repository backup, so displaying a "tag" names in the GUI's list of backups would be helpful. Now it is just date and ID's.

Suggestion to improve dealing with file errors during backup

Windows 11

Run backup

Point 1 - Backup failed pops up on the screen of which 1) is a problem itself as it disrupts the user in the worst way possible) and 2) the backup didn't actually fail so the message is completely wrong here.

Log:
Warning: at least one source file could not be read
...
error: open ...ntuser.dat.LOG1: The process cannot access the file because it is being used by another process. ...........blah blah yes VSS.
...
error: open ...AppData\Local\Temp\nsdBAB3.tmp: Access is denied. .........problem.

Point 2 - Its possible to make the GUI show a list of failed files with checkboxes or exclude all or whatever so that its quick and easy and awesome. The user shouldn't have to think "The backup failed. The backup is no good. I don't know what to do. Oh no! What's a log file? I can't do that. Where's the log file? I have to hunt through the log file. I have to manually copy and paste and exclude all files or manually exclude folders."

Of course, running in the background from the task in Task Scheduler does solve all of that to an extent. But, people won't see that way what files are not being backed up and would have to check the log every time. Instead, its possible to popup a non focus stealing notification such as "click here to view files that were not backed up" or whatever that shows the same thing as in point 2.

Point 3 - Should be able to just list them in their own file and then import the bytes into Strings and lines and have checkboxes or some other way. Technically, just putting them into their own log file would be tons better and should be quick to code.

Just a suggestion :)

Version 3 alpha3 not parsing date in negative timezone

The npbackup/npbackup/gui/main.py line 547

r"[0-9]{4}-[0-1][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-5][0-9]..*+[0-2][0-9]:[0-9]{2}",

This + needs to be + or -
I'm in EST which is -04:00

Windows x64 Executable Flagged by Virus Scanner

npbackupx64.exe from the Releases page is flagged by Windows Defender as containing an active trojan:

Detected: Trojan:Win32/Wacatac.B!ml

Microsoft directs users to this help page after downloading.

Alert on backup smaller than

As it says, it would make an alert if a backup is suspiciously too small

Investigate password protected actions

Optional protection for:

• Restore browser
• Forget/prune (local housekeeping option ?)
• check

Investigate other backends

The easiest alternative backend would be rustic.

Other alternatives include borg backup, kopia and bupstash.

Allow backup failure logs upload to central server

Improve upgrade server so it can take backup logs for easier diagnostics

Retention policy

Looking for convert my scripts to Npbackup, only three things are missing: retention policy, deleting a snapshots and prune option. As I understand, currently they are not directly available? I suppose "post-exec" command may be used for this purpose.