>> CHANGELOG <<

[zilexa]

>> CHANGELOG <<

[zilexa]

NOTIFICATION - I am testing a switch to an immutable OS, Fedora Silverblue. This would greatly simplify the management, maintenance, updates and stability of the server. An immutable OS means the OS is simply a base image, with a few layers over it for system drivers/tools. Updating = switching to other base image.

This is similar to how iOS, Android and the latest MacOS and ChromeOS work. The OS becomes a static image, no modifications are possible. Only user-level configurations.
I already have my home laptop running smoothly! With all apps we use daily like the browser etc.

Additionally, With Fedora comes Cockpit to manage the OS remotely via a webGUI. Which means you really do not need to have physical access to your server anymore.

I expect to switch my server to Fedora Silverblue in December and will update the guide + server-prep script.
Until then, progress will be visible here soon: https://github.com/zilexa/Fedora-Silverblue-Intuitive-Postinstall

This will include other changes such as switching to Yacht instead of Portainer and including Immich as Google Photos alternative. It is an experimental but very promising tool. Also, Syncthing will be included again, because webDAV (via Filerun) has filesize limits, preventing HQ videos made by digital cameras and phones to be synced to the server.
A lot of good things coming!

[zilexa]

Updated 29/8 see changelog. Thanks to P3nkiln in #13.

[zilexa]

Updated 4/11: Nightly & Monthly maintenance fixes:
https://github.com/zilexa/Homeserver/blob/master/docker/HOST/nightly.sh#L57 now unmounts the backup drive after performing monthly maintenance
https://github.com/zilexa/Homeserver/blob/master/docker/HOST/monthly.sh removed balancing of btrfs metadata as that should only be done for specific reasons.

Guide update!
AdGuard Home configuration instructions updated:

• Added instructions to disable AdGuard Home own "Browsing Security Service" as it will sent every request to AdGuard, makes no sense, since you can get the same level of security by adding a malware filter.
• added instructions to a add a good malware filter.

[zilexa]

Updated 28/02:

FileRun maintenance tasks were executed from Nightly, as root. This is not OK and leads to issues, since users might not be able to delete their folders when FileRun maintenance had created thumbnails (a user cannot delete files created by root).
I do not want to split cronjobs over multiple crontabs, per user. For home use I believe it makes more sense to have a single overview of your cronjobs and a single place to manage them. Basically any other tasks requires root (cleaning the OS, creating backups and executing drive management) or can be executed by root just fine (like cleaning up shows and movies).
For this one task, we simply add a job to root cron, but let it execute as the logged in user.
To do so without personalizing files (adding user names in bash scripts):

• prep-server.sh: will now create an env variable $LOGUSER that is available for root cronjobs and contains the regular user. For more info see: prep-server.sh This is required because cron only has very limited environment. For example ${USER} points to root and ${LOGNAME} are not available for cron.
• Filerun maintenance is moved (again) to a separate script for easy manual execution when needed. Nightly.sh will execute this script as regular user using the env variable. This way, Nightly.sh stays a script that you execute via sudo while filerun.sh is a script you execute as regular user, instead of mixing it up.
• https://github.com/zilexa/Homeserver/tree/master/maintenance-tasks#step-6-schedule-nightly-and-monthly has been updated to reflect the new cronjobs for FileRun.

Monthly.sh now simply executes docker pull and compose up to update all containers monthly with the latest images. While I believe you should never auto-update, because any update could break your configuration, the goal has always been to require minimal user attention, the server should just run. Since we have a robust backup system and it is easy to rollback, this saves the only monthly time I was spending on my server.

BTRFS maintenance: Balance now no longer balances metadata as that should never have been the case.
Also balancing data no longer happens in steps due to BTRFS filesystem improvements. The balance command will automatically handle this. This simplifies the Monthly script.

System updates are now properly executed monthly, by first updating the OS repository mirrors list to find fastest server, update local databases to prevent conflicts (common if you do not update for 6 months, less common if you update monthly) and then updates are performed.
In a next update, I want to add a command that will sent you an additional email when a reboot is required after updating :)

Mediacleaner is no longer developed, its now https://github.com/terrelsa13/MUMC.
prep-server.sh is updated with new downloadlinks but will still use the name "mediacleaner" as it makes more sense.
The guide for mediacleaner configuration is also updated.

Maintenance tasks guide now has an updated URL but I did not go through the entire guide yet to update the URL everywhere:
https://github.com/zilexa/Homeserver/tree/master/maintenance-tasks

[zilexa]

UPDATE 2023-03-01

• prep-server.sh now limits journal log filesize to 50MB. See link, execute the command to apply to your running server.

• prep-server.sh Bleachbit never properly executed as root. Fixed. See link, execute the command to apply to your running server an also update your Monthly.

• Monthly.sh freshed up

  • fixed Bleachbit not running for the regular user by adding $LOGUSER (see previous update) to the command
  • Added clear labels
  • Stopped using DIUN to notify for updates, 1 email per image makes no sense to me. Not possible to send 1 email with a list of available image updates. Plus there was a bug with DIUN.
  • Stopped using Pullio for auto-updating *arr and download apps. Instead just auto updating all images now.

Server works more autonomous with these last changes by updating the system itself and also updating all docker images itself.

Final(?) step: auto-send email when a reboot is required after OS update. This would be the only maintenance required.

[zilexa]

UPDATE 2023-03-20 (IMPORTANT!)

• I discovered during Nightly, when you run Filerun maintenance it will index ALL files and folders, not just the user-assigned folders. This means it will index the snapshots folder and all its snapshots, exponentially increasing its database and using lots of time every night to index those files.
  With this change, the snapshots folder is excluded. NOTE ! !: remove your Filerun container via Portainer (toggle persistent volume removal on) then use this command to recreate the container:

docker-compose up -V -d

Just to be sure, add -V.

• I also noticed I had not updated compose.yml in a while, only locally. The biggest difference I noticed is there were networks missing in this repository version. But there might be other differences. I didn't do a compare.
  In the past, web-proxy network was only used for services exposed online through Caddy (https). But we have been using Caddy (together with AGH) also to proxy local services (for example portainer is accessible via http://docker.o/), so local services also use web-proxy network (unless they use network mode Host).
  To isolate Filerun database, it runs in its own network.
  Unbound runs in its own network because there is no website to proxy anyway. At least by declaring a network for it, docker will not use some random name for it.

[zilexa]

UPDATE 2023-03-21

FANTASTIC NEWS !
OnlyOffice DocumentServer is now working again!
This means you can now use your browser with Filerun even to create and edit your Office documents.
The docker-compose.yml has been updated with 2 containers, OnlyOffice Documentserver and its database. Make sure your subdomain https://office.yourdomain.tld is working!
After running compose, check Office container log via Portainer and wait until it is finished. Note it takes a couple of minutes so be patient.
Then go to your office domain, you should see a welcome page. After that, you can configure Filerun via Settings > Plugins. You need the token from the .env file. Do not forget to set Default apps in Filerun to use OnlyOffice.

Enjoy!!

[zilexa]

UPDATE 2023-03-21

• Reduced Nightly.sh to the absolute minimum: commented out the monthly maintenance tasks because doing it right after nightly backup only makes sense if you have spinning disks (to reduce spin ups, which uses a lot of power compared to the rest of the server). Uncomment and adjust if you still have HDDs.
• Upgraded Monthly:
  • Added the monthly maintenance tasks for backup drive (scrub and balance) from the Nightly.
  • Added a storage usage status report!
  • The email is now created differently, at the top of the email, you will see if your system needs to be restarted. Next, you will see a storage usage report (for Media filesystem, for Users filesystem and then per Users). After that the output of the regular maintenance tasks is shown.
  • To accommodate this, a few changes have been made at the bottom. Because storage status report should run after BTRFS Balance. And if you include OS filesystem storage status, it should also run at the end after all maintenance tasks. But you want this info at the top of your email, this is why the changes were needed.

Correction: now tested, corrected, tested again. Works!

[zilexa]

UPDATE 2023-03-23

• OnlyOffice: switched from postgresql to mariadb and now using a separate container for rabbitmq.
  • reason: OnlyOffice takes 20min to start when using its built-in rabbitmq server. Now we use a separate container for rabbitmq.
  • Also switched to MariaDB, smaller footprint and less resources/more performant for small databases compared to postgresql.
• changed path of OnlyOffice volumes, its now stored in $HOME/docker/filerun/office instead of its own office folder. Because we will only use it with filerun this makes more sense.
• Plugsy labels for OnlyOffice containers updated.

[zilexa]

UPDATE 2023-04-12

Updated Compose:

• Plugsy labels are now descriptive.
• Pullio labels were old remnants since we don't use Pullio anymore. Cleaned up.
• Unifi: changed to use network_mode HOST, makes more sense.
  • With this mode, the Port section becomes useless - removed.
  • Unifi should be reachable via http://unifi.o/ but its stubborn so you still need to use a port number (http://unifi.o:8080/ or 8443/.

Updated Nightly:

• Removed mediacleaner/MUMC because a Jellyfin plugin is now available!! Finally :) I will update the guide later. This means you just have to add the repository via the Jellyfin web UI, install the plugin + configure it via the UI!
• Removed mediacleaner download from prep-server.sh script.

NOTE: with this change Nightly has now become an optional script only containing optional tasks! Because it only runs the btrbk backup script. This script can now replace Nightly in Cron. I will update Maintenance guide reflecting these changes.

[zilexa]

UPDATE 2023-04-14

Updated Maintenance Guide

• Almost a full rewrite, pointing out the awesome Monthly script taking care of unattended, automatic maintenance and also listing the manual maintenance tasks left to do.
• Clear separation between Backup Guide (referring to Maintenance Guide for the scheduling part) and Maintenance Guide (referring to Backup Guide).
• Nightly script now completely optional.
• Made more clear if you use Nightly, that should be in cron, while backup should then be performed via Nightly.
• Made more clear how to adjust the schedule/time.

Updated prep-server.sh

• Server will now automatically email you on S.M.A.R.T. errors and high temperature of your storage devices, using built in, already present smartd service. No additional tools required! See S.M.A.R.T. monitoring. HIGHLY recommended to apply to your existing server!

[zilexa]

UPDATE 2023-04-16

BACKUPS - IMPORTANT
I interpreted a setting in btrbk configuration incorrectly:

• snapshot_preserve_min, target_preserve_min and archive_preserve_min were set to LATEST.
• This causes your retention policy in ..._preserve to be ignored and only latest to be saved.
• This has now been changed to the longest value of the retention policy. For example:
  snapshot_preserve set to 7d 4w 6m and snapshot_preserve_min to 6m (instead of latest). Same for archive and target.
  See: btrbk.conf in docker/HOST/btrbk.

[zilexa]

UPDATE 2023-04-19

Fixed a small inconvenience: btrbk-mail.sh
I use an email rule in my Gmail to filter server emails to a folder/label and mark as read, with the exception if the subject or body contains the word ERROR.
It is annoying you get ERROR emails just because a backup drive was already mounted (I mounted it manually to restore something and forgot to unmount it). Now, the error is not triggered for this device.
I also made sure it is added to the list of mounted drives, to unsure the script will unmount the drive when done. Previously, the script did not unmount already mounted drives, only drives that were mounted by the script.

[zilexa]

UPDATE 2023-04-20

docker-compose.yml minor change:
tv related containers names prefixed with "tv-" to easily identify them in the list of containers.

[zilexa]

UPDATE 2023-05-08

prep-server.sh:

• Added ability to run docker-compose from any dir without specifying the location of compose file by adding a user environment variable to both zsh and bash. Unfortunately, docker doesn't look for the environment variable on a system-wide level in /etc/environment. It must be for user only.
• renamed docker-compose.yml to compose.yml.

monthly.sh:

• running docker commands should be run as the regular user, but the script is executed via root cronjob. Adjusted the 2 docker commands to ensure they are run as regular user (just like Bleachbit is run for regular user). Note the script is in root cron instead of regular cron because btrfs and btrbk commands do need root privileges.
• Docker update command now generates a nice list of what images have been updated!
• Bleachbit output for both runs is now added to email.
• docker system prune now replaces previous 2 cleanup commands and only the last line of this command is added to email (showing how much space has been freed up).

[zilexa]

compose.yml

• Renamed all containers ("container_name") and volumes to categorize each container.

• This also means all volume paths changed. I did not like the long list of folders in $HOME/docker, having to search for the right one.
  Now there are just 4 categories of Docker services:

  • Cloud
  • Dashboards
  • Networking
  • TV
    In the same folder, you will have:
  • HOST
  • compose.yml
  • .env
    This is much cleaner now, instead of having all containers volumes in 1 folder.

• Added InfluxDB for system monitoring (reason: energy consumption monitoring) and Telegraf for the required metrics. But I do not have a telegraf configuration, so that container will not start. Not sure if I will ever use InfluxDB+Telegraf. But this is supposed to be the most lightweight combination to visualise metrics of your server and docker services. Nothing else needed.

• Unbound now runs in host network mode, this requires you change its config, which can only be done if you put it in a named volume. The unbound config is available here. This is Klutchell Unbound image standard config, just the port changed to 5335 so that it doesn't conflict with Adguard.

[zilexa]

docker/HOST/btrbk/btrbk-mail.sh

To prevent data corruption, docker containers should be stopped before creating a snapshot of the @docker subvolume.
For example, containers could be writing data to your docker mapped volumes at the moment you snapshot. To prevent this from happening, the btrbk-mail.sh script wil stop containers before snapshotting and start them again when finished.
Currently this is just happening at the start and end of the script, instead of only when performing the snapshot action. That means containers will be offline for the entire duration of the script to finish (sending snapshots to backup locations could take a while).
I will change this when I have more time.

[zilexa]

compose.yml
Lot's of small changes to docker-compose (mostly cosmetical, changing names of services, containers and networks to easily identify what is what).

Monthly.sh

• Trying to fix the docker update summary. A bit difficult to test, but I think it should be OK now. Lets see end of June.
• Commented out Bleachbit user run section. Will probably remove Bleachbit completely. Instead will use arch wiki recommendations to cleanup system and home folders.

Reducing dependency on 3rd party apps and sticking as close to common practices (Arch Wiki) as possible hopefully makes it easier to switch to MicroOS (OpenSUSE Aeon) in the future (this is my plan post-summer 2023).

[zilexa]

btrbk-mail
Now properly stops docker containers. This command was executed as sudo and failed because root user has no compose.yml file. Now its properly executed as the regular user. Additionally, the btrbk run command options moved to its own variable, just like in the original script.

Filerun advanced configuration file
This file is required for docker users to set the correct timezone, enable a fix for webDAV and optionally configure your own url shortener. See the file for more details. Ensure it is in this folder and restart your container (requires latest Filerun version 2023).

compose.yml
Finally decided I am not going down the dashboarding rabbithole with InfluxDB+Telegraph. It needs a config file to run, which I am not going to figure out. Commented out these 2 services. Feel free to use and share your configs.

unbound.conf
Removed the path to logfile because this just leads to a HUGE logfile that is never cleaned up. Also, you can't see the log in Portainer. By removing the path, you can now always see the latest log via Portainer.

[zilexa]

Maintenance Guide
The crontab had a typo, preventing the tasks from being executed, changed $LOGUSER to ${LOGUSER}.

[zilexa]

Backups - btrbk-mail.sh
From now on, docker containers will be stopped for snapshot creation only, and restarted after snapshots have been created.
Since snapshotting is instant, this means your services will no longer be down for the entire duration of sending snapshots to their destinations. Instead, downtime will be a matter of seconds only.

Enjoy !!