An Unofficial Script to Update or Restore Docker Engine and Docker Compose on Synology
License: MIT License
Mark Dumay is managing partner and owner of Squadra Analytics, a boutique consulting firm based in the Netherlands. With nearly 20 years of experience, Mark has helped numerous clients in Utilities, Telecom, and Wholesale to get more value from data and analytics.
Mark holds a Master in Computer Science from Delft University of Technology and is a certified Lean Six Sigma Black Belt. He maintains several open-source repositories on GitHub focused on Hugo, Synology, and Docker.
doctorpangloss@diskstation-1:~/synology-docker$ sudo ./syno_docker_update.sh --force download ./docker
Update Docker Engine and Docker Compose on Synology to target version
Current DSM version: 6.2.3
Current Docker version: Unknown
Current Docker Compose version: Unknown
Target Docker version: 19.03.12
Target Docker Compose version: 1.26.2
Step 1 from 2: Downloading target Docker binary (https://download.docker.com/linux/static/stable/x86_64/docker-19.03.12.tgz)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 57.9M 100 57.9M 0 0 80.7M 0 --:--:-- --:--:-- --:--:-- 80.7M
Step 2 from 2: Downloading target Docker Compose binary (https://github.com/docker/compose/releases/download/1.26.2/docker-compose-Linux-x86_64)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 638 100 638 0 0 1923 0 --:--:-- --:--:-- --:--:-- 1927
100 11.6M 100 11.6M 0 0 5988k 0 0:00:01 0:00:01 --:--:-- 7569k
Done.
Observe the install command fails to copy docker-compose (it was downloaded correctly)
doctorpangloss@diskstation-1:~/synology-docker$ sudo ./syno_docker_update.sh --force install ./docker
Update Docker Engine and Docker Compose on Synology to target version
Current DSM version: 6.2.3
Current Docker version: Unknown
Current Docker Compose version: Unknown
Target Docker version: 19.03.12
Target Docker Compose version: Unknown
Step 1 from 6: Stopping Docker service
Step 2 from 6: Backing up current Docker binaries (/volume1/homes/doctorpangloss/synology-docker/docker_backup_20200703_123301.tgz)
bin/
bin/containerd-shim
bin/docker-proxy
bin/dockerd
bin/docker
bin/auplink
bin/docker-compose
bin/ctr
bin/containerd
bin/runc
bin/docker-init
dockerd.json
Step 3 from 6: Extracting target Docker binary (/volume1/homes/doctorpangloss/synology-docker/docker/docker-19.03.12.tgz)
docker/
docker/containerd-shim
docker/docker
docker/dockerd
docker/docker-init
docker/runc
docker/docker-proxy
docker/containerd
docker/ctr
Step 4 from 6: Installing binaries
cp: cannot stat ‘/tmp/docker_update/docker-compose’: No such file or directory
Step 5 from 6: Configuring log driver
Step 6 from 6: Starting Docker service
pkgctl-Docker started.
The synoservicectl daemon does not always terminate as expected, possibly due to a conflict with Docker's live restore. To make the script more robust, a time-out should be added when calling such service, or when invoking any other command for that matter.
Revise the function execute_install_bin() to ensure the Docker binaries are owned by root. Below an excerpt of a related post on the Synology forum.
Make sure the binaries are owned by the
root:rootuser/group (chown -R root:root /var/packages/Docker/target/usr/bin) and executablechmod +x /var/packages/Docker/target/usr/bin/*
The script uses a timeout of 5 minutes waiting for synoservicectl to start/stop the Docker daemon. The script is unresponsive capturing terminal input during the waiting, e.g. ctrl-c is not processed to stop the script execution.
When specifying the -b flag to restore from a backup, the working directory is not correctly updated when specifying a fully qualified path. The script tries to extract /volume1/docker/synology-docker//volume1/docker/synology-docker/docker_backup_200611115000.tgz for example, while this should be /volume1/docker/synology-docker/docker_backup_200611115000.tgz.
The script invokes synoservicectl --stop pkgctl-Docker to stop the Docker daemon. Certain containers prevent a successful shutdown, possibly linked to Docker's Live Restore feature.
Usually, manually stopping any running containers prior to running syno_docker_update.sh does the trick. A new flag (e.g. --force-stop) for all commands except download is to automate this step. Possibly, the stopped containers can also be automatically restarted once the script has completed.
I am using a synology 513. with docker installed. I cannot determine what the proper location to install synology-docker folder into?
Sorry for the NOOB question
Replace the current web scraping instruction with a GitHub API request. Below example detects the latest restic version that can be adapted:
RESTIC_TAG_LATEST=$(curl --silent "https://api.github.com/repos/restic/restic/releases/latest" | grep -Po '"tag_name": "v\K.*?(?=")')
There is a new Docker Engine (20.10.03) released on https://www.synology.com/de-de/dsm/packages/Docker
Maybe this content can help to understand how to build newer versions
Prompt the user to confirm an install, restore, or update command.
Just did the full update process and found that my containers wouldn't start due to the "db" log driver not being found.
After a little digging, I noticed that the:
/var/packages/Docker/etc/dockerd.json
Still contains:
{
"data-root" : "/var/packages/Docker/target/docker",
"log-driver" : "db",
"registry-mirrors" : [],
"storage-driver" : "aufs"
}
The last time this file was changed was a month ago... guess that is when I installed a DSM update.
Using Synology-Docker to update your Synology Docker package is known to bring a few issues. They are listed below, including their workaround if available.
db, which is not supported by the Docker binaries installed by syno_docker_update.sh. The script updates the default logging driver to json-file to address this. However, containers that were started prior to the update might still refer to the now obsolete db driver. Removing and restarting the container(s) should fix the error.Failed to create btrfs snapshot: inappropriate ioctl for device on volumes formatted with BTRFS. @frobnicaty recommends to use Synology's BTRFS driver, which can be found in /usr/sbin/btrfs. Run which btrfs to test which driver you are currently using.synoservicectl daemon (DSM 6) does not always terminate as expected, possibly due to a conflict with Docker's live restore functionality. Home Assistant is a known example to use live restore. Manually shutting down the container(s) will ensure Synology-Docker runs correctly.Just wanted to say thank you for making this, looks like it was a lot of work, and helped me a bunch!
If I was to run the UPDATE command without any options specified, would Step L) delete the backup made in Step D)?
Fortunately I had a backup because I ran BACKUP with a path specified first. After running the UPDATE, I checked both the directory where the script was located and /tmp/docker_update... there was no backup in the script folder, and /tmp/docker_update didn't even exist.
Suggestions/Points:
If the backup is being deleted at the end, then you should definitely change that. Making a backup as part of the workflow is pointless if it is just going to be deleted at the end, because you are left with nothing to restore back to.
The heading for Workflow Step D) in the README.md file has a minor typo:
D) ackup current files
WORKING_DIR as dirname $0 (where the script is located) and have options which allow users to define an alternative path. Using /tmp makes sense for scripts that are run regularly because you know the OS (or at least something) will do the housekeeping for that directory and your script doesn't need to cater for it. For this though, it makes more sense for everything to be relative to the script directory, especially since that is where the binaries get downloaded to.-p (path) or -d (directory) option to allow the user to specify a working directory. -b (backup) would then be just the filename, which if not specified, defaults to what is currently in the script.I may be going a little over the top with this, so apologies for that. Geeky stuff like this is my idea of fun lol :D
I was wondering now that the built in Synology Docker is Version 20.10.x would it be possible to use a similar method you are using to only add in the NVIDIA Container Toolkit "nvidia-docker2" so that GPU's can be used in the Synology Docker environment?
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html
This would allow for Machine Learning and Transcoding support on Docker images that support it.
Containers attached to the default bridge network work as expected. The following command should spin up portainer correctly.
docker run -d -p 8000:8000 -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock portainer/portainer-ce
However, attaching it to a user-defined bridge network doesn't work correctly yet. Steps to reproduce:
docker network create my-net
docker run -d -p 8000:8000 -p 9000:9000 --name=portainer --network my-net --restart=always -v /var/run/docker.sock:/var/run/docker.sock portainer/portainer-ce
Docker provides extensive documentation about bridge networking. Especially the section Enable forwarding from Docker containers to the outside world seems relevant.
- Configure the Linux kernel to allow IP forwarding.
sysctl net.ipv4.conf.all.forwarding=1- Change the policy for the iptables FORWARD policy from DROP to ACCEPT.
sudo iptables -P FORWARD ACCEPT
Step 2 has been addressed in version v1.2.0 of the script. The first step doesn't work on Synology DSM yet.
The syno_docker_update.sh script updates the file /var/packages/Docker/etc/dockerd.json to fix an issue with Synology's db log driver. However, the script replaces the entire configuration file, instead of fixing the offending log driver only. This might introduce unwanted side effects.
Despite changing the logging driver to a json-file driver, docker on synology still seems to default to db-driver. Verification is required.
The following binaries could be created - text file busy?
Step 4 from 7: Installing binaries
cp: cannot create regular file ‘/var/packages/Docker/target/usr/bin/containerd’: Text file busy
cp: cannot create regular file ‘/var/packages/Docker/target/usr/bin/containerd-shim’: Text file busy
cp: cannot create regular file ‘/var/packages/Docker/target/usr/bin/docker’: Text file busy
cp: cannot create regular file ‘/var/packages/Docker/target/usr/bin/docker-init’: Text file busy
cp: cannot create regular file ‘/var/packages/Docker/target/usr/bin/docker-proxy’: Text file busy
cp: cannot create regular file ‘/var/packages/Docker/target/usr/bin/dockerd’: Text file busy
Step 5 from 7: Configuring log driver
Step 6 from 7: Enabling IP forwarding
Step 7 from 7: Starting Docker service
See docopt for examples
This isn't an issue with the script but thought I would ask here because it's caused by the update.
The uptime stats visible in the Synology Docker UI are invalid following the update.
Has the way the uptime is calculated / stored been changed between Docker versions? Is it something we can manually fix?
Setting up Docker on your Synology with the synology-docker script might result in difficulty connecting with containers in bridge mode.
Describe the bug
Exactly as it says in the title, the Docker service doesn't start after being updated with this script. Fortunately, everything still works if I restore the binaries from the backup but, the entire reason for me to use this script was to get a more up-to-date version of Docker due to the known issue of not being able to update environment variables with the version currently supplied by Synology.
Here is the specific error message:
Step 9 from 10: Starting Docker service
ERROR: Could not bring Docker Engine back online
To reproduce
I ran the following command:
sudo ./syno_docker_update.sh update
and it failed at "Step 9 from 10: Starting Docker service".
Expected behavior
I expected Docker to simply start running again after the update.
Log file
CardcaptorRLH85@Rin-san:~/synology-docker$ sudo ./syno_docker_update.sh update
Update Docker Engine and Docker Compose on Synology to target version
Current DSM version: 6.2.3
Current Docker version: 18.09.8
Current Docker Compose version: 1.24.0
Target Docker version: 20.10.2
Target Docker Compose version: 1.27.4
WARNING! This will replace:
- Docker Engine
- Docker Compose
- Docker daemon log driver
Are you sure you want to continue? [y/N] y
Step 1 from 10: Downloading target Docker binary (https://download.docker.com/linux/static/stable/x86_64/docker-20.10.2.tgz)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 65.7M 100 65.7M 0 0 10.7M 0 0:00:06 0:00:06 --:--:-- 11.1M
Step 2 from 10: Downloading target Docker Compose binary (https://github.com/docker/compose/releases/download/1.27.4/docker-compose-Linux-x86_64)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 651 100 651 0 0 4247 0 --:--:-- --:--:-- --:--:-- 4254
100 11.6M 100 11.6M 0 0 7434k 0 0:00:01 0:00:01 --:--:-- 8938k
Step 3 from 10: Stopping Docker service
pkgctl-Docker stoped.
Step 4 from 10: Backing up current Docker binaries (/volume1/homes/CardcaptorRLH85/synology-docker/docker_backup_20210116_160503.tgz)
bin/
bin/docker-init
bin/docker
bin/auplink
bin/ctr
bin/docker-compose
bin/runc
bin/containerd-shim
bin/containerd-shim-runc-v2
bin/containerd
bin/docker-proxy
bin/dockerd
dockerd.json
start-stop-status
Step 5 from 10: Extracting target Docker binary (/tmp/docker_update/docker-20.10.2.tgz)
docker/
docker/containerd
docker/docker-init
docker/ctr
docker/containerd-shim
docker/containerd-shim-runc-v2
docker/runc
docker/docker-proxy
docker/dockerd
docker/docker
Step 6 from 10: Installing binaries
Step 7 from 10: Configuring log driver
Step 8 from 10: Enabling IP forwarding
Step 9 from 10: Starting Docker service
ERROR: Could not bring Docker Engine back online
CardcaptorRLH85@Rin-san:~/synology-docker$
Docker daemon configuration
{
"data-root" : "/var/packages/Docker/target/docker",
"log-driver" : "json-file",
"registry-mirrors" : [],
"group": "administrators"
}
Additional context
As I was going back through the steps to collect the data for this bug report, I decided to try installing Docker version 19.03.14 (currently the latest version of the 19.x branch) instead of the latest version (as of now 20.10.2). Unfortunately, that too failed in the same manner. So, I finally tried an upgrade to the most recent version of the 18.x branch, 18.09.9 which is only one revision (and just under two months) newer than the eighteen-month-old 18.09.8 version that Synology currently ships. However, that also failed in the same manner.
Hi
I provided additional details on issue #3 which is still exists for me.
Rui
Probably I am missing something but the script is stuck at step 3. I can see Docker has stopped but the containers are still running. Is that even possible? Or should I let docker run and shut the containers?
Describe the bug
Getting error provided below while trying to make update.
To reproduce
The invoked command: sudo ./syno_docker_update.sh update
Expected behavior
Docker should be updated without errors
Log file
sudo ./syno_docker_update.sh update
Password:
Update Docker Engine and Docker Compose on Synology to target version
Current DSM version: 7.2
Current Docker version: 20.10.23
Current Docker Compose version: 2.9.0
Target Docker version: 24.0.1
Target Docker Compose version: 2.18.1
WARNING! This will replace:
- Docker Engine
- Docker Compose
- Docker daemon log driver
Are you sure you want to continue? [y/N] y
Step 1 from 10: Downloading target Docker binary (https://download.docker.com/linux/static/stable/x86_64/docker-24.0.1.tgz)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 66.3M 100 66.3M 0 0 12.8M 0 0:00:05 0:00:05 --:--:-- 13.5M
Step 2 from 10: Downloading target Docker Compose binary (https://github.com/docker/compose/releases/download/v2.18.1/docker-compose-linux-x86_64)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 52.0M 100 52.0M 0 0 4439k 0 0:00:11 0:00:11 --:--:-- 3666k
Step 3 from 10: Stopping Docker service
Step 4 from 10: Backing up current Docker binaries (/volume1/homes/vitalii/github/synology-docker/docker_backup_20230522_111542.tgz)
tar: /var/packages/Docker/target/usr: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
ERROR: Backup issue
Docker daemon configuration
Capture the content of the file /var/packages/Docker/etc/dockerd.json.
{
"data-root" : "/var/packages/Docker/target/docker",
"log-driver" : "json-file",
"registry-mirrors" : [],
"group": "administrators"
}
Additional context
Seems like Docker package was renamed from Docker to ContainerManager, so paths should be aligned with new package name
Revise the function execute_install_bin() to ensure the folder /var/lib/docker/volumes exists. Below an excerpt of a related post on the Synology forum.
Create an empty volumes directory using
sudo mkdir /var/lib/docker/volumesto make sure some poorly written code, like Portainer Agent, can find a valid directory there. It will not affect where volumes are saved (they will be correctly saved in/volume1/@docker
Describe the bug
with the script docker-compose cannot be downloaded because since docker-compose v2 the version part of the download url contains a leading 'v' which the script doesn't reflect.
The correct URL has to be
https://github.com/docker/compose/releases/download/v2.1.1/docker-compose-Linux-x86_64
not (https://github.com/docker/compose/releases/download/2.1.1/docker-compose-Linux-x86_64
To reproduce
./syno_docker_update.sh update
Expected behavior
Download should work
Log file
sudo ./syno_docker_update.sh update
Update Docker Engine and Docker Compose on Synology to target version
Current DSM version: 7.0.1
Current Docker version: 20.10.3
Current Docker Compose version: 1.28.5
Target Docker version: 20.10.11
Target Docker Compose version: 2.1.1
WARNING! This will replace:
Are you sure you want to continue? [y/N] y
Step 1 from 10: Downloading target Docker binary (https://download.docker.com/linux/static/stable/x86_64/docker-20.10.11.tgz)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 60.4M 100 60.4M 0 0 4426k 0 0:00:13 0:00:13 --:--:-- 5157k
Step 2 from 10: Downloading target Docker Compose binary https://github.com/docker/compose/releases/download/2.1.1/docker-compose-Linux-x86_64
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 9 100 9 0 0 17 0 --:--:-- --:--:-- --:--:-- 17
ERROR: Binary could not be downloaded
Docker daemon configuration
Capture the content of the file /var/packages/Docker/etc/dockerd.json.
Additional context
Add any other context about the problem here.
It had been a while since I messed around with an update so decided to give it a shot.
Docker v24.0.6 refused to start, tried v23.0.6 instead. That at least started but wasn't able to access any containers which used the default bridge network. Tried various ip routes but none of them helped. My macvlan still worked though, so just ended up switching all containers over to that.
Just wondering if anyone else has tried updating to v23 or v24 and whether similar issues were seen :)
Sources:
Hi @markdumay,
Each time i performed a reboot, i lost everything.
Docker is running but no of my container is available. I not sure what is the issue.
I solved this by restoring and upgrading all with your script.
Do you experience the same issue with reboot of your synology nas ?
Hi, after an upgrade with the script I cannot start any of my container.
I have this error message : "COI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: "bin/bash"...
Current DSM version: 6.2.3
Current Docker version: 19.03.8
Current Docker Compose version: 1.27.4
My Synology is a DS918+
The Synology Docker package comes with a user interface (UI) to monitor and launch containers. Unfortunately, the launching of containers via the UI no longer works after having upgraded Docker with Synology-Docker. This could be caused by the specific Docker logging driver of Synology. Launching containers from the command line (via either Docker or Docker Compose) still works. Portainer could also be an alternative, but has not been tested by the author yet.
Thanks for an excellent script!
Unfortunately, the updated binaries appear to be incompatible with BTRFS; given docker stack deploy stack -c stack.yaml all containers fail to launch with error: Failed to create btrfs snapshot: inappropriate ioctl for device.
Does anyone know what happens to Docker and macvlan network configurations when the Synology NAS is updated to DSM 7?
I'm guessing that everything will need to be updated again and the network reconfigured, but just thought I would ask so I can be prepared :)
The current script has been tested with DSM 6 and rejects other major versions. On DSM 6, the script uses synoservicectl to verify, start, and stop the Docker package pkgctl-Docker. DSM 7 has replaced synoservicectl with synopkg.
The following commands should replace the old commands on DSM 7 (root access required):
synopkg stop Dockersynopkg status Docker should return Docker package is stopped\nStatus: [1]synopkg start Dockersynopkg status Docker should return Docker package is startedCurrent DSM version: 6.2.3
Current Docker version: 18.09.8
Current Docker Compose version: 1.24.0
Target Docker version: 19.03.11
Target Docker Compose version: Unknown
Step 1 from 6: Stopping Docker service
pkgctl-Docker stoped.
Should there be a Docker Compose version displayed, rather than "Unknown"?
Also, "stopped" is spelt incorrectly :p
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.