deajan/osync

A robust two way (bidirectional) file sync script based on rsync with fault tolerance, POSIX ACL support, time control and near realtime sync

osync

A two way filesync script running on bash Linux, BSD, Android, MacOSX, Cygwin, MSYS2, Win10 bash and virtually any system supporting bash. File synchronization is bidirectional, and can be run manually, as scheduled task, or triggered on file changes in monitor mode. It is a command line tool rsync wrapper with a lot of additional features baked in.

This is a quickstart guide, you can find the full documentation on the author's site.

About

osync provides the following capabilities:

• Local-Local and Local-Remote sync
• Fault tolerance with resume scenarios
• POSIX ACL and extended attributes synchronization
• Full script Time control
• Soft deletions and multiple backups handling
• Before/after run command execution
• Email alerts
• Logging facility
• Directory monitoring
• Running on schedule or as daemon
• Batch runner for multiple sync tasks with rerun option for failed sync tasks

osync is a stateful synchronizer. This means it's agentless and doesn't have to monitor files for changes. Instead, it compares replica file lists between two runs. A full run takes about 2 seconds on a local-local replication and about 7 seconds on a local-remote replication. Disabling some features file like attributes preservation and disk space checks may speed up execution. osync uses a initiator/target sync schema. It can sync local to local or local to remote directories. By definition, initiator replica is always a local directory on the system osync runs on. osync uses pidlocks to prevent multiple concurrent sync processes on/to the same initiator/target replica. You may launch concurrent sync processes on the same system but as long as the replicas to synchronize are different. Multiple osync tasks may be launched sequentially by osync osync-batch tool.

Tested platforms

Operating system	Version
AlmaLinux	9
Android*	Not known
CentOS	5.x, 6.x, 7.x
Fedora	22-25
FreeBSD	8.3-11
Debian	6-11
Linux Mint	14-18
macOS	Not known
pfSense	2.3.x
QTS (x86)	4.5.1
Ubuntu	12.04-22.04
Windows**	10

* via Termux. ** via MSYS, Cygwin and WSL.

Some users also have successfully used osync on Gentoo and created an OpenRC init script for it.

Installation

osync has been designed to not delete any data, but rather make backups of conflictual files or soft deletes. Nevertheless, you should always have a neat backup of your data before trying a new sync tool.

Getting osync via GitHub (remove the -b "stable" if you want latest dev snapshot)

$ git clone -b "stable" https://github.com/deajan/osync
$ cd osync
$ bash install.sh

Installer script accepts some parameters for automation. Launch install.sh --help for options.

There is also an RPM file that should fit RHEL/CentOS/Fedora and basically any RPM based distro, see the GitHub release. Please note that RPM files will install osync to /usr/bin instead of /usr/local/bin in order to enforce good practices.

osync will install itself to /usr/local/bin and an example configuration file will be installed to /etc/osync.

osync needs to run with bash shell. Using any other shell will most probably result in errors. If bash is not your default shell, you may invoke it using

$ bash osync.sh [options]

On *BSD and BusyBox, be sure to have bash installed.

If you can't install osync, you may just copy osync.sh where you needed and run it from there.

Arch Linux packages are available at https://aur.archlinux.org/packages/osync/ (thanks to Shadowigor, https://github.com/shadowigor).

Upgrade from previous configuration files

Since osync v1.1 the config file format has changed in semantics and adds new config options. Also, master is now called initiator and slave is now called target. osync v1.3 also added multiple new configuration options.

You can upgrade all v1.0x-v1.3-dev config files by running the upgrade script

$ ./upgrade-v1.0x-v1.3x.sh /etc/osync/your-config-file.conf

The script will backup your config file, update it's content and try to connect to initiator and target replicas to update the state dir.

Usage

Osync can work in 3 modes:

1. 🚀 Quick sync mode
2. ⚙️ Configuration file mode
3. 🔎 Monitor mode

Note

Please use double quotes as path delimiters. Do not use escaped characters in path names.

🚀 Quick sync mode

Quick sync mode is convenient to do fast syncs between some directories. However, the configuration file mode gives much more functionality.

# osync.sh --initiator="/path/to/dir1" --target="/path/to/remote dir2"
# osync.sh --initiator="/path/to/another dir" --target="ssh://[email protected]:22//path/to/dir2" --rsakey=/home/user/.ssh/id_rsa_private_key_example.com

Quick sync with minimal options

In order to run osync the quickest (without transferring file attributes, without softdeletion, without prior space checks and without remote connectivity checks, you may use the following:

# MINIMUM_SPACE=0 PRESERVE_ACL=no PRESERVE_XATTR=no SOFT_DELETE_DAYS=0 CONFLICT_BACKUP_DAYS=0 REMOTE_HOST_PING=no osync.sh --initiator="/path/to/another dir" --target="ssh://[email protected]:22//path/to/dir2" --rsakey=/home/user/.ssh/id_rsa_private_key_example.com

All the settings described here may also be configured in the conf file.

Summary mode

osync will output only file changes and errors with the following:

# osync.sh --initiator="/path/to/dir1" --target="/path/to/dir" --summary --errors-only --no-prefix

This also works in configuration file mode.

⚙️ Configuration file mode

You'll have to customize the sync.conf file according to your needs.

If you intend to sync a remote directory, osync will need a pair of private/public RSA keys to perform remote SSH connections. Also, running sync as superuser requires to configure the /etc/sudoers file.

Tip

Read the example configuration file for documentation about remote sync setups.

Once you've customized a sync.conf file, you may run osync with the following test run:

# osync.sh /path/to/your.conf --dry

If everything went well, you may run the actual configuration with:

# osync.sh /path/to/your.conf

To display which files and attrs are actually synchronized and which files are to be soft deleted / are in conflict, use --verbose (you may mix it with --silent to output verbose input only in the log files):

# osync.sh /path/to/your.conf --verbose

Use --no-maxtime to disable execution time checks, which is usefull for big initial sync tasks that might take long time. Next runs should then only propagate changes and take much less time:

# osync.sh /path/to/your.conf --no-maxtime

Once you're confident about your first runs, you may add osync as a cron task like the following in /etc/crontab which would run osync every 30 minutes:

*/30 * * * * root /usr/local/bin/osync.sh /etc/osync/my_sync.conf --silent

Please note that this syntax works for RedHat/CentOS. On Debian you might want to remove the username (i.e. root) in order to make the crontab entry work.

Batch mode

You may want to sequentially run multiple sync sets between the same servers. In that case, osync-batch.sh is a nice tool that will run every osync conf file, and, if a task fails, run it again if there's still some time left.

To run all .conf files found in /etc/osync, and retry 3 times every configuration that fails if the whole sequential run took less than 2 hours, use:

# osync-batch.sh --path=/etc/osync --max-retries=3 --max-exec-time=7200

Having multiple conf files can then be run in a single cron command like

00 00 * * * root /usr/local/bin/osync-batch.sh --path=/etc/osync --silent

🔎 Monitor mode

Note

Monitoring changes requires inotifywait command (inotify-tools package for most Linux distributions). BSD, macOS and Windows are not yet supported for this operation mode, unless you find an inotify-tool package on these OSes.

Monitor mode will perform a sync upon file operations on initiator replica. This can be a drawback on functionality versus scheduled mode because this mode only launches a sync task if there are file modifications on the initiator replica, without being able to monitor the target replica. Target replica changes are only synced when initiator replica changes occur, or when a given amount of time (600 seconds by default) passed without any changes on initiator replica.

This mode can also be launched as a daemon with an init script. Please read the documentation for more info.

To use this mode, use --on-changes:

# osync.sh /etc/osync/my_sync.conf --on-changes

To run this mode as a system service with the osync-srv script, you can run the install.sh script (which should work in most cases) or copy the files by hand:

• osync.sh to /usr/bin/local
• sync.conf to /etc/osync
• For InitV, osync-srv to /etc/init.d
• For systemd, [email protected] to /usr/lib/systemd/system
• For OpenRC, osync-srv-openrc to /etc/init.d/osync-srv-openrc

For InitV (any configuration file found in /etc/osync will create an osync daemon instance when service is launched on initV):

$ service osync-srv start
$ chkconfig osync-srv on

For systemd, launch service (one service per config file to launch) with:

$ systemctl start [email protected]
$ systemctl enable [email protected]

For OpenRC (user contrib), launch service (one service per config file to launch) with:

$ rc-update add osync-srv.configfile default

Security enhancements

Remote SSH connection security can be improved by limiting what hostnames may connect, disabling some SSH options and using ssh filter. Please read full documentation in order to configure ssh filter.

Contributions

All kind of contribs are welcome.

When submitting a PR, please be sure to modify files in dev directory (dev/n_osync.sh, dev/ofunctions.sh, dev/common_install.sh etc) as most of the main files are generated via merge.sh. When testing your contribs, generate files via merge.sh or use bootstrap.sh which generates a temporary version of n_osync.sh with all includes.

Unit tests are run by travis on every PR, but you may also run them manually which adds some tests that travis can't do, via dev/tests/run_tests.sh. SSH port can be changed on the fly via environment variable SSH_PORT, e.g.:

# SSH_PORT=2222 dev/tests/run_tests.sh

Consider reading CODING_CONVENTIONS.TXT before submitting a patch.

Troubleshooting

You may find osync's logs in /var/log/osync.[INSTANCE_ID].log (or current directory if /var/log is not writable). Additionnaly, you can use the --verbose flag see to what actions are going on.

When opening an issue, please post the corresponding log files. Also, you may run osync with _DEBUG option in order to have more precise logs, e.g.:

# _DEBUG=yes ./osync.sh /path/to/conf

Uninstalling

The installer script also has an uninstall mode that will keep configuration files. Use it with

$ ./install.sh --remove

Author

Feel free to open an issue on GitHub or mail me for support in my spare time :) Orsiris de Jong | [email protected]

osync's Issues

Replace all master / slave code and semantics

No more master / slave.
Only multimaster code (basically the same, it's about naming essentially).

Fully share codebase with obackup

Improve code sharing between both programs and add code checks.

enchancement?: detect folder renames

E.g.

• rename a folder on your local machine (the master)
• run a sync

Both locations end up correctly up to date, but what actually happens is that the new folder name and all its contents are transferred to the slave (i.e. the central server), and then the old folder is deleted. It would be nice, especially for larger folders, to detect deleted folders in the presence of new folders, looking at the contents and simply renaming if appropriate.

Tree functions can be slow

Tree functions execute piped commands (grep, awk) on local system when launched on remote slave which can cause more bandwith usage

Line 374: syntax error near unexpected token `)'

I'm receiving this error when run osync on Linux Mint 17:

/usr/local/bin/osync.sh: line 374: syntax error near unexpected token )' /usr/local/bin/osync.sh: line 374: "SSH")'

Regards,

Bill

[1.1.3] Install script error when executed from another directory

• GNU bash, version 4.3.30(1)-release (x86_64-pc-linux-gnu)
• Debian Version 8.6

Downloaded and extracted from
https://github.com/newtork/group-domain/archive/v1.1.3.tar.gz

Error, when outside directory:

root@debian:~# ./osync-1.1.3/install.sh
Created directory [/etc/osync].
cp: cannot stat â./osync.shâ: No such file or directory
Cannot copy osync.sh to [/usr/local/bin]. Make sure to run install script in the directory containing all other files.
Also make sure you have permissions to write to [/usr/local/bin].

No problem, when inside directory:

root@debian:~/osync-1.1.3# ./install.sh
[...]
osync installed. Use with /usr/local/bin/osync
[...]

osync changes folder owners to root

Dear programmers,

First things first, thanks for creating and making osync. It does has some nice functions superior to other sync programs.

However, I encountered a problem while testing my set-up: osync changed the owner of my files to root. This is annoying as other programs, like R running on my data lose the ability to work with the data due to missing rights.

Here is my setup: I have one mega folder named "projects". I have three machines, a Ubuntu desktop for computational power, a nas for backup and server sharing purposes and a laptop for remote access to my data.

My main folder "projects" is on my Ubuntu machine in my home directory. osync is tasked via crontab to synchronise to /mnt/nas which itself is connected to the nas server. From there I can mount it on my laptop via vpn.
It does sound complicated and cumbersome but it suits my needs as I always have two physical copies on two different machines, have a my data on my Ubuntu machine (main computer) without the need to transfer data through the network and I can access it through my laptop without any interference. osync is perfect in that regard, that I bidirectional sync the data between my laptop (via nas) and my Ubuntu machine.
I hope that makes sense to you. If you have recommendations to make it easier, they are more than welcome!

In any case, I believe the problem is that I have the nas mounted in /mnt/nas from where I osync to my home.

It would immensely helpful if you could assist me with that!

Best wishes,
Ruben

add support for termux (android)

Termux is a terminal emulator and Linux environment bringing powerful terminal access to Android.

Please add support for installing osync with install.sh in a termux envirionment on Android. There is a documentation available at https://termux.com/linux.html

• Common folders such as /bin, /usr/, /var and /etc does not exist.
• Termux installs its packages in a folder exposed through the $PREFIX environment variable (with e.g. binaries in $PREFIX/bin, and configuration in $PREFIX/etc).
• Shared libraries are installed in $PREFIX/lib, which are available from binaries due to Termux setting the $LD_LIBRARY_PATH environment variable. These may clash with Android system binaries in /system/bin, which may force LD_LIBRARY_PATH to be cleared before running system binaries.
• Besides the file system being different, Termux is running as a single-user system without root - each Android app is running as its own Linux user, so running commands inside Termux may not interfere with other installed applications.

Reassign function arguments

Function arguments get called $1, $2, etc in bash which is unreadable.
Have them reassigned to variables and add comments when needed (ex: does the path include a trailing slash ?)

function Whatever
{
echo "$1/$2"
}

becomes

function Whatever
{
// Does not contain and ending slash
MASTER_STATE_DIR="$1"
// Does contain an ending slash
FILE_TO_PROCEED="$2"

echo "$MASTER_STATE_DIR/$FILE_TO_PROCEED"

}

Improve daemon

Instead of writing a "helper" daemon for the slave, write a daemon that can discuss with other daemons to know who's syncing (and set a topology).

Busybox Bionic Support

ie. Cyanogen OS (and other Android OS's) support.

osync already syncs to Android, but when it comes to cleaning up on Busybox Bionic shells, osync starts to fail.
I've encountered 2 issues preventing the cleanup from working on BBB: ctime and xargs.

ctime can be worked around by replacing the flag with mtime (which is probably preferable anyway?), and xargs can be replaced with a "while read" loop.

eg.
$FIND_CMD "$replica_deletion_path/" -type f -ctime +$change_time -print0 | xargs -0 -I {} echo "Will delete file {}" > "$RUN_DIR/osync.$FUNCNAME.$SCRIPT_PID"
Logger "Command output:\n$(cat $RUN_DIR/osync.$FUNCNAME.$SCRIPT_PID)" "NOTICE"
$FIND_CMD "$replica_deletion_path/" -type d -empty -ctime +$change_time -print0 | xargs -0 -I {} echo "Will delete directory {}" > "$RUN_DIR/osync.$FUNCNAME.$SCRIPT_PID"
Logger "Command output:\n$(cat $RUN_DIR/osync.$FUNCNAME.$SCRIPT_PID)" "NOTICE"

could be replaced with:
$FIND_CMD "$replica_deletion_path/" -type f -mtime +$change_time -print0 | while read foo; do Logger "Command output:\nWill delete file $foo" "NOTICE"; done
$FIND_CMD "$replica_deletion_path/" -type d -empty -mtime +$change_time -print0 | while read foo; do Logger "Command output:\nWill delete directory $foo" "NOTICE"; done

As shown in the example, this would also clean up the "ugly xarg hacks" used for logging command output.

systemd service file

Could you provide the service file for systemd intestead of init.d?

Investigate busybox support

Busybox ps -p -o state= command can't work.
-o state= is supposed to deal with zombie and interruptible sleep states. Does this apply to busybox ?
ps -p $pid must be changed to ps | grep $pid ?

Special note for termux:
Add installer note (don't use installer on busybox / termux, directly use osync.sh)
Add note for termux-fix-shebang

Add readonly capability to multimaster

With multimaster, there must also be a readonly option where a a readonly master only replicates fiels from other masters.

Smb/cifs mount permission issue.

Hello there,

I've some strange permission problem on a smb/cifs mount drive
Slave drive is a smb mount drive which have uid an gid force.

I think there is something need to be done on the permission...

Thank You

Configuration and log below

https://gist.github.com/dcmwai/ae2f9d00258267db29ec

Update unit tests with more cases

installing osync doesn't copy sync.conf to /etc/osync

I installed osync and then went to edit /etc/osync/sync.conf, only to find it didn't exist. Examining the install.sh code revealed this:

function CopyExampleFiles {
if [ -f "./sync.conf.example" ]; then
cp "./sync.conf.example" "/etc/$PROGRAM/sync.conf.example"
fi

if [ -f "./host_backup.conf.example" ]; then
    cp "./host_backup.conf.example" "/etc/$PROGRAM/host_backup.conf.example"
fi

if [ -f "./exlude.list.example" ]; then
    cp "./exclude.list.example" "/etc/$PROGRAM"
fi

if [ -f "./snapshot.conf.example" ]; then
    cp "./snapshot.conf.example" "/etc/$PROGRAM/snapshot.conf.example"
fi

}

There is no 'sync.conf.example' file in the tarball, just 'sync.conf'
The only one that is there is 'exclude.list.example'

This is on version 1.1.1

Integrate a realpath emulation

For systems lacking readlink / realpath, integrate the following emulation
https://github.com/mkropat/sh-realpath

Permissions and ownership not correctly synched and suggested solution

Hello.
There is the following situation where osync doesn't correctly synchronize permissions and ACLs:

• one files is correctly synchronized
• slave modify ACL only (or ownership or other permissions)
• master run the synchronization (with PRESERVE_ACL=yes) and osync wrongly copy the ACL from master to slave, and not the contrary

Problem is that rsync compare the mtime of source and destination files to decide which one is more recent, but modificatrion of ACL and other permissions only modify the ctime, so the two files still have the same mtime and rsync synchronize the files in the direction run first.

My proposition is to run a similar script before the first "sync_update":

#!/bin/bash
scriptname="$(basename "$(test -L "$0" && readlink "$0" || echo "$0")")"
local_path="$1"
remote_ip="$2"
remote_path="$3"
RSYNC_ARGS="$4"

[ -z "$RSYNC_ARGS" ] && RSYNC_ARGS="-rlptgoD -A -X -z -E"
RSYNC_ARGS="$(<<<"$RSYNC_ARGS" sed -e 's/--checksum[[:space:]]*//')"

local_path="${local_path%%/}"
remote_path="${remote_path%%/}"
tmp_file="/tmp/${scriptname}_$RANDOM"

/usr/bin/rsync --rsync-path="rsync" $RSYNC_ARGS -ui -e "/usr/bin/ssh -C -i /root/.ssh/id_rsa -p 22" --exclude ".osync_workdir" \
    --exclude-from="$local_path/.osync_workdir/state/master-deleted-list-sync_test" \
    --exclude-from="$local_path/.osync_workdir/state/slave-deleted-list-sync_test" \
    "$local_path/" root@$remote_ip:"$remote_path/" --dry-run | \
    grep -Ev "^[^ ]*(c|s|t)[^ ]* " | \
    grep -E "^[^ ]*(p|o|g|a)[^ ]* " | \
    sed -e 's/^[^ ]* //' >"$tmp_file"

cd "$local_path"
cat "$tmp_file" | xargs -I {} stat -c '%n|%Z|%Y' "{}" | sort >"${tmp_file}.1"

cat "$tmp_file" | /usr/bin/ssh -C -i /root/.ssh/id_rsa -p 22 root@$remote_ip "cd \"$remote_path\" ; xargs -I {} stat -c '%n|%Z|%Y' \"{}\"" | sort >"${tmp_file}.2"

join -j 1 -t \| -o 1.1,1.2,2.2 "${tmp_file}.1" "${tmp_file}.2" | awk -F'|' '{if ($2 < $3) print $1}' | \
    /usr/bin/rsync --rsync-path="rsync" $RSYNC_ARGS -ui -e "/usr/bin/ssh -C -i /root/.ssh/id_rsa -p 22" \
    root@$remote_ip:"$remote_path/" "$local_path/" --files-from=-

rm -f "$tmp_file"*

Probably this solution is not the most efficient, because run a one more rsync command (also if with --dry-run option) before the synchronization of file content inside sync_update.
An alternative could be to use "find" to report the list of files with mtime and ctime, but maybe it's more difficult to give support for RSYNC_EXCLUDE_* options.
Another alternative, only possible including this code inside osync.sh script, is to write the output unmodified of rsync --dry-run to file, then run two rsync with " --files-from=-", one for acls, one for real copy, conveniently grepping the input file.

Stefano

File-based locking is not atomic when locking remote directory

I've run into this when I set MIN_WAIT=5 and used two initiator scripts running in --on-changes mode. :)

As I looked into code, the file-based locking is implemented using first trying to read remote lock (trying to delete it if it is stale lock), and then creating it using "echo pid | tee lock ".

With this approach two racing initiator scripts can both ensure that there is no lock at target, but then simply write to it and continue to work. The lock file will contain PID of the last script who performed "tee lock".

This issue is not very important if the clocks on all three parties (two racing initiators and one target) are well synchronized. But if they not, and if the user edits more than one copy, then the final version after osync`ing may have inconsistent changes.

It is wise to implement locking in atomic fashion. As I briefly googled, this is done using making directories instead of lock files although this is not secure if the lock dir is situated on a NFS file system.

idea: "lite mode" for webdesigners who 90/95% of the time just update/append existing files

This idea means that I found myself constantly editing several files without the need osync to check local/remote deletion lists and free hdd space etc ...

...so if we could wrap each such operation in some if-clause ( = if ( ! $lite_mode ) ) we could skip those on purpose.

The cmdline option will be boolean - if it is given, its true, we skip, and if not given - we act as normal. Also a new var could be added in the config file.

This would be relatively easy I think to implement and will speed up drastically the program, making the UX for webdevs signifficantly better, as we do frequent little edits...

Questions/Opinions?

Wrong ACL on directory when set on remote host

I discovered that there is an issue with ACL on a directory when you change it from a remote host.
Version:1.1.5
Steps to reproduce:

1. Create directory "testdir" and a file inside named "testfile". Doesn't matter whether on the master or slave.
2. Run osync
3. On the remote host: write something into the file and save
4. On the remote host: setfacl -m u:someuser:rwx -R testdir
5. Run osync
6. Check ACL on sync master.
   When i do that i can see that the directory has lost the ACL setting from the remote host. Although the file has the correct acl with someuser.

PS: Great software, saved me some headache :-)

/usr/local/bin/osync.sh: line 778: [: -ne: unary operator expected

Run with commit 39c60d5
Actual line is:

....
 if [ $log_time -ne $SECONDS ]; then
...

Probably it should be:

 if [ $log_time -lt $SECONDS ]; then

Connect to rsyncd (not only to rsync via SSH) on the remote slave

Reading the docs I understand that osync only uses SSH to connect to a remote server. However, rsync also offers the option to run as a server (--daemon option). I prefer this option, as here the other devices syncing to the server do not need login credentials, but only permissions to access the server.

The rsync usage seems to be pretty much the same, only the URI is different from SSH connections.

Would it be possible to adapt osync to this kind of rsync server?

function GetRemoteOS silently failing

Getting the following error when trying to run osync with a remote slave

TIME: 241 - Max soft execution time exceeded for task.
/usr/local/bin/osync: line 405: [: 241: unary operator expected
/usr/local/bin/osync: line 405: [: 242: unary operator expected
/usr/local/bin/osync: line 405: [: 243: unary operator expected

and on and on.

The problem appears to be on line 332 of osync.sh:
eval "$SSH_CMD "uname -spio" > $RUN_DIR/osync_remote_os_$SCRIPT_PID 2>&1"
should have a '&' at the end in order for the next line
child_pid=$!
to succeed.

Adding the ampersand seems to fix the issue.

Add ssh password option

Sometimes its not possible to use ssh keys, ie certain chrooted/rssh based accounts

I am aware of the disadvantages of using passwords for ssh logins, however most of them are mitigated with 64 random character passwords and fail2ban/denyhosts for login bruteforce protection.

I would like to add the functionality and submit a pull request, would you be willing to include it ?

I do not want to spent time forking, coding, testing and then you are not willing to have the functionality for some obscure reason.

Thanks

Proposal: Change the format of build numbers to YYYY-MM-DD.NN

Hello,

This is really a detail, but it is maybe better to deal with this minor issue before the pending release:

The fact is that the timestamp format currently used is "DDMMYYYYNN", which is not that readable, and does not yield increasing build numbers.
So I would suggest to use something like "YYYY-MM-DD.NN" (that is, using ISO 8601 for the date), so the corresponding strings would be increasing with respect to the lexicographic order.

To sum up,

PROGRAM_BUILD=2606201501

would become

PROGRAM_BUILD=2015-06-26.01

Do you see any inconvenient to perform such a change?

(By the way, the date itself can be obtained in this format by using the date -I or date +%F shell command.)

Best regards.

Meta Issue for v1.2

v1.2 can go final when projects Compatibility, Compliance and Documentation don't have any TODO / IN PROGRESS notes left.

Some remote checks aren't done with sudo permissions

$SSH_CMD if [ -w somedir ]; then sudo do something; fi

if [ -w somedir] will fail without proper sudo
Need to fix to
$SSH_CMD $COMMAND_SUDO bash -c 'if [ -w somedir ]; then do something; fi'

Mistakes in removing deleted files at remote targets

Hello!

I first knew about osync several days ago. I started to use it since v.1.1RC and noticed that my files are not deleted and synced back next syncs. I found where the problem is and fixed it today. I didn't know that you were working on #53 touching the same place until I tried to rebase my work against master. But now I finished it and I'd like to share it. I'll send a pull request when I figure how. :)

why wait for this ssh to finnish?

hi,
I am working on making osync sync faster.

Thus I have a question:

why this around line 2020 in osync.sh:

---

$SSH_CMD ERROR_ALERT=0 sync_on_changes=$sync_on_changes _SILENT=$_SILENT _DEBUG=$_DEBUG _DRYRUN=$_DRYRUN _VERBOSE=$_VERBOSE COMMAND_SUDO=$COMMAND_SUDO FILE_LIST="$(EscapeSpaces "$TARGET_STATE_DIR/$deleted_list_file")" REPLICA_DIR="$(EscapeSpaces "$replica_dir")" DELETE_DIR="$(EscapeSpaces "$deletion_dir")" FAILED_DELETE_LIST="$(EscapeSpaces "${TARGET[1]}${TARGET[3]}/$deleted_failed_list_file")" 'bash -s' << 'ENDSSH' > "$RUN_DIR/$PROGRAM.remote_deletion.$SCRIPT_PID" 2>&1 &

---

runs with & = run in background?

LAter on line 2124 there is a "stupid"
sleep 5

---

I think if we remove the &, then we can remove the sleep too?
Where am I mistaken?

PS. most probably there is sth that made you do that, I think with some ssh options we can un-make you do that waiting ...

Cheers.

Cannot move /var/lib/samba/sysvol/ramls/folder2 to deletion directory

Hello there,

It seem that osync will have issue if the folder remove
/var/lib/samba/sysvol/ramls/folder2

is already exist on the backup directory before (possible because of it was created and remove later)

Thank You

ACLS typo

First of all, thanks for this great script!
I found a small typo in sync.conf causing the ACLS flag to not be switched correctly.
line 52 from:
PRESERVE_ACL=no
to:
PRESERVE_ACLS=no
fixes the issue.
Thanks =D

Use inotify to watch for deleted files instead of directory compare between two runs ?

osync makes directory comparaisons to know which file has been deleted, in async runs.
It could use inotifywait instead to create lists of moved / deleted files before launching rsync.
This would also resolve move detection for synchronization.

Drawback, it only works if the osync daemon is running. If not, legagy file deletion method can be used.

Change detection on target not works!

demon mode on target cant monitor changes , if it possible provide a example for master master replication on tow servers.

turn off soft delete?

Hi,
I use osync to synchronize my two web server's web directories together, and I have run into a problem:
When osync syncs, it does a soft delete, which means that any files my web server (running wordpress, nginx, mysql) deletes, don't get deleted on the other end.
Could it be possible to have an option to turn off soft deletes (if there is one, I'm just not seeing it).
Thanks,
-Michael.

TASK: Have osync tested on multiple platforms before release v1.00

Tests must be done on:

CentOS 5 / 6 / 7 (maybe write a systemd file too)
Debian 7 / Mint 17
FreeBSD 10.1
MacOS X (i don't have any macs, testers welcome).

Trying to set remote attributes although params are set to "no"

Hello,

I've started using osync recently and I am trying to sync 2 large folders containing images grouped in some subfolders.

Initiator folders are owned by root:root and are 777.
Target folders are owned by syncuser:www-data and are 777.

Attributes are set to no:
PRESERVE_ACL=no
PRESERVE_XATTR=no

The connection is done through ssh + key from a dedicated user.

Initiator and target folders are the same = /mnt/storage/tezaur-amanet/

From the log:

TIME: 0 - -------------------------------------------------------------
TIME: 0 - Tue Aug 9 12:53:32 EEST 2016 - osync 1.1.1 script begin.
TIME: 0 - -------------------------------------------------------------
TIME: 0 - Sync task [appsrv] launched as syncuser@appsrv (PID 66787)
TIME: 19 - Checking minimum disk space in [/mnt/storage/tezaur-amanet/].
TIME: 19 - Checking minimum disk space on target [/mnt/storage/tezaur-amanet/].
TIME: 46 - Starting synchronization task.
TIME: 46 - WARN:WARNING: Trying to resume aborted osync execution on 2016-08-09 12:50:50.353990066 +0300 at task [noresume]. [0] previous tries.
TIME: 46 - Creating initiator replica file list [/mnt/storage/tezaur-amanet/].
TIME: 173 - Creating target replica file list [/mnt/storage/tezaur-amanet/].
TIME: 384 - Creating target replica file list [/mnt/storage/tezaur-amanet/].
TIME: 226 - Creating initiator replica deleted file list.
TIME: 437 - Creating initiator replica deleted file list.
TIME: 227 - Creating target replica deleted file list.
TIME: 438 - Creating target replica deleted file list.
TIME: 227 - Getting list of files that need updates.
TIME: 438 - Getting list of files that need updates.
TIME: 244 - Getting ctimes for pending files on initiator.
TIME: 244 - Getting ctimes for pending files on target.
TIME: 455 - Getting ctimes for pending files on initiator.
TIME: 455 - Getting ctimes for pending files on target.
TIME: 250 - Updating file attributes on initiator not required
TIME: 250 - Updating target replica.
TIME: 461 - Updating file attributes on initiator not required
TIME: 461 - Updating target replica.
TIME: 319 - Updating target replica succeded.
TIME: 319 - Updating initiator replica.
TIME: 530 - Updating target replica succeded.
TIME: 530 - Updating initiator replica.
TIME: 543 - CRITICAL:Updating initiator replica failed. Stopping execution.
TIME: 332 - CRITICAL:Updating initiator replica failed. Stopping execution.
TIME: 336 - Rsync output:
rsync: chgrp "/mnt/storage/tezaur-amanet/contracts" failed: Operation not permitted (1)
rsync: chgrp "/mnt/storage/tezaur-amanet/contracts/C0212000203" failed: Operation not permitted (1)
rsync: chgrp "/mnt/storage/tezaur-amanet/contracts/C0212000203/C0212000203_01.jpg" failed: Operation not permitted (1)
rsync: chgrp "/mnt/storage/tezaur-amanet/contracts/C0212000204" failed: Operation not permitted (1)

Is this working correctly? If so, what is the config problem?

Thank you,
Ionut Dumitru

Refactor master / slave code

dev-1.1 is a good moment to refactor most osync functions from:

function x {
local_master_code;
local_slave_code;
remote_slave_code;
}
to

function _x_localcode { stuff }
function _x_remotecode { remote_stuff }
function x {
_x_localcode;
_x_remotecode;
}

Install script has a problem

root@kam02:/usr/src/osync# cat install.sh

!/usr/bin/env bash

SCRIPT_BUILD=2015070201

Osync daemon install script

Tested on RHEL / CentOS 6 & 7

Please adapt this to fit your distro needs

if [ "$(whoami)" != "root" ]
then
echo "Must be run as root."
exit 1
fi

mkdir /etc/osync
cp ./sync.conf /etc/osync/sync.conf.example
cp ./exclude.list.example /etc/osync
cp ./osync.sh /usr/local/bin
cp ./osync-batch.sh /usr/local/bin
cp ./ssh_filter.sh /usr/local/bin
cp ./osync-srv /etc/init.d
chmod 755 /usr/local/bin/osync.sh
chmod 755 /usr/local/bin/osync-batch.sh
chmod 755 /usr/local/bin/ssh_filter.sh
chown root:root /usr/local/bin/ssh_filter.sh
chmod 755 /usr/local/bin/osync-srv

chmod: no se puede acceder a «/usr/local/bin/osync-srv»: No existe el fichero o el directorio

i think something is missed

BR

Upgrade path from v1.0x to v1.1

Need an upgrade path here so existing installs can easily upgrade.

Console output

I would like to run my osync's with only errors and a list of transferred files output to the console. I haven't been able to figure out how to do this... Possible feature request(?)

hosts from ~/.ssh/config

Hello, I kave tried to sync test folder using git version of osync.sh:

osync.sh --initiator=./ --target=ssh://dev/~development/Symlink --rsakey=~/.ssh/id_rsa

And it failed:

TIME: 0 - \e[93mThis is an unstable dev build. Please use with caution.\e[0m
TIME: 0 - -------------------------------------------------------------
TIME: 0 -  Mon Nov  7 13:29:34 CET 2016 - osync 1.2-beta2 script begin.
TIME: 0 - -------------------------------------------------------------
TIME: 0 - Sync task [quicksync_task] launched as ak@osx-andrius (PID 64898)
TIME: 10 - \e[93mCannot ping [dev]. Return code [1].\e[0m
TIME: 16 - \e[91mCannot Get remote OS type.\e[0m
TIME: 16 - \e[41mRunning on remote OS failed. Please report to the author if the OS is not supported.\e[0m
TIME: 16 - \e[41mRemote OS said:
\e[0mssion denied (publickey).
TIME: 16 - \e[91mosync finished with errors.\e[0m

ssh dev works as exepected. Mac OS

Synchronization left both master and slave in broken states. Rollback?

I did a master folder to slave folder sync earlier today. Didn't bother to check if everything went all.

Came back this evening, and majority of both the master and slave files and folders are missing. How do I recover, or go back to a previous state?

The folders involved completed with success, so absolutely no idea why the missing syncs. I checked the deleted folders in the hidden osync dir, and under the deleted folder, I see most of the missing files, with some folders with empty content.

Is there anyway I can recover my stuff?

TASK: Update documentation for release 1.00

Issues that I have run into using Osync

I have been using osync for about 8 months and I wanted to share with you some of the issues that I have run into. I recently updated to version 1.1 and I have noticed all of these behaviors in the new version. I am running osync on an up to date Ubuntu server syncing to another ubuntu server over ssh.

1. If a file has been deleted on both the initiator and the receiving folder between syncs, osync will produce an error that it cannot move the file to the deleted folder and it will continue to produce the error for every sync into the future. The only way I have been able to fix it is to delete the .osync_workdir on both and start over.
2. Sometimes osync will get stuck on a file. I'm not sure why this happens, but I will delete the file in the initiating folder, osync will copy it back from the secondary folder. I'll delete it again, osync will copy it back again, this will go on forever until I delete the .osync_workdir on both and start over. I think I have observed this most often with files that were generated by or moved by scripts.
3. Sometimes osync runs two instances of itself. I just noticed this today but it's happened twice. Osync gets interrupted, reinitiates, then the next osync that tries to start sees the lock file and halts, but the next osync after that (while the first one is still going) ignores the lock file and starts a new instance of rsync. Sorry if I have worded that in a confusing way, I'll attach a log file. If I send a sigterm to the second rsync pid then the second running osync cancels and everything goes back to normal. (There are extra complications in the log file due to this because when the second instance is launched it maxes out my upload bandwidth and then produces connection errors.)
4. If a directory contains "[" or "]" it cannot be excluded by pattern in the conf file, it can only be excluded by being put in an exclude list file. I have had this problem with rsync itself, maybe something to inform people about instead of an actual fix.
5. I had some AppleScript applications in my sync folder (executable directories with extension .app). When I would delete them in the initiator folder osync would not delete them in the receiving folder.

Sorry GitHub is not letting me upload the log file. Let me know if it would be helpful to see it (or any other info) and maybe I can send it to you some other way.

Correct PR14 with readlink -f

readlink -f doesn't exist on MacOSX

how to install on windows

Microsoft Windows is supported via MSYS or Cygwin.

I tried to install osync with the provided install.sh but the shell keeps telling me Must be run as root.. I ran the shell as Administrator, so I'm not sure what to do here.

Any suggestions?

#71 meta issue

question: what are the main differences between osync and unison

Great work with osync, I last checked your progress at around commit 450, now you are at 950+ :)
I always wondered how are unison and osync different - what do they do differently - give me the big picture?