datto/dattobd

kernel module for taking block-level snapshots and incremental backups of Linux block devices

dattobd's Issues

reload-snapshot on already tracked device results in oops

Steps to reproduce:

dbdctl setup-snapshot /dev/loop7 /tmp/dattobd/ext4/cow.snap 1
umount /dev/loop7
dbdctl reload-snapshot /dev/loop7 /cow.snap 0
mount /dev/loop7 /tmp/dattobd/ext4

Important that the minor for reload-snapshot is less than the minor for setup-snapshot.

Jun 14 12:33:04 fedora-23-srv-std kernel: datto: detected block device umount: /tmp/dattobd/ext4 : 0
Jun 14 12:33:04 fedora-23-srv-std kernel: datto: block device umount detected for device 1
Jun 14 12:33:04 fedora-23-srv-std kernel: datto: stopping cow thread
Jun 14 12:33:04 fedora-23-srv-std kernel: datto: destroying cow manager
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: post umount check succeeded
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: umount returned: 0
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: ioctl command received: 1075876098
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: received reload snap ioctl - 0 : /dev/loop7 : /cow.snap
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: allocating device struct
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: initializing tracer
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: minor range = 0 - 1
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: detected block device mount: /dev/loop7 -> /tmp/dattobd/ext4 : 0
Jun 14 12:33:05 fedora-23-srv-std kernel: EXT4-fs (loop7): mounted filesystem with ordered data mode. Opts: (null)
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: block device mount detected for unverified device 0
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: finding block device
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: checking block device is not already being traced
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: block device is already being traced: -22
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: error setting up base block device: -22
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: freeing base block device
Jun 14 12:33:05 fedora-23-srv-std kernel: datto: error transitioning snapshot tracer to active state: -22
Jun 14 12:33:05 fedora-23-srv-std kernel: BUG: unable to handle kernel paging request at ffffed3592708420
Jun 14 12:33:05 fedora-23-srv-std kernel: IP: [<ffffffff8121a447>] kfree+0x57/0x170
Jun 14 12:33:05 fedora-23-srv-std kernel: PGD 0 
Jun 14 12:33:05 fedora-23-srv-std kernel: Oops: 0000 [#1] SMP 
Jun 14 12:33:05 fedora-23-srv-std kernel: Modules linked in: dattobd(OE) loop ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfn
Jun 14 12:33:05 fedora-23-srv-std kernel: CPU: 1 PID: 1844 Comm: mount Tainted: G           OE   4.5.5-201.fc23.x86_64 #1
Jun 14 12:33:05 fedora-23-srv-std kernel: Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jun 14 12:33:05 fedora-23-srv-std kernel: task: ffff8800de04db80 ti: ffff8800dc0fc000 task.ti: ffff8800dc0fc000
Jun 14 12:33:05 fedora-23-srv-std kernel: RIP: 0010:[<ffffffff8121a447>]  [<ffffffff8121a447>] kfree+0x57/0x170
Jun 14 12:33:05 fedora-23-srv-std kernel: RSP: 0018:ffff8800dc0ffe18  EFLAGS: 00010286
Jun 14 12:33:05 fedora-23-srv-std kernel: RAX: 000077ff80000000 RBX: 000055649c210250 RCX: 000000000000e193
Jun 14 12:33:05 fedora-23-srv-std kernel: RDX: 000000000000e192 RSI: ffff88011fd1a1c0 RDI: ffffea0000000000
Jun 14 12:33:05 fedora-23-srv-std kernel: RBP: ffff8800dc0ffe30 R08: 000000000001a1c0 R09: ffffffffa0408ac4
Jun 14 12:33:05 fedora-23-srv-std kernel: R10: ffffed3592708400 R11: 00000000ffffffea R12: 0000000000000000
Jun 14 12:33:05 fedora-23-srv-std kernel: R13: ffffffffa0408851 R14: ffff88000252ca70 R15: ffff8800de02d200
Jun 14 12:33:05 fedora-23-srv-std kernel: FS:  00007ffadc86c840(0000) GS:ffff88011fd00000(0000) knlGS:0000000000000000
Jun 14 12:33:05 fedora-23-srv-std kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 14 12:33:05 fedora-23-srv-std kernel: CR2: ffffed3592708420 CR3: 00000000023dd000 CR4: 00000000000406e0
Jun 14 12:33:05 fedora-23-srv-std kernel: Stack:
Jun 14 12:33:05 fedora-23-srv-std kernel:  000055649c210250 0000000000000000 ffff88000252c990 ffff8800dc0ffea0
Jun 14 12:33:05 fedora-23-srv-std kernel:  ffffffffa0408851 ffffffff811af587 0000000000000010 ffff8800dc0ffeb0
Jun 14 12:33:05 fedora-23-srv-std kernel:  ffffffeadc0ffe68 0000000000000000 000055649c210250 0000000032172a4b
Jun 14 12:33:05 fedora-23-srv-std kernel: Call Trace:
Jun 14 12:33:05 fedora-23-srv-std kernel:  [<ffffffffa0408851>] auto_transition_active+0x171/0x4a0 [dattobd]
Jun 14 12:33:05 fedora-23-srv-std kernel:  [<ffffffff811af587>] ? printk+0x57/0x73
Jun 14 12:33:05 fedora-23-srv-std kernel:  [<ffffffffa0408e07>] handle_bdev_mount_event+0x287/0x2c0 [dattobd]
Jun 14 12:33:05 fedora-23-srv-std kernel:  [<ffffffffa040908a>] mount_hook+0xaa/0x150 [dattobd]
Jun 14 12:33:05 fedora-23-srv-std kernel:  [<ffffffff817c90ee>] entry_SYSCALL_64_fastpath+0x12/0x71
Jun 14 12:33:05 fedora-23-srv-std kernel: Code: 00 00 00 80 ff 77 00 00 48 bf 00 00 00 00 00 ea ff ff 49 01 da 48 0f 42 05 d7 6b 9f 00 49 01 c
Jun 14 12:33:05 fedora-23-srv-std kernel: RIP  [<ffffffff8121a447>] kfree+0x57/0x170
Jun 14 12:33:05 fedora-23-srv-std kernel:  RSP <ffff8800dc0ffe18>
Jun 14 12:33:05 fedora-23-srv-std kernel: CR2: ffffed3592708420
Jun 14 12:33:05 fedora-23-srv-std kernel: ---[ end trace ddee4fa1131710ed ]---

'failed to locate system call table, persistence disabled' on load of kernel module

Debian Testing here, kernel 4.3.3 64bit - on insmoding the v0.8.14 module I get the following kernel message:

datto: failed to locate system call table, persistence disabled

This doesn't appear to disturb my ability to use the created snapshot device file though (haven't played with incremental mode, I'm just interested in getting a full snapshot, backing it up and then rmmoding).

Request for some kind of status sub-command

Loving the potential of dattobd now it's open source. One thing I think would be a great enhancement would be a dbdctl sub-command to output the status (with a choice between nicely formatted, and parseable) to allow both humans and scripts to find out the status of dattobd devices, e.g. their existence, the devices being snapshotted, the paths of COW files, maybe even an idea of the amount of changed data. Is this something in the pipeline?

Cannot retrieve repository metadata

https://cpkg.datto.com/datto-rpm/EnterpriseLinux/6/i386/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
To address this issue please refer to the below knowledge base article

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.

Error: Cannot retrieve repository metadata (repomd.xml) for repository: datto-rpm. Please verify its path and try again

Getting this after I install repo via

yum localinstall https://cpkg.datto.com/datto-rpm/repoconfig/datto-el-rpm-release-$(rpm -E %rhel)-latest.noarch.rpm

Did I do something wrong?

Fails to build on 4.7.0

Building on Arch 4.7.0-1

make -C src
make[1]: Entering directory '/home/tim/Projects/dattobd/src'
if [ ! -f kernel-config.h ] || tail -1 kernel-config.h | grep -qv '#endif'; then ./genconfig.sh 4.7.0-1-ARCH; fi;
generating configurations
make[2]: Entering directory '/home/tim/Projects/dattobd/src/configure-tests/feature-tests'
make[3]: Entering directory '/usr/lib/modules/4.7.0-1-ARCH/build'
make[3]: Leaving directory '/usr/lib/modules/4.7.0-1-ARCH/build'
make[2]: Leaving directory '/home/tim/Projects/dattobd/src/configure-tests/feature-tests'
performing configure test: HAVE_BDOPS_OPEN_INODE - not present
performing configure test: HAVE_BDOPS_OPEN_INT - not present
performing configure test: HAVE_BIO_BI_POOL - present
performing configure test: HAVE_BIO_BI_REMAINING - not present
performing configure test: HAVE_BIO_ENDIO_1 - present
performing configure test: HAVE_BIO_LIST - present
performing configure test: HAVE_BLKDEV_GET_BY_PATH - present
performing configure test: HAVE_BLKDEV_PUT_1 - not present
performing configure test: HAVE_BLK_QUEUE_MAX_SECTORS - not present
performing configure test: HAVE_BLK_SET_STACKING_LIMITS - present
performing configure test: HAVE_BVEC_ITER - present
performing configure test: HAVE___DENTRY_PATH - not present
performing configure test: HAVE_DENTRY_PATH_RAW - present
performing configure test: HAVE_D_UNLINKED - present
performing configure test: HAVE_FILE_INODE - present
performing configure test: HAVE_GENHD_FL_NO_PART_SCAN - present
performing configure test: HAVE_IOPS_FALLOCATE - not present
performing configure test: HAVE_KERN_PATH - present
performing configure test: HAVE_MAKE_REQUEST_FN_INT - not present
performing configure test: HAVE_MAKE_REQUEST_FN_VOID - not present
performing configure test: HAVE_MERGE_BVEC_FN - not present
performing configure test: HAVE_NOOP_LLSEEK - present
performing configure test: HAVE_NO_QUEUE_LIMITS - not present
performing configure test: HAVE_NOTIFY_CHANGE_2 - not present
performing configure test: HAVE_PART_NR_SECTS_READ - present
performing configure test: HAVE_QUEUE_LIMITS_MAX_HW_SEGMENTS - not present
performing configure test: HAVE_SB_START_WRITE - present
performing configure test: HAVE_SUBMIT_BIO_WAIT - present
performing configure test: HAVE_SYS_OLDUMOUNT - not present
performing configure test: HAVE_TASK_STRUCT_TASK_WORKS_CB_HEAD - present
performing configure test: HAVE_TASK_STRUCT_TASK_WORKS_HLIST - not present
performing configure test: HAVE_THAW_BDEV_INT - present
performing configure test: HAVE_VFS_FALLOCATE - present
performing configure test: HAVE_VFS_UNLINK_2 - not present
performing configure test: HAVE_VZALLOC - present
make[2]: Entering directory '/home/tim/Projects/dattobd/src/configure-tests/feature-tests'
make[3]: Entering directory '/usr/lib/modules/4.7.0-1-ARCH/build'
  CLEAN   /home/tim/Projects/dattobd/src/configure-tests/feature-tests/.tmp_versions
  CLEAN   /home/tim/Projects/dattobd/src/configure-tests/feature-tests/Module.symvers
make[3]: Leaving directory '/usr/lib/modules/4.7.0-1-ARCH/build'
make[2]: Leaving directory '/home/tim/Projects/dattobd/src/configure-tests/feature-tests'
performing sys_mount lookup
grep: /boot/System.map-4.7.0-1-ARCH: No such file or directory
performing sys_umount lookup
grep: /boot/System.map-4.7.0-1-ARCH: No such file or directory
performing sys_oldumount lookup
grep: /boot/System.map-4.7.0-1-ARCH: No such file or directory
make -I/usr/include -C /lib/modules/4.7.0-1-ARCH/build SUBDIRS=/home/tim/Projects/dattobd/src modules
make[2]: Entering directory '/usr/lib/modules/4.7.0-1-ARCH/build'
  CC [M]  /home/tim/Projects/dattobd/src/dattobd.o
/home/tim/Projects/dattobd/src/dattobd.c: In function ‘do_truncate2’:
/home/tim/Projects/dattobd/src/dattobd.c:808:29: error: ‘struct inode’ has no member named ‘i_mutex’; did you mean ‘i_mode’?
  mutex_lock(&dentry->d_inode->i_mutex);
                             ^~
/home/tim/Projects/dattobd/src/dattobd.c:815:31: error: ‘struct inode’ has no member named ‘i_mutex’; did you mean ‘i_mode’?
  mutex_unlock(&dentry->d_inode->i_mutex);
                               ^~
/home/tim/Projects/dattobd/src/dattobd.c: In function ‘__file_unlink’:
/home/tim/Projects/dattobd/src/dattobd.c:965:23: error: ‘struct inode’ has no member named ‘i_mutex’; did you mean ‘i_mode’?
  mutex_lock(&dir_inode->i_mutex);
                       ^~
/home/tim/Projects/dattobd/src/dattobd.c:979:25: error: ‘struct inode’ has no member named ‘i_mutex’; did you mean ‘i_mode’?
  mutex_unlock(&dir_inode->i_mutex);
                         ^~
make[3]: *** [scripts/Makefile.build:296: /home/tim/Projects/dattobd/src/dattobd.o] Error 1
make[2]: *** [Makefile:1457: _module_/home/tim/Projects/dattobd/src] Error 2
make[2]: Leaving directory '/usr/lib/modules/4.7.0-1-ARCH/build'
make[1]: *** [Makefile:9: default] Error 2
make[1]: Leaving directory '/home/tim/Projects/dattobd/src'
make: *** [Makefile:12: driver] Error 2

struct inode: 4.6, 4.7

NULL pointer dereference on module exit if MAX_SNAP_DEVICES=0

insmod dattobd.ko MAX_SNAP_DEVICES=0
rmmod dattobd

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa0454672>] agent_exit+0x82/0xa10 [dattobd]

(gdb) list *(agent_exit+0x82)
0x56a2 is in agent_exit (/home/tim/Projects/dattobd/src/dattobd.c:4888).
4883            remove_proc_entry(INFO_PROC_FILE, NULL);
4884
4885            //destroy our snap devices
4886            LOG_DEBUG("destroying snap devices");
4887            if(snap_devices){
4888                    tracer_for_each(dev, i){
4889                            if(dev){
4890                                    LOG_DEBUG("destroying minor - %d", i);
4891                                    tracer_destroy(dev);
4892                            }

Reference count issue on 2.6.18; can't remove module

Fedora Core 6 (2.6.18-1.2798.fc6)
Module fails to unload after creating a snapshot.

Adding the follow to track reference count:

LOG_DEBUG("%u: ref count = %d", __LINE__, (int)module_refcount(THIS_MODULE));

CentOS 5: /dev/disk/by-uuid/ entries disappear on transition-to-incremental

Snapshotting a volume that have a corresponding entry in /dev/disk/by-{label,uuid} and then transitioning it to incremental state causes the entry to disappear.

> ls /dev/disk/by-uuid/
10711ec6-ac6d-4094-adb9-2d8a2d809e57  6d6325c4-88ba-48a4-b802-5d2b96fc9a75

> dbdctl setup-snapshot /dev/sda1 /boot/cow.snap 1
> ls /dev/disk/by-uuid/
10711ec6-ac6d-4094-adb9-2d8a2d809e57  6d6325c4-88ba-48a4-b802-5d2b96fc9a75

> dbdctl transition-to-incremental 1
> ls /dev/disk/by-uuid/
10711ec6-ac6d-4094-adb9-2d8a2d809e57

dattobd_proc_show: NULL pointer dereference

Removing the COW file while a volume is unmounted (such as by reformatting), then remounting, will put the driver in the error state. Attempting to read from the proc file will result in a NULL pointer dereference.

[  169.107938] datto: detected block device mount: /dev/sdb1 -> /mnt : 0
[  169.109125] XFS (sdb1): Mounting V5 Filesystem
[  169.157629] XFS (sdb1): Ending clean mount
[  169.159965] datto: block device mount detected for dormant device 1
[  169.159968] datto: reopening the cow manager with file '/mnt/cow.snap'
[  169.159969] datto: reopening cow file
[  169.160080] datto: error creating/opening file '/mnt/cow.snap' - 0: -2
[  169.160163] datto: error opening specified file: -2
[  169.160202] datto: error reopening cow manager: -2
[  169.160242] datto: error setting up cow manager: -2
[  169.160281] datto: destroying cow manager
[  169.160285] datto: error transitioning tracer to active state: -2
[  169.160328] datto: mount returned: 0
[  174.258217] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[  174.258274] IP: [<ffffffffc0342753>] dattobd_proc_show+0x1a3/0x200 [dattobd]
[  174.258323] PGD 0 
[  174.258358] Oops: 0000 [#1] SMP 
[  174.258397] Modules linked in: dattobd(OE) nls_utf8 hfsplus hfs isofs vfat fat xfs libcrc32c snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ppdev snd_timer snd joydev soundcore parport_pc parport acpi_cpufreq i2c_piix4 video tpm_tis tpm nfsd auth_rpcgss nfs_acl lockd grace sunrpc e1000 serio_raw fjes [last unloaded: dattobd]
[  174.258633] CPU: 1 PID: 1029 Comm: cat Tainted: G           OE   4.6.4-201.fc23.x86_64 #1
[  174.258680] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[  174.258764] task: ffff8800ddd6db80 ti: ffff8800de114000 task.ti: ffff8800de114000
[  174.258810] RIP: 0010:[<ffffffffc0342753>]  [<ffffffffc0342753>] dattobd_proc_show+0x1a3/0x200 [dattobd]
[  174.258862] RSP: 0018:ffff8800de117d98  EFLAGS: 00010246
[  174.258911] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000002
[  174.258956] RDX: 0000000000000002 RSI: ffffffffc034aa88 RDI: ffff8800020a6f00
[  174.259001] RBP: ffff8800de117db0 R08: 0000000000001000 R09: 0000000000000000
[  174.259045] R10: 0000000000000003 R11: ffff88011a1830a0 R12: ffff88003e104600
[  174.259090] R13: ffff8800020a6f00 R14: ffff8800020a6f00 R15: ffff88003e665d88
[  174.259135] FS:  00007facd9c49700(0000) GS:ffff88011fd00000(0000) knlGS:0000000000000000
[  174.259182] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  174.259223] CR2: 0000000000000028 CR3: 0000000002084000 CR4: 00000000000406e0
[  174.259270] Stack:
[  174.259304]  0000000000000000 ffff8800de117f18 ffff880119106300 ffff8800de117e20
[  174.259353]  ffffffff8126bd7c 0000000000000025 00007facd9c27000 ffff8800020a6f40
[  174.260471]  0000000000020000 0000000000000000 0000000000000001 00000000cb08e01a
[  174.260951] Call Trace:
[  174.261422]  [<ffffffff8126bd7c>] seq_read+0x2cc/0x390
[  174.262737]  [<ffffffff812b7902>] proc_reg_read+0x42/0x70
[  174.264139]  [<ffffffff81246a17>] __vfs_read+0x37/0x100
[  174.264673]  [<ffffffff81357a50>] ? security_file_permission+0xa0/0xc0
[  174.265252]  [<ffffffff81247a26>] vfs_read+0x86/0x130
[  174.265780]  [<ffffffff81248f15>] SyS_read+0x55/0xc0
[  174.266274]  [<ffffffff817da3b2>] entry_SYSCALL_64_fastpath+0x1a/0xa4
[  174.266772] Code: ff ff 49 8b 54 24 10 48 c7 c6 71 aa 34 c0 4c 89 ef 31 db 48 c1 e2 14 e8 0c 98 f2 c0 49 8b 44 24 58 48 c7 c6 88 aa 34 c0 4c 89 ef <48> 8b 50 28 e8 f4 97 f2 c0 48 c7 c6 9c aa 34 c0 4c 89 ef e8 e5 
[  174.267927] RIP  [<ffffffffc0342753>] dattobd_proc_show+0x1a3/0x200 [dattobd]
[  174.268471]  RSP <ffff8800de117d98>
[  174.269084] CR2: 0000000000000028
[  174.269901] ---[ end trace 90beeb7c25c67fbb ]---

(gdb) list *(dattobd_proc_show+0x1a3)
0x1783 is in dattobd_proc_show (/root/dattobd/src/dattobd.c:4330).
4325            seq_printf(m, "\t\t\t\"block_device\": \"%s\",\n", dev->sd_bdev_path);
4326            seq_printf(m, "\t\t\t\"max_cache\": %lu,\n", (dev->sd_cache_size)? dev->sd_cache_size : COW_MAX_MEMORY_DEFAULT);
4327    
4328            if(!test_bit(UNVERIFIED, &dev->sd_state)){
4329                seq_printf(m, "\t\t\t\"fallocate\": %llu,\n", ((unsigned long long)dev->sd_falloc_size) * 1024 * 1024);
4330                seq_printf(m, "\t\t\t\"seq_id\": %llu,\n", (unsigned long long)dev->sd_cow->seqid);
4331    
4332                seq_printf(m, "\t\t\t\"uuid\": \"");
4333                for(i = 0; i < COW_UUID_SIZE; i++){
4334                    seq_printf(m, "%02x", dev->sd_cow->uuid[i]);

cow file not left in clean state

Seen on Fedora 24, Fedora 25. On reboot, dattobd fails to reload the incremental state for the root volume when remounted as R/W. Other volumes work as expected.

Reload script that gets run:

#!/bin/sh

echo "Reloading incremental - 2 ..." > /dev/kmsg
/usr/bin/dbdctl reload-incremental /dev/disk/by-uuid/92963e08-947c-450c-852c-655ecfc4aaf8 /.datto1 2

dmesg:

Nov 23 10:49:29 segv-fed-24 kernel: EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: (null)
Nov 23 10:49:30 segv-fed-24 unknown: datto looking for datto_reload
Nov 23 10:49:30 segv-fed-24 unknown: datto found /sbin/datto_reload
Nov 23 10:49:30 segv-fed-24 unknown: Reloading incremental - 0 ...
Nov 23 10:49:30 segv-fed-24 unknown: Reloading incremental - 1 ...
Nov 23 10:49:30 segv-fed-24 unknown: Reloading incremental - 2 ...
Nov 23 10:49:31 segv-fed-24 kernel: EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: (null)
Nov 23 10:49:31 segv-fed-24 kernel: random: crng init done
Nov 23 10:49:36 segv-fed-24 systemd-journald[171]: Received SIGTERM from PID 1 (systemd).
Nov 23 10:49:36 segv-fed-24 kernel: systemd: 16 output lines suppressed due to ratelimiting
Nov 23 10:49:36 segv-fed-24 kernel: SELinux: 32768 avtab hash slots, 106590 rules.
Nov 23 10:49:36 segv-fed-24 kernel: SELinux: 32768 avtab hash slots, 106590 rules.
Nov 23 10:49:36 segv-fed-24 kernel: SELinux:  8 users, 14 roles, 5044 types, 305 bools, 1 sens, 1024 cats
Nov 23 10:49:36 segv-fed-24 kernel: SELinux:  94 classes, 106590 rules
Nov 23 10:49:36 segv-fed-24 kernel: SELinux:  Permission validate_trans in class security not defined in policy.
Nov 23 10:49:36 segv-fed-24 kernel: SELinux:  Permission module_load in class system not defined in policy.
Nov 23 10:49:36 segv-fed-24 kernel: SELinux: the above unknown classes and permissions will be allowed
Nov 23 10:49:36 segv-fed-24 kernel: SELinux:  Completing initialization.
Nov 23 10:49:36 segv-fed-24 kernel: SELinux:  Setting up existing superblocks.
Nov 23 10:49:36 segv-fed-24 systemd[1]: Successfully loaded SELinux policy in 157.580ms.
Nov 23 10:49:36 segv-fed-24 systemd[1]: Relabelled /dev and /run in 12.764ms.
Nov 23 10:49:36 segv-fed-24 kernel: audit_printk_skb: 108 callbacks suppressed
Nov 23 10:49:36 segv-fed-24 kernel: audit: type=1130 audit(1479916173.738:47): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system
Nov 23 10:49:36 segv-fed-24 kernel: audit: type=1131 audit(1479916173.738:48): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system
Nov 23 10:49:36 segv-fed-24 kernel: audit: type=1130 audit(1479916173.738:49): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system
Nov 23 10:49:36 segv-fed-24 kernel: audit: type=1131 audit(1479916173.738:50): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system
Nov 23 10:49:36 segv-fed-24 kernel: audit: type=1130 audit(1479916173.738:51): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system
Nov 23 10:49:36 segv-fed-24 kernel: audit: type=1131 audit(1479916173.741:52): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system
Nov 23 10:49:36 segv-fed-24 kernel: audit: type=1130 audit(1479916174.128:53): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system
Nov 23 10:49:36 segv-fed-24 kernel: audit: type=1305 audit(1479916174.205:54): audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=syste
Nov 23 10:49:36 segv-fed-24 kernel: audit: type=1130 audit(1479916174.237:55): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system
Nov 23 10:49:36 segv-fed-24 kernel: EXT4-fs (sda3): re-mounted. Opts: (null)
Nov 23 10:49:36 segv-fed-24 kernel: datto: cow file not left in clean state: 0: -22
Nov 23 10:49:36 segv-fed-24 kernel: datto: error opening cow manager header: -22
Nov 23 10:49:36 segv-fed-24 kernel: datto: error during cow manager initialization: -22
Nov 23 10:49:36 segv-fed-24 kernel: datto: error setting up cow manager: -22
Nov 23 10:49:36 segv-fed-24 kernel: datto: error transitioning incremental to active state: -22

/proc/datto-info:

{
	"version": "0.9.11",
	"devices": [
		{
			"minor": 0,
			"cow_file": "/.datto1",
			"block_device": "/dev/sda1",
			"max_cache": 314572800,
			"fallocate": 20971520,
			"seq_id": 190,
			"uuid": "9cadf9e31dc34d4cbfb6e28f46802206",
			"state": 2
		},
		{
			"minor": 1,
			"cow_file": "/.datto1",
			"block_device": "/dev/sda2",
			"max_cache": 314572800,
			"fallocate": 56623104,
			"seq_id": 190,
			"uuid": "a6d3b766e9f743deb642b73621d8a0e3",
			"state": 2
		},
		{
			"minor": 2,
			"cow_file": "/.datto1",
			"block_device": "/dev/disk/by-uuid/92963e08-947c-450c-852c-655ecfc4aaf8",
			"max_cache": 314572800,
			"error": -22,
			"state": 4
		}
	]
}

dattobd module not loaded at boot time on CentOS 6

The dkms-dattobd rpm built for CentOS (el6) includes a file /etc/modules-load.d/dattobd.conf. This is in fact the correct location for specifying modules to be loaded at boot time for CentOS 7 (see this link). In contrast, for CentOS 6, the file should be at /etc/sysconfig/modules (see this link).

Just noticed this after the first reboot since I installed dattobd a few weeks ago. Everything working fine up until that point, many thanks!

Contents of the new file would be something like:

#!/bin/sh

if [ ! -c /dev/datto-ctl ] ; then
	exec /sbin/modprobe dattobd >/dev/null 2>&1
fi

Incremental files not updated in update-img

I performed the following steps in ubuntu 14.04 with kernel version 4.2.0-27-generic 64 bit which has single ext4 partition /dev/sda1

1. dbdctl setup-snapshot /dev/sda1 /.datto 0
2. dd if=/dev/datto0 of=/backups/sda1-bkp bs=1M
3. created files on base file system (total 5018847 bytes)
4. dbdctl transition-to-incremental 0
5. created file on base file system (total 30 bytes)
6. dbdctl transition-to-snapshot /.datto1 0
7. ran update-img /dev/datto0 /.datto /backups/sda1-bkp showed 109 blocks changes (109*4096 = 446464 bytes)

After incremental backup i mounted /backups/sda1-bkp to /mnt, New files created in step 3 and 5 not visible. Checked hexedit and saw that block offsets reported by update-img are updated but looks like the actual data was not copied.

Is it a problem with this specific kernel or something we are doing wrong.

remove the limit that cow file must exist on the volume that will be snapshotted

some use cases involves taking block-level snapshots and incremental backups on a raw device(without filesystem), such as a kvm virtual machine, it's disk image directly stored on a raw disk. Or a oracle database, it use raw disk to store data.
If we use different volume to store cow file, the base device that will be snapshotted will have less i/o pressure, as cow file will be write to other device. But now, dattobd only supports cow file exist on the volume that will be snapshotted. maybe we could remove the limit and we can also support cow file directly stored on a another raw device to gain more flexibility.

Use parameter to enable/disable logging

Instead of having to recompile the module, create a parameter that can be set to enable/disable verbose logging.

img-merge: command not found

Could you please confirm, from where I can install img-merge command? I could not find this command in both centos and ubuntu.

tracing_mrf general protection fault

One of our VMs had the following issue.
Fedora 23 (4.5.5-201.fc23.x86_64) with dattobd 0.9.6

Jun 30 12:24:25 screenshot-fedora23workstation kernel: __get_request: dev 8:0: request aux data allocation failed, iosched may be disturbed
Jun 30 12:24:25 screenshot-fedora23workstation kernel: __get_request: dev 8:0: request aux data allocation failed, iosched may be disturbed
Jun 30 12:24:25 screenshot-fedora23workstation kernel: __get_request: dev 8:0: request aux data allocation failed, iosched may be disturbed
Jun 30 12:24:25 screenshot-fedora23workstation kernel: __get_request: dev 8:0: request aux data allocation failed, iosched may be disturbed
Jun 30 12:24:25 screenshot-fedora23workstation kernel: __get_request: dev 8:0: request aux data allocation failed, iosched may be disturbed
Jun 30 12:24:26 screenshot-fedora23workstation kernel: general protection fault: 0000 [#1] SMP 
Jun 30 12:24:26 screenshot-fedora23workstation kernel: Modules linked in: binfmt_misc fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 t
Jun 30 12:24:26 screenshot-fedora23workstation kernel: 
Jun 30 12:24:26 screenshot-fedora23workstation kernel: CPU: 0 PID: 649 Comm: jbd2/sda1-8 Tainted: G           OE   4.5.5-201.fc23.x86_64 #1
Jun 30 12:24:26 screenshot-fedora23workstation kernel: Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jun 30 12:24:26 screenshot-fedora23workstation kernel: task: ffff880035d21e80 ti: ffff88003beec000 task.ti: ffff88003beec000
Jun 30 12:24:26 screenshot-fedora23workstation kernel: RIP: 0010:[<ffffffffa00023cf>]  [<ffffffffa00023cf>] tracing_mrf+0x16f/0x730 [dattobd]
Jun 30 12:24:26 screenshot-fedora23workstation kernel: RSP: 0018:ffff88003beefb60  EFLAGS: 00010246
Jun 30 12:24:26 screenshot-fedora23workstation kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff880009bf1b00
Jun 30 12:24:26 screenshot-fedora23workstation kernel: RDX: dead000000000400 RSI: ffff88003d881000 RDI: ffff8800367f8000
Jun 30 12:24:26 screenshot-fedora23workstation kernel: RBP: ffff88003beefbc8 R08: 0000000000000000 R09: ffff880009bf1b00
Jun 30 12:24:26 screenshot-fedora23workstation kernel: R10: ffff8800367f8000 R11: 0000000000001000 R12: ffff880009bf1b88
Jun 30 12:24:26 screenshot-fedora23workstation kernel: R13: 0000000000000000 R14: 0000000000000400 R15: 0000000000000400
Jun 30 12:24:26 screenshot-fedora23workstation kernel: FS:  0000000000000000(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
Jun 30 12:24:26 screenshot-fedora23workstation kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jun 30 12:24:26 screenshot-fedora23workstation kernel: CR2: 00007f4d03f5f860 CR3: 0000000035e2d000 CR4: 00000000000406f0
Jun 30 12:24:26 screenshot-fedora23workstation kernel: Stack:
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  ffffffff8139df19 0000000002011200 ffff88003d8a8ea0 ffff88003d8a8ed0
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  ffff88003beefbd8 ffff88000332a650 ffff8800217b2c00 00000000780eb28f
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  ffff880009bf1b00 00000000ffffffff ffff8800367f8000 ffff880035d21e80
Jun 30 12:24:26 screenshot-fedora23workstation kernel: Call Trace:
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff8139df19>] ? generic_make_request_checks+0x229/0x520
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff813a1032>] generic_make_request+0xf2/0x1d0
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff813a1186>] submit_bio+0x76/0x170
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff8132cabf>] ? jbd2_journal_write_metadata_buffer+0x29f/0x430
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff812787ff>] submit_bh_wbc+0x12f/0x160
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff81278862>] submit_bh+0x12/0x20
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff81324048>] jbd2_journal_commit_transaction+0x668/0x18b0
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff81020dc9>] ? sched_clock+0x9/0x10
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff817c8b6e>] ? _raw_spin_unlock_irqrestore+0xe/0x10
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff8110fd7e>] ? try_to_del_timer_sync+0x5e/0x90
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff81328e1a>] kjournald2+0xca/0x250
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff810e87d0>] ? wake_atomic_t_function+0x70/0x70
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff81328d50>] ? commit_timeout+0x10/0x10
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff810c4728>] kthread+0xd8/0xf0
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff810c4650>] ? kthread_worker_fn+0x170/0x170
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff817c944f>] ret_from_fork+0x3f/0x70
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff810c4650>] ? kthread_worker_fn+0x170/0x170
Jun 30 12:24:26 screenshot-fedora23workstation kernel: Code: 48 89 d8 48 c1 e0 04 4c 01 e0 44 8b 70 08 48 8b 00 48 8b 50 08 45 29 ee 45 39 fe 
Jun 30 12:24:26 screenshot-fedora23workstation kernel: RIP  [<ffffffffa00023cf>] tracing_mrf+0x16f/0x730 [dattobd]
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  RSP <ffff88003beefb60>
Jun 30 12:24:26 screenshot-fedora23workstation kernel: ---[ end trace 650a5568da2b4f99 ]---
Jun 30 12:24:26 screenshot-fedora23workstation kernel: ------------[ cut here ]------------
Jun 30 12:24:26 screenshot-fedora23workstation kernel: WARNING: CPU: 0 PID: 649 at kernel/exit.c:659 do_exit+0x62/0xb50()
Jun 30 12:24:26 screenshot-fedora23workstation kernel: Modules linked in: binfmt_misc fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 t
Jun 30 12:24:26 screenshot-fedora23workstation kernel: 
Jun 30 12:24:26 screenshot-fedora23workstation kernel: CPU: 0 PID: 649 Comm: jbd2/sda1-8 Tainted: G      D    OE   4.5.5-201.fc23.x86_64 #1
Jun 30 12:24:26 screenshot-fedora23workstation kernel: Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  0000000000000286 00000000780eb28f ffff88003beef968 ffffffff813d08de
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  0000000000000000 ffffffff81a7221d ffff88003beef9a0 ffffffff810a5e22
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  ffff880035d21e80 000000000000000b ffff88003beefab8 0000000000000000
Jun 30 12:24:26 screenshot-fedora23workstation kernel: Call Trace:
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff813d08de>] dump_stack+0x63/0x85
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff810a5e22>] warn_slowpath_common+0x82/0xc0
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff810a5f6a>] warn_slowpath_null+0x1a/0x20
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff810a8712>] do_exit+0x62/0xb50
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff8101a97a>] oops_end+0x9a/0xd0
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff8101ae3b>] die+0x4b/0x70
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff81018229>] do_general_protection+0xc9/0x150
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff817cb428>] general_protection+0x28/0x30
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffffa00023cf>] ? tracing_mrf+0x16f/0x730 [dattobd]
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff8139df19>] ? generic_make_request_checks+0x229/0x520
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff813a1032>] generic_make_request+0xf2/0x1d0
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff813a1186>] submit_bio+0x76/0x170
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff8132cabf>] ? jbd2_journal_write_metadata_buffer+0x29f/0x430
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff812787ff>] submit_bh_wbc+0x12f/0x160
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff81278862>] submit_bh+0x12/0x20
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff81324048>] jbd2_journal_commit_transaction+0x668/0x18b0
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff81020dc9>] ? sched_clock+0x9/0x10
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff817c8b6e>] ? _raw_spin_unlock_irqrestore+0xe/0x10
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff8110fd7e>] ? try_to_del_timer_sync+0x5e/0x90
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff81328e1a>] kjournald2+0xca/0x250
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff810e87d0>] ? wake_atomic_t_function+0x70/0x70
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff81328d50>] ? commit_timeout+0x10/0x10
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff810c4728>] kthread+0xd8/0xf0
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff810c4650>] ? kthread_worker_fn+0x170/0x170
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff817c944f>] ret_from_fork+0x3f/0x70
Jun 30 12:24:26 screenshot-fedora23workstation kernel:  [<ffffffff810c4650>] ? kthread_worker_fn+0x170/0x170
Jun 30 12:24:26 screenshot-fedora23workstation kernel: ---[ end trace 650a5568da2b4f9a ]---

(gdb) list *(tracing_mrf+0x16f)
0x23ff is in tracing_mrf (/usr/src/dattobd-0.9.6/dattobd.c:1893).
1888    /****************************BIO HELPER FUNCTIONS*****************************/
1889    
1890    static inline struct inode *page_get_inode(struct page *pg){
1891        if(!pg->mapping) return NULL;
1892        if((unsigned long)pg->mapping & PAGE_MAPPING_ANON) return NULL;
1893        if(!pg->mapping->host) return NULL;
1894        return pg->mapping->host;
1895    }
1896    
1897    static int bio_needs_cow(struct bio *bio, struct inode *inode){

Ubuntu RAID0 kernel oops

On Ubuntu 12.04.5 and Ubuntu 14.04.3, using a RAID0 setup will lead to a kernel oops on the 2nd backup (via Datto Linux Agent).

dmesg output with debugging enabled: https://gist.github.com/crawfxrd/4539164f680bd11204b3

root@tahr-srv-raid:~# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda           8:0    0    16G  0 disk  
└─sda1        8:1    0    16G  0 part  
  └─md0       9:0    0    32G  0 raid0 
    ├─md0p1 259:0    0 486.3M  0 md    /boot
    ├─md0p2 259:1    0  14.9G  0 md    /
    └─md0p3 259:2    0  16.6G  0 md    /home
sdb           8:16   0    16G  0 disk  
└─sdb1        8:17   0    16G  0 part  
  └─md0       9:0    0    32G  0 raid0 
    ├─md0p1 259:0    0 486.3M  0 md    /boot
    ├─md0p2 259:1    0  14.9G  0 md    /
    └─md0p3 259:2    0  16.6G  0 md    /home
sr0          11:0    1  1024M  0 rom

Driver not loading snapshots/Incrementals after reboot

Built and installed 0.9.6 version of dattobd and have run it in snapshot mode and then rebooted the machine. on reboot cat /proc/datto-info shows no devices. There is no /dev/datto* devices. Checked that the module was loading in initramfs. Attempting to reload-snapshot also does not work as the device is not created under /dev. Is there any configuration we are missing to make this persist across reboots and have it start tracking changes before the volumes are mounted as read write

Thanks
Shiva

Remounting NTFS volume hangs file system

Attempting to unmount and remount an NTFS (ntfs-3g) file system with an active snapshot causes mount to hang.

Enabling debug logging shows it hangs after "reopening cow file" in cow_reopen.
Adding debugging lines to file_open shows it never returns from filp_open.

fallocate -l 2G ntfs.img
mkntfs -F -f ntfs.img
losetup /dev/loop0 ntfs.img
ntfs-3g -o rw /dev/loop0 /mnt
dbdctl setup-snapshot /dev/loop0 /mnt/cow.snap 1
dbdctl transition-to-incremental 1
umount /mnt
ntfs-3g -o rw,debug /dev/loop0 /mnt

ntfs-3g does not produce any output, even with the debug option.

RPM repository doesn't work

Hi,
when trying to download this: https://cpkg.datto.com/datto-rpm/EnterpriseLinux/7/x86_64/datto-el-rpm-release-1.0.0-1.noarch.rpm, I get a 404 error.

CentOS 5 does not use full UUID

CentOS release 5.11 (Final)
kernel=2.6.18-412.el5

{
    "minor": 0,
    "cow_file": "/.datto",
    "block_device": "/dev/sda1",
    "max_cache": 314572800,
    "fallocate": 11534336,
    "seq_id": 1,
    "uuid": "e0d57fba7f8547018000000000000000",
    "state": 2
}

Dattobd not working on Ubuntu Server 16.04.1

I have recently installed dattobd to be used together with UrBackup to make full file backups with a snapshot. We have 2 Ubuntu servers, both the same version. One server works fine and the other one won't even start a backup.

I have only performed a fresh install of UrBackup (choosing dattobd) and a fresh installation of dattobd according to the install file.

sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 29FF164C
echo "deb https://cpkg.datto.com/repositories $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/datto-linux-agent.list
sudo apt-get update
sudo apt-get install dattobd-dkms dattobd-utils

The following is output from dmesg on both servers:

Server A (working):

root@VVICT23:~# dmesg | grep datto
[    3.400489] dattobd: module verification failed: signature and/or required key missing - tainting kernel
[ 1478.065268] datto: illegal to perform setup while unmounted: -22
[ 1478.065389] datto: error during setup ioctl handler: -22
[95846.519870] datto: illegal to perform setup while unmounted: -22
[95846.519954] datto: error during setup ioctl handler: -22

I don't know what the last 4 errors mean (or if they even matter) but the backups are working fine.

Server B (not working):

root@VVICT14:~# dmesg | grep datto
[    3.144832] dattobd: module verification failed: signature and/or required key missing - tainting kernel
[    3.193966] IP: [<ffffffffc016811d>] agent_init+0x11d/0x1000 [dattobd]
[    3.193984] Modules linked in: dattobd(OE+) autofs4 dm_mirror dm_region_hash dm_log hid_generic usbhid hid psmouse ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pata_acpi ahci libahci floppy
[    3.193992] RIP: 0010:[<ffffffffc016811d>]  [<ffffffffc016811d>] agent_init+0x11d/0x1000 [dattobd]
[    3.194111] RIP  [<ffffffffc016811d>] agent_init+0x11d/0x1000 [dattobd]

I have absolutely no idea what these messages mean but they don't look good to me.

Here is also the log output from UrBackup on the working server:

Level Time Message
Info 24/08/16 10:10 Starting full file backup...
Info 24/08/16 10:10 Snapshotting device /dev/mapper/TetraSSH--vg-root via dattobd...
Info 24/08/16 10:10 Using /dev/datto0...
Info 24/08/16 10:10 Mounting /dev/mapper/wsnap-66a48c9364f7486614c74c9493f6bf4c0557d4788d32b663...
Info 24/08/16 10:10 Indexing of "root" done. 16245 filesystem lookups 0 db lookups and 0 db updates
Info 24/08/16 10:10 VVICT23: Loading file list...
Info 24/08/16 10:10 VVICT23: Started loading files...
Info 24/08/16 10:15 Waiting for file transfers...
Info 24/08/16 10:18 Waiting for file hashing and copying threads...
Info 24/08/16 10:18 Saving file metadata...
Info 24/08/16 10:18 Writing new file list...
Info 24/08/16 10:18 All metadata was present
Info 24/08/16 10:18 Transferred 1.49572 GB - Average speed: 26.3852 MBit/s
Info 24/08/16 10:18 Time taken for backing up client VVICT23: 8m 40s
Info 24/08/16 10:18 Backup succeeded

Snapshotting and open files

How do you deal with files whose writes are in process and a snapshot command is issued.

I had faced this issues, since if the file whose writes are still in process, the final ext3/4 headers of the new blocks numbers and new size are written after all the block level data writes are completed. If snapshot command is issued in between it will corrupt the file system if its not handled correctly.

setup_snapshot: NULL pointer dereference

setup_snapshot(0, bdev, NULL, 0, 0);

Passing NULL for the cow file results in the following:

[29142.279348] datto: ioctl command received: 1076400385
[29142.279350] datto: received setup snap ioctl - 0 : /dev/loop5 : (null)
[29142.279355] datto: allocating device struct
[29142.279356] datto: initializing tracer
[29142.279357] datto: finding block device
[29142.279358] datto: checking block device is not already being traced
[29142.279359] datto: fetching the absolute pathname for the base device
[29142.279361] datto: calculating block device size and offset
[29142.279362] datto: bdev size = 4194304, offset = 0
[29142.279363] datto: creating cow manager
[29142.279364] datto: allocating cow manager, seqid = 1
[29142.279364] datto: creating cow file
[29142.279380] BUG: unable to handle kernel NULL pointer dereference at           (null)
[29142.279880] IP: [<ffffffff813d8e00>] strlen+0x0/0x20
[29142.280336] PGD dd10e067 PUD dc0cb067 PMD 0 
[29142.280759] Oops: 0000 [#1] SMP 
[29142.281348] Modules linked in: loop dattobd(OE) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_filter ebtable_broute bridge stp llc ebtable_nat ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw iptable_security vfat fat crct10dif_pclmul snd_intel8x0 crc32_pclmul xfs libcrc32c crc32c_intel ghash_clmulni_intel snd_ac97_codec ac97_bus joydev ppdev snd_pcm snd_timer snd parport_pc i2c_piix4 parport soundcore video acpi_cpufreq tpm_tis tpm nfsd auth_rpcgss nfs_acl lockd grace sunrpc serio_raw e1000 fjes [last unloaded: dattobd]
[29142.284420] CPU: 1 PID: 16008 Comm: dbdtest Tainted: G           OE   4.5.5-201.fc23.x86_64 #1
[29142.284923] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[29142.285432] task: ffff8800dc098000 ti: ffff88003e86c000 task.ti: ffff88003e86c000
[29142.285957] RIP: 0010:[<ffffffff813d8e00>]  [<ffffffff813d8e00>] strlen+0x0/0x20
[29142.288536] RSP: 0018:ffff88003e86fcb0  EFLAGS: 00010246
[29142.289221] RAX: 0000000000000018 RBX: ffff8800dedcbe00 RCX: 0000000000000006
[29142.289855] RDX: 0000000000000000 RSI: 0000000000008242 RDI: 0000000000000000
[29142.290447] RBP: ffff88003e86fcd8 R08: 0000000000044344 R09: 0000000000000790
[29142.290989] R10: ffff880002581f80 R11: 0000000000000790 R12: 0000000000000000
[29142.291527] R13: fffffffffffffff4 R14: 0000000000000000 R15: ffff8800df2acd10
[29142.292067] FS:  00007f1620ca3700(0000) GS:ffff88011fd00000(0000) knlGS:0000000000000000
[29142.292615] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[29142.293289] CR2: 0000000000000000 CR3: 0000000002575000 CR4: 00000000000406e0
[29142.293897] Stack:
[29142.294513]  ffffffff8125011f ffff8800dedcbe00 0000000000000000 0000000000008242
[29142.295180]  0000000000000000 ffff88003e86fd00 ffffffff8123f459 ffff8800dedcbe00
[29142.295736]  ffff880002581f80 0000000000000000 ffff88003e86fd30 ffffffffa03e06e3
[29142.296312] Call Trace:
[29142.296874]  [<ffffffff8125011f>] ? getname_kernel+0x1f/0x120
[29142.297521]  [<ffffffff8123f459>] filp_open+0x19/0x60
[29142.298094]  [<ffffffffa03e06e3>] file_open+0x23/0xf0 [dattobd]
[29142.298644]  [<ffffffffa03e43e3>] __tracer_setup_cow.isra.25+0x103/0x760 [dattobd]
[29142.299199]  [<ffffffffa03e4bc7>] __ioctl_setup+0x187/0x2f0 [dattobd]
[29142.299760]  [<ffffffffa03e50ec>] ctrl_ioctl+0x3bc/0x9b0 [dattobd]
[29142.300313]  [<ffffffff81254251>] do_vfs_ioctl+0xa1/0x5a0
[29142.300882]  [<ffffffff812547c9>] SyS_ioctl+0x79/0x90
[29142.301463]  [<ffffffff817c90ee>] entry_SYSCALL_64_fastpath+0x12/0x71
[29142.302110] Code: 89 f8 48 89 e5 f6 82 00 b9 85 81 20 74 10 48 83 c0 01 0f b6 10 f6 82 00 b9 85 81 20 75 f0 5d c3 66 66 2e 0f 1f 84 00 00 00 00 00 <80> 3f 00 55 48 89 e5 74 11 48 89 f8 48 83 c0 01 80 38 00 75 f7 
[29142.303458] RIP  [<ffffffff813d8e00>] strlen+0x0/0x20
[29142.304065]  RSP <ffff88003e86fcb0>
[29142.304644] CR2: 0000000000000000
[29142.307109] ---[ end trace 57ee77d432f4a4cf ]---

Centos 7 with default xfs file system

Thought I'd try dattobd on centos 7 server running xfs, doesnt work. kicks an error.

driver returned an error performing specified action. check dmesg for more info: Invalid argument

Then via DMESG

[207102.424197] dattobd: module verification failed: signature and/or required key missing - tainting kernel
[207286.766656] datto: fallocate is not supported for this file system, falling back on writing zeros
[207286.805662] datto: file specified is not on block device specified: -22
[207286.805855] datto: error setting up cow manager: -22
[207288.386614] datto: error setting up tracer as active snapshot: -22
[207288.386664] datto: error during setup ioctl handler: -22
[207303.521472] datto: fallocate is not supported for this file system, falling back on writing zeros
[207303.559139] datto: file specified is not on block device specified: -22
[207303.559186] datto: error setting up cow manager: -22
[207304.997633] datto: error setting up tracer as active snapshot: -22
[207304.997686] datto: error during setup ioctl handler: -22

Backup issues under high I/O load

Having a snapshot on a disk with a lot of I/O will cause lockups.

(Would be resolved as part of implementing #45.)

Rearrange proc file

Change the proc file from a single JSON output to a directory, writing out each minor as a file.

e.g.,

/proc/dattobd/devices/{0..MAX_SNAP_DEVICES}

Have each minor file output a simple list of field:value pairs instead of JSON.

cowfile size limit

Hi,

I have a query about cowfile max size limit and max free space required to create a snapshot.
I created a snapshot for a 800gb drive which has 150gb free space available. After creating the snapshot a cowfile created with 97gb. I want to decrease cowfile size to as less as possible. What is the calculation i have to follow to get the max cowfile size(fallocate) to create while creating a snapshot.

I tried using a machine with 185gb data and 27 gb free space. After creating snapshot cowfile occupied entire freespace.

What is the free space required in the mountpoint to create a snapshot.

Thanks ..

Idea to use raw disk writes instead of file writes for cow

We could reap many performance benefits if we changed dattobd to use raw block operations instead of file writes. This would be good for several reasons:

• Many filesystems lock an entire file for IO, meaning we cant hope to parallelize the cow process
• Raw IO could greatly simplify the driver code since we won't need separate threads for dealing with journal writes
• The current code cannot create limits on the queue size for cow write / snapshot read operations or else a deadlock could occur. This queue can grow extremely quickly since every write to disk becomes 3 IOs during the COW processes, eating an unbound amount of memory and blocking other IOs for an unbound amount of time.

The fix would essentially involve a few parts:

1. Move to raw IO
   • Create functions equivalent to vfs_read() and vfs_write() using raw IO and the bmap() function.
   • Flagging the cow file as a swap file so that it cannot be moved.
   • Remove code related to the snap_mrf_thread since it is no longer needed now that we are not dealing with the journal.
2. Parallelization
   • Give each struct snap_device access to a threadpool to use for dispatching IOs
   • Add a mutex to the cow manager to allow multiple threads to work with it at once
   • Create an interval tree-like structure for the IOs since we still do not want concurrent overlapping reads and writes. The tree would ultimately be responsible for dispatching dependent IOs once the previous overlapping IO was finished.

Fails to build on Linux 4.10

Compile time issues:

• req_op was renamed req_opf
• READ_SYNC has been dropped.

if [ ! -f kernel-config.h ] || tail -1 kernel-config.h | grep -qv '#endif'; then ./genconfig.sh 4.10.0-0.rc7.git1.1.fc26.x86_64; fi;
make -C /lib/modules/4.10.0-0.rc7.git1.1.fc26.x86_64/build M=/root/dattobd/src modules
make[1]: Entering directory '/usr/src/kernels/4.10.0-0.rc7.git1.1.fc26.x86_64'
  CC [M]  /root/dattobd/src/dattobd.o
/root/dattobd/src/dattobd.c:254:62: warning: ‘enum req_op’ declared inside parameter list will not be visible outside of this definition or declaration
 static inline void dattobd_set_bio_ops(struct bio *bio, enum req_op op, unsigned op_flags){
                                                              ^~~~~~
/root/dattobd/src/dattobd.c:254:69: error: parameter 2 (‘op’) has incomplete type
 static inline void dattobd_set_bio_ops(struct bio *bio, enum req_op op, unsigned op_flags){
                                                                     ^~
/root/dattobd/src/dattobd.c:254:20: error: function declaration isn’t a prototype [-Werror=strict-prototypes]
 static inline void dattobd_set_bio_ops(struct bio *bio, enum req_op op, unsigned op_flags){
                    ^~~~~~~~~~~~~~~~~~~
/root/dattobd/src/dattobd.c: In function ‘snap_handle_read_bio’:
/root/dattobd/src/dattobd.c:2443:40: error: ‘READ_SYNC’ undeclared (first use in this function); did you mean ‘REQ_SYNC’?
  dattobd_set_bio_ops(bio, REQ_OP_READ, READ_SYNC);
                                        ^~~~~~~~~
                                        REQ_SYNC
/root/dattobd/src/dattobd.c:2443:40: note: each undeclared identifier is reported only once for each function it appears in
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:301: /root/dattobd/src/dattobd.o] Error 1
make[1]: *** [Makefile:1494: _module_/root/dattobd/src] Error 2
make[1]: Leaving directory '/usr/src/kernels/4.10.0-0.rc7.git1.1.fc26.x86_64'
make: *** [Makefile:10: default] Error 2

mount hook does not always work

Tested on Fedora 23 (4.5.5-201.fc23.x86_64)

Using the following script:

#!/bin/bash

BDEV="/dev/loop7"
MOUNT="/tmp/dattobd/ext4"

while true; do
    rmmod dattobd
    insmod ../src/dattobd.ko
    umount $BDEV
    mount $BDEV $MOUNT
    sleep 1
done

I sometimes see

Jun 21 13:19:08 fedora-23-srv-std kernel: datto: module init
Jun 21 13:19:08 fedora-23-srv-std kernel: datto: get major number
Jun 21 13:19:08 fedora-23-srv-std kernel: datto: allocate global device array
Jun 21 13:19:08 fedora-23-srv-std kernel: datto: registering proc file
Jun 21 13:19:08 fedora-23-srv-std kernel: datto: registering control device
Jun 21 13:19:08 fedora-23-srv-std kernel: datto: locating system call table
Jun 21 13:19:08 fedora-23-srv-std kernel: EXT4-fs (loop7): mounted filesystem with ordered data mode. Opts: (null)

Instead of (expected)

Jun 21 13:19:09 fedora-23-srv-std kernel: datto: module init
Jun 21 13:19:09 fedora-23-srv-std kernel: datto: get major number
Jun 21 13:19:09 fedora-23-srv-std kernel: datto: allocate global device array
Jun 21 13:19:09 fedora-23-srv-std kernel: datto: registering proc file
Jun 21 13:19:09 fedora-23-srv-std kernel: datto: registering control device
Jun 21 13:19:09 fedora-23-srv-std kernel: datto: locating system call table
Jun 21 13:19:09 fedora-23-srv-std kernel: datto: detected block device umount: /tmp/dattobd/ext4 : 0
Jun 21 13:19:09 fedora-23-srv-std kernel: datto: umount returned: 0
Jun 21 13:19:09 fedora-23-srv-std kernel: datto: detected block device mount: /dev/loop7 -> /tmp/dattobd/ext4 : 0
Jun 21 13:19:09 fedora-23-srv-std kernel: EXT4-fs (loop7): mounted filesystem with ordered data mode. Opts: (null)
Jun 21 13:19:09 fedora-23-srv-std kernel: datto: mount returned: 0

Kernel BUG on insmod (v0.9.9, kernel v4.7.5-1, Devuan)

I have just upgraded from kernel 4.5 to 4.7, and after building the latest dattobd in a Devuan VM (very similar to Debian Testing), insmod fails (is 'Killed') with the following BUG reported by the kernel:

Oct  2 20:17:23 devuan-testing-vm kernel: [16313.548838] BUG: unable to handle kernel paging request at ffff88007fff0000
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.548884] IP: [<ffffffffc09e2119>] agent_init+0x119/0x1000 [dattobd]
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.548904] PGD d556067 PUD d559067 PMD d55a067 PTE 0
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.548919] Oops: 0000 [#1] SMP
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.548928] Modules linked in: dattobd(OE+) aufs(OE) nf_log_ipv4 nf_log_common xt_LOG xt_conntrack nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_ipv4 nf_defrag_ipv4 iptable_raw xt_CT nf_conntrack xt_tcpudp iptable_mangle iptable_filter ip_tables x_tables appletalk ax25 ipx p8023 p8022 psnap llc bnep bluetooth rfkill cpufreq_stats cpufreq_conservative cpufreq_userspace cpufreq_powersave vboxsf(OE) binfmt_misc fuse loop lp ppdev vboxvideo(OE) joydev ttm evdev crct10dif_pclmul crc32_pclmul serio_raw pcspkr ghash_clmulni_intel drm_kms_helper parport_pc sg drm acpi_cpufreq tpm_tis vboxguest(OE) i2c_piix4 parport video battery ac tpm button ext4 ecb crc16 jbd2 mbcache btrfs crc32c_generic xor raid6_pq sr_mod cdrom sd_mod ata_generic hid_generic usbhid hid crc32c_intel aesni_intel aes_x86_64 glue_helper ohci_pci lrw gf128mul ehci_pci ablk_helper ohci_hcd psmouse ata_piix ahci cryptd ehci_hcd libahci fjes libata usbcore scsi_mod e1000 usb_common
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549145] CPU: 2 PID: 5294 Comm: insmod Tainted: G           OE   4.7.0-1-amd64 #1 Debian 4.7.5-1
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549163] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549179] task: ffff880069cac0c0 ti: ffff880001000000 task.ti: ffff880001000000
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549194] RIP: 0010:[<ffffffffc09e2119>]  [<ffffffffc09e2119>] agent_init+0x119/0x1000 [dattobd]
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549213] RSP: 0018:ffff880001003ca8  EFLAGS: 00010286
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549223] RAX: ffff88007ffefad8 RBX: 0000000000000000 RCX: 0000000000000000
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549245] RDX: ffffffffc0791018 RSI: ffffffff8d2c4490 RDI: ffffffff8d2c4460
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549259] RBP: ffff880001003ea0 R08: ffffffffc0791018 R09: ffffffff8cb1d370
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549284] R10: ffff8800f8adb9ff R11: 0000000000000000 R12: ffffffffc09e2000
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549298] R13: ffffffffc09df340 R14: ffffffffc09df390 R15: ffff880001003ec0
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549317] FS:  00007f5855651700(0000) GS:ffff88007fc80000(0000) knlGS:0000000000000000
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549332] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549342] CR2: ffff88007fff0000 CR3: 000000007ad40000 CR4: 00000000000406e0
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549357] Stack:
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549363]  0000000000000000 ffffffff8c80217c ffff88007ffebbd0 0000000000000202
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549379]  000000000000001f 0000000012ee3c5c 0000000000000286 ffff880010fe9800
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549401]  ffffffff8c9bd30d 0000000012ee3c5c ffffffffc09df340 0000000012ee3c5c
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549417] Call Trace:
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549429]  [<ffffffff8c80217c>] ? do_one_initcall+0x4c/0x180
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549443]  [<ffffffff8c9bd30d>] ? __vunmap+0x6d/0xc0
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549457]  [<ffffffff8c976adb>] ? do_init_module+0x5b/0x1d6
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549470]  [<ffffffff8c8ff982>] ? load_module+0x22a2/0x2950
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549482]  [<ffffffff8c8fc460>] ? __symbol_put+0x60/0x60
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549495]  [<ffffffff8c900279>] ? SYSC_finit_module+0xc9/0x100
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549510]  [<ffffffff8cddb136>] ? system_call_fast_compare_end+0xc/0x96
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549523] Code: c6 48 c7 c7 e8 dd 9d c0 e8 87 46 f9 cb e9 b7 00 00 00 31 c0 80 3d f7 d1 ff ff 00 0f 84 af 00 00 00 48 b8 00 00 00 00 00 88 ff ff <48> 81 b8 28 05 00 00 70 ae 21 81 75 0d 48 81 b8 30 05 00 00 70 
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549602] RIP  [<ffffffffc09e2119>] agent_init+0x119/0x1000 [dattobd]
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549622]  RSP <ffff880001003ca8>
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549630] CR2: ffff88007fff0000
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.549642] fbcon_switch: detected unhandled fb_set_par error, error code -16
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.550560] fbcon_switch: detected unhandled fb_set_par error, error code -16
Oct  2 20:17:23 devuan-testing-vm kernel: [16313.551525] ---[ end trace 83597f808acaf260 ]---

Subsequent insmods hang, but are SIGINTable. I first saw the problem in a real machine, confirmed it in this VM.

Fails to build on Linux 2.6.31

[root@segv-fc12 src]# make
if [ ! -f kernel-config.h ] || tail -1 kernel-config.h | grep -qv '#endif'; then ./genconfig.sh 2.6.31.5-127.fc12.x86_64; fi;
make -C /lib/modules/2.6.31.5-127.fc12.x86_64/build M=/root/dattobd/src modules
make[1]: Entering directory `/usr/src/kernels/2.6.31.5-127.fc12.x86_64'
  CC [M]  /root/dattobd/src/dattobd.o
/root/dattobd/src/dattobd.c:354: error: conflicting types for ‘blk_stack_limits’
include/linux/blkdev.h:920: note: previous declaration of ‘blk_stack_limits’ was here
/root/dattobd/src/dattobd.c: In function ‘blk_stack_limits’:
/root/dattobd/src/dattobd.c:357: error: ‘struct request_queue’ has no member named ‘max_sectors’
/root/dattobd/src/dattobd.c:357: error: ‘struct request_queue’ has no member named ‘max_sectors’
/root/dattobd/src/dattobd.c:357: error: ‘struct request_queue’ has no member named ‘max_sectors’
/root/dattobd/src/dattobd.c:357: error: ‘struct request_queue’ has no member named ‘max_sectors’
/root/dattobd/src/dattobd.c:357: error: ‘struct request_queue’ has no member named ‘max_sectors’
/root/dattobd/src/dattobd.c:357: error: ‘struct request_queue’ has no member named ‘max_sectors’
/root/dattobd/src/dattobd.c:357: warning: type defaults to ‘int’ in declaration of ‘_min1’
/root/dattobd/src/dattobd.c:357: error: ‘struct request_queue’ has no member named ‘max_sectors’
/root/dattobd/src/dattobd.c:357: error: ‘struct request_queue’ has no member named ‘max_sectors’
/root/dattobd/src/dattobd.c:357: warning: type defaults to ‘int’ in declaration of ‘_min2’
/root/dattobd/src/dattobd.c:357: error: ‘struct request_queue’ has no member named ‘max_sectors’
/root/dattobd/src/dattobd.c:358: error: ‘struct request_queue’ has no member named ‘max_hw_sectors’
/root/dattobd/src/dattobd.c:358: error: ‘struct request_queue’ has no member named ‘max_hw_sectors’
/root/dattobd/src/dattobd.c:358: error: ‘struct request_queue’ has no member named ‘max_hw_sectors’
/root/dattobd/src/dattobd.c:358: error: ‘struct request_queue’ has no member named ‘max_hw_sectors’
/root/dattobd/src/dattobd.c:358: error: ‘struct request_queue’ has no member named ‘max_hw_sectors’
/root/dattobd/src/dattobd.c:358: error: ‘struct request_queue’ has no member named ‘max_hw_sectors’
/root/dattobd/src/dattobd.c:358: warning: type defaults to ‘int’ in declaration of ‘_min1’
/root/dattobd/src/dattobd.c:358: error: ‘struct request_queue’ has no member named ‘max_hw_sectors’
/root/dattobd/src/dattobd.c:358: error: ‘struct request_queue’ has no member named ‘max_hw_sectors’
/root/dattobd/src/dattobd.c:358: warning: type defaults to ‘int’ in declaration of ‘_min2’
/root/dattobd/src/dattobd.c:358: error: ‘struct request_queue’ has no member named ‘max_hw_sectors’
/root/dattobd/src/dattobd.c:359: error: ‘struct request_queue’ has no member named ‘bounce_pfn’
/root/dattobd/src/dattobd.c:359: error: ‘struct request_queue’ has no member named ‘bounce_pfn’
/root/dattobd/src/dattobd.c:359: error: ‘struct request_queue’ has no member named ‘bounce_pfn’
/root/dattobd/src/dattobd.c:359: error: ‘struct request_queue’ has no member named ‘bounce_pfn’
/root/dattobd/src/dattobd.c:359: error: ‘struct request_queue’ has no member named ‘bounce_pfn’
/root/dattobd/src/dattobd.c:359: error: ‘struct request_queue’ has no member named ‘bounce_pfn’
/root/dattobd/src/dattobd.c:359: warning: type defaults to ‘int’ in declaration of ‘_min1’
/root/dattobd/src/dattobd.c:359: error: ‘struct request_queue’ has no member named ‘bounce_pfn’
/root/dattobd/src/dattobd.c:359: error: ‘struct request_queue’ has no member named ‘bounce_pfn’
/root/dattobd/src/dattobd.c:359: warning: type defaults to ‘int’ in declaration of ‘_min2’
/root/dattobd/src/dattobd.c:359: error: ‘struct request_queue’ has no member named ‘bounce_pfn’
/root/dattobd/src/dattobd.c:360: error: ‘struct request_queue’ has no member named ‘seg_boundary_mask’
/root/dattobd/src/dattobd.c:360: error: ‘struct request_queue’ has no member named ‘seg_boundary_mask’
/root/dattobd/src/dattobd.c:360: error: ‘struct request_queue’ has no member named ‘seg_boundary_mask’
/root/dattobd/src/dattobd.c:360: error: ‘struct request_queue’ has no member named ‘seg_boundary_mask’
/root/dattobd/src/dattobd.c:360: error: ‘struct request_queue’ has no member named ‘seg_boundary_mask’
/root/dattobd/src/dattobd.c:360: error: ‘struct request_queue’ has no member named ‘seg_boundary_mask’
/root/dattobd/src/dattobd.c:360: warning: type defaults to ‘int’ in declaration of ‘_min1’
/root/dattobd/src/dattobd.c:360: error: ‘struct request_queue’ has no member named ‘seg_boundary_mask’
/root/dattobd/src/dattobd.c:360: error: ‘struct request_queue’ has no member named ‘seg_boundary_mask’
/root/dattobd/src/dattobd.c:360: warning: type defaults to ‘int’ in declaration of ‘_min2’
/root/dattobd/src/dattobd.c:360: error: ‘struct request_queue’ has no member named ‘seg_boundary_mask’
/root/dattobd/src/dattobd.c:362: error: ‘struct request_queue’ has no member named ‘max_phys_segments’
/root/dattobd/src/dattobd.c:362: error: ‘struct request_queue’ has no member named ‘max_phys_segments’
/root/dattobd/src/dattobd.c:362: error: ‘struct request_queue’ has no member named ‘max_phys_segments’
/root/dattobd/src/dattobd.c:362: error: ‘struct request_queue’ has no member named ‘max_phys_segments’
/root/dattobd/src/dattobd.c:362: error: ‘struct request_queue’ has no member named ‘max_phys_segments’
/root/dattobd/src/dattobd.c:362: error: ‘struct request_queue’ has no member named ‘max_phys_segments’
/root/dattobd/src/dattobd.c:362: warning: type defaults to ‘int’ in declaration of ‘_min1’
/root/dattobd/src/dattobd.c:362: error: ‘struct request_queue’ has no member named ‘max_phys_segments’
/root/dattobd/src/dattobd.c:362: error: ‘struct request_queue’ has no member named ‘max_phys_segments’
/root/dattobd/src/dattobd.c:362: warning: type defaults to ‘int’ in declaration of ‘_min2’
/root/dattobd/src/dattobd.c:362: error: ‘struct request_queue’ has no member named ‘max_phys_segments’
/root/dattobd/src/dattobd.c:364: error: ‘struct request_queue’ has no member named ‘max_hw_segments’
/root/dattobd/src/dattobd.c:364: error: ‘struct request_queue’ has no member named ‘max_hw_segments’
/root/dattobd/src/dattobd.c:364: error: ‘struct request_queue’ has no member named ‘max_hw_segments’
/root/dattobd/src/dattobd.c:364: error: ‘struct request_queue’ has no member named ‘max_hw_segments’
/root/dattobd/src/dattobd.c:364: error: ‘struct request_queue’ has no member named ‘max_hw_segments’
/root/dattobd/src/dattobd.c:364: error: ‘struct request_queue’ has no member named ‘max_hw_segments’
/root/dattobd/src/dattobd.c:364: warning: type defaults to ‘int’ in declaration of ‘_min1’
/root/dattobd/src/dattobd.c:364: error: ‘struct request_queue’ has no member named ‘max_hw_segments’
/root/dattobd/src/dattobd.c:364: error: ‘struct request_queue’ has no member named ‘max_hw_segments’
/root/dattobd/src/dattobd.c:364: warning: type defaults to ‘int’ in declaration of ‘_min2’
/root/dattobd/src/dattobd.c:364: error: ‘struct request_queue’ has no member named ‘max_hw_segments’
/root/dattobd/src/dattobd.c:366: error: ‘struct request_queue’ has no member named ‘max_segment_size’
/root/dattobd/src/dattobd.c:366: error: ‘struct request_queue’ has no member named ‘max_segment_size’
/root/dattobd/src/dattobd.c:366: error: ‘struct request_queue’ has no member named ‘max_segment_size’
/root/dattobd/src/dattobd.c:366: error: ‘struct request_queue’ has no member named ‘max_segment_size’
/root/dattobd/src/dattobd.c:366: error: ‘struct request_queue’ has no member named ‘max_segment_size’
/root/dattobd/src/dattobd.c:366: error: ‘struct request_queue’ has no member named ‘max_segment_size’
/root/dattobd/src/dattobd.c:366: warning: type defaults to ‘int’ in declaration of ‘_min1’
/root/dattobd/src/dattobd.c:366: error: ‘struct request_queue’ has no member named ‘max_segment_size’
/root/dattobd/src/dattobd.c:366: error: ‘struct request_queue’ has no member named ‘max_segment_size’
/root/dattobd/src/dattobd.c:366: warning: type defaults to ‘int’ in declaration of ‘_min2’
/root/dattobd/src/dattobd.c:366: error: ‘struct request_queue’ has no member named ‘max_segment_size’
make[2]: *** [/root/dattobd/src/dattobd.o] Error 1
make[1]: *** [_module_/root/dattobd/src] Error 2
make[1]: Leaving directory `/usr/src/kernels/2.6.31.5-127.fc12.x86_64'
make: *** [default] Error 2

Fails to build on Linux 2.6.21

[root@segv-fc7 src]# ./genconfig.sh $(uname -r)
generating configurations
performing configure test: HAVE_BDEV_STACK_LIMITS - not present
performing configure test: HAVE_BDOPS_OPEN_INODE - present
performing configure test: HAVE_BDOPS_OPEN_INT - not present
performing configure test: HAVE_BD_SUPER - not present
performing configure test: HAVE_BIO_BI_POOL - not present
performing configure test: HAVE_BIO_BI_REMAINING - not present
performing configure test: HAVE_BIO_ENDIO_1 - not present
performing configure test: HAVE_BIO_ENDIO_INT - present
performing configure test: HAVE_BIO_LIST - not present
performing configure test: HAVE_BIO_RW_FLAGS - not present
performing configure test: HAVE_BIOSET_CREATE_3 - present
performing configure test: HAVE_BLKDEV_GET_2 - not present
performing configure test: HAVE_BLKDEV_GET_BY_PATH - not present
performing configure test: HAVE_BLKDEV_PUT_1 - present
performing configure test: HAVE_BLK_QUEUE_MAX_SECTORS - present
performing configure test: HAVE_BLK_SET_DEFAULT_LIMITS - not present
performing configure test: HAVE_BLK_SET_STACKING_LIMITS - not present
performing configure test: HAVE_BVEC_ITER - not present
performing configure test: HAVE_BVEC_MERGE_DATA - not present
performing configure test: HAVE___DENTRY_PATH - not present
performing configure test: HAVE_DENTRY_PATH_RAW - not present
performing configure test: HAVE_D_UNLINKED - not present
performing configure test: HAVE_FILE_INODE - not present
performing configure test: HAVE_FMODE_T - not present
performing configure test: HAVE_GENHD_FL_NO_PART_SCAN - not present
performing configure test: HAVE_INODE_LOCK - not present
performing configure test: HAVE_IOPS_FALLOCATE - not present
performing configure test: HAVE_KERN_PATH - not present
performing configure test: HAVE_MAKE_REQUEST_FN_INT - present
performing configure test: HAVE_MAKE_REQUEST_FN_VOID - not present
performing configure test: HAVE_MERGE_BVEC_FN - not present
performing configure test: HAVE_MNT_WANT_WRITE - not present
performing configure test: HAVE_NOOP_LLSEEK - not present
performing configure test: HAVE_NO_QUEUE_LIMITS - present
performing configure test: HAVE_NOTIFY_CHANGE_2 - present
performing configure test: HAVE_PART_NR_SECTS_READ - not present
performing configure test: HAVE_PATH_PUT - not present
performing configure test: HAVE_QUEUE_LIMITS_MAX_HW_SEGMENTS - not present
performing configure test: HAVE_SB_START_WRITE - not present
performing configure test: HAVE_STRUCT_PATH - present
performing configure test: HAVE_SUBMIT_BIO_1 - not present
performing configure test: HAVE_SUBMIT_BIO_WAIT - not present
performing configure test: HAVE_SYS_OLDUMOUNT - not present
performing configure test: HAVE_TASK_STRUCT_TASK_WORKS_CB_HEAD - not present
performing configure test: HAVE_TASK_STRUCT_TASK_WORKS_HLIST - not present
performing configure test: HAVE_THAW_BDEV_INT - not present
performing configure test: HAVE_USER_PATH_AT - not present
performing configure test: HAVE_VFS_FALLOCATE - not present
performing configure test: HAVE_VFS_UNLINK_2 - present
performing configure test: HAVE_VZALLOC - not present
performing sys_mount lookup
performing sys_umount lookup
performing sys_oldumount lookup
performing sys_call_table lookup
performing printk lookup

[root@segv-fc7 dattobd]# make driver
make -C src
make[1]: Entering directory `/root/dattobd/src'
if [ ! -f kernel-config.h ] || tail -1 kernel-config.h | grep -qv '#endif'; then ./genconfig.sh 2.6.21-1.3194.fc7; fi;
make -C /lib/modules/2.6.21-1.3194.fc7/build M=/root/dattobd/src modules
make[2]: Entering directory `/usr/src/kernels/2.6.21-1.3194.fc7-x86_64'
  CC [M]  /root/dattobd/src/dattobd.o
/root/dattobd/src/dattobd.c: In function ‘dattobd_lookup_bdev’:
/root/dattobd/src/dattobd.c:131: error: ‘struct nameidata’ has no member named ‘path’
/root/dattobd/src/dattobd.c:146: warning: implicit declaration of function ‘path_put’
/root/dattobd/src/dattobd.c:146: error: ‘struct nameidata’ has no member named ‘path’
/root/dattobd/src/dattobd.c: At top level:
/root/dattobd/src/dattobd.c:340: warning: conflicting types for ‘path_put’
/root/dattobd/src/dattobd.c:146: warning: previous implicit declaration of ‘path_put’ was here
/root/dattobd/src/dattobd.c: In function ‘kern_path’:
/root/dattobd/src/dattobd.c:398: error: ‘struct nameidata’ has no member named ‘path’
/root/dattobd/src/dattobd.c: In function ‘user_path_at’:
/root/dattobd/src/dattobd.c:425: error: ‘struct nameidata’ has no member named ‘path’
/root/dattobd/src/dattobd.c:426: error: ‘struct nameidata’ has no member named ‘path’
/root/dattobd/src/dattobd.c: In function ‘real_fallocate’:
/root/dattobd/src/dattobd.c:1205: error: ‘const struct file_operations’ has no member named ‘fallocate’
/root/dattobd/src/dattobd.c:1210: error: ‘const struct file_operations’ has no member named ‘fallocate’
/root/dattobd/src/dattobd.c: In function ‘path_get_absolute_pathname’:
/root/dattobd/src/dattobd.c:1408: warning: passing argument 1 of ‘d_path’ from incompatible pointer type
/root/dattobd/src/dattobd.c:1408: warning: passing argument 2 of ‘d_path’ from incompatible pointer type
/root/dattobd/src/dattobd.c:1408: warning: passing argument 3 of ‘d_path’ makes pointer from integer without a cast
/root/dattobd/src/dattobd.c:1408: error: too few arguments to function ‘d_path’
/root/dattobd/src/dattobd.c: In function ‘tracing_mrf’:
/root/dattobd/src/dattobd.c:2859: warning: implicit declaration of function ‘ACCESS_ONCE’
/root/dattobd/src/dattobd.c:2859: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c:2859: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c: In function ‘bdev_is_already_traced’:
/root/dattobd/src/dattobd.c:2958: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c:2958: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c: In function ‘find_orig_mrf’:
/root/dattobd/src/dattobd.c:2976: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c:2976: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c: In function ‘__tracer_should_reset_mrf’:
/root/dattobd/src/dattobd.c:2998: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c:2998: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c: In function ‘minor_range_recalculate’:
/root/dattobd/src/dattobd.c:3466: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c:3466: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c: In function ‘__handle_bdev_mount_nowrite’:
/root/dattobd/src/dattobd.c:4357: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c:4357: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c: In function ‘__handle_bdev_mount_writable’:
/root/dattobd/src/dattobd.c:4383: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c:4383: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c: In function ‘agent_exit’:
/root/dattobd/src/dattobd.c:4771: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c:4771: warning: assignment makes pointer from integer without a cast
/root/dattobd/src/dattobd.c: In function ‘agent_init’:
/root/dattobd/src/dattobd.c:4820: warning: implicit declaration of function ‘proc_create’
/root/dattobd/src/dattobd.c:4820: warning: assignment makes pointer from integer without a cast
make[3]: *** [/root/dattobd/src/dattobd.o] Error 1
make[2]: *** [_module_/root/dattobd/src] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.21-1.3194.fc7-x86_64'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/root/dattobd/src'
make: *** [driver] Error 2

very poor read performance from snapshot on CentOS 6 with dattobd 0.9.12

Just upgraded to 0.9.12 from 0.9.10, and am now seeing terrible read rates from the snapshot, around 1MB/s. I will investigate further and provide full details of kernel version etc. when I am able, but just wanted to open this now, in case others are seeing similar.

Snapshot changes with snapshot origin

Steps to reproduce (with dattobd 0.8.10, kernel 3.10.0-327.4.4.el7.x86_6):

rm /tmp/test.db || true
echo "CREATE TABLE test (tmp TEXT); INSERT INTO test (tmp) VALUES ('test1');" | sqlite3 /tmp/test.db
dbdctl setup-snapshot /dev/vda1 /.datto 0
mkdir -p /mnt/snap
mount /dev/datto0 /mnt/snap -o ro,norecovery
echo "SELECT * FROM test;" | sqlite3 /mnt/snap/tmp/test.db
echo "UPDATE test SET tmp='test2';" | sqlite3 /tmp/test.db
echo 1 > /proc/sys/vm/drop_caches
echo "SELECT * FROM test;" | sqlite3 /mnt/snap/tmp/test.db
echo "UPDATE test SET tmp='test3';" | sqlite3 /tmp/test.db
echo "SELECT * FROM test;" | sqlite3 /mnt/snap/tmp/test.db
echo 1 > /proc/sys/vm/drop_caches
echo "SELECT * FROM test;" | sqlite3 /mnt/snap/tmp/test.db
umount /mnt/snap
dbdctl destroy 0

Expected output:
4x test1

Actual output:
test1 test2 test2 test3

Dattobd kernel module error

I've tried Ubuntu 16.04.1 LTS as well as 14.04 LTS running under VMWare ESXi and I keep getting kernel errors. I installed using the REPO in the install documentation. Here is the exact error.

dattobd: module verification failed: signature and/or required key missing - tainting kernel

Any suggestions on how to get around this issue?

RAID0: Error creating read clones of write bio

During the 2nd backup of Ubuntu 14.04.3 (3.19.0-25-generic) with software RAID0 (running in VirtualBox) using the Datto Linux Agent I get an error (EAGAIN) during the transition to incremental.

[ 2203.391549] datto: ioctl command received: 1074041093
[ 2203.391553] datto: received transition inc ioctl - 0
[ 2203.391554] datto: allocating device struct
[ 2203.391555] datto: initializing tracer
[ 2203.391556] datto: creating kernel cow thread
[ 2203.391647] datto: getting the base block device's make_request_fn
[ 2203.391649] datto: freezing block device
[ 2203.392416] datto: error creating read clones of write bio: -11
[ 2203.392502] datto: error tracing bio for snapshot: -11
[ 2203.399013] datto: error detected in cow thread, cleaning up cow
[ 2203.411935] datto: starting tracing
[ 2203.411938] datto: thawing block device
[ 2203.412004] datto: stopping cow thread
[ 2203.412120] datto: errors occurred while cleaning up cow thread, putting incremental into error state: -11
[ 2203.412195] datto: stopping mrf thread
[ 2203.412211] datto: freeing gendisk
[ 2203.412330] datto: freeing request queue
[ 2203.418628] datto: error during transition to incremental ioctl handler: -11
[ 2204.424094] datto: ioctl command received: 1074041093
[ 2204.424098] datto: received transition inc ioctl - 0
[ 2204.424099] datto: device specified is in the fail state: -22
[ 2204.424163] datto: error during transition to incremental ioctl handler: -22

How I set up RAID0:

1. Create 2 disks of size 16G
2. Create new empty partition table on both drives
3. Create MD device (RAID0)
4. Select /dev/sda and /dev/sdb
5. Undo changes to partition (removes partition table added by Ubuntu on the RAID0 device)
6. Create new empty partition table on RAID0 device
7. Create partitions
   • 512 MB /boot partition formatted as ext4
   • 16 GB / partition formatted as ext4
   • /home partition formatted as XFS using remaining space

This issue only happens for /boot, the first partition on the RAID0 device.

Running hdparm on a file on this partition shows it has a blocksize of 1024 (compared to 4096 of the other partitions).

/boot/.datto:
 filesystem blocksize 1024, begins at LBA 63; assuming 512 byte sectors.
 byte_offset  begin_LBA    end_LBA    sectors
           0     393281     393282          2
        1024     393283     395272       1990

Repository links broken

E.g.

https://cpkg.datto.com/datto-rpm/EnterpriseLinux/6/x86_64/datto-el-rpm-release-6-8.1.noarch.rpm

does not work.

Snapshot driver failing with cow file max size exceeded

dattobd driver failling with cow file size exceed error.

kernel version - 2.6.32-358.6.2.el6.x86_64.
fileSystem - ext4 (Total Size:- 985G Used:- 514G Free:- 422G)
CowFile Size - 100Gb

There is no huge data copies in snapshot mode(less than 2Gb), but after some time driver returning errors as below.

kernel: datto: cow file max size exceeded: -27
kernel: datto: error writing cow data: -27
kernel: datto: error writing cow data and mapping: -27
kernel: datto: error handling write bio: -27

I tried with datto-0.9.10 and datto-0.9.15.

Feature request: dynamically grow COW file

it is technically feasible to have the space allocated for the COW data to grow dynamically whenever it nears being full, rather than having to choose a fixed size for the file in advance?

Fails to build on Linux 4.8.0

Linux 4.8 will be out soon. It includes changes to struct bio, submit_bio, and submit_bio_wait.

Print device/file path in log messages

Log the device (e.g., /dev/sda2) or file path (e.g., /.datto) instead of logging "device specified", "block device", or "file specified".

datto: file specified is not on block device specified: -22

Filesystem on snapshot is not commited

Steps to reproduce (with dattobd 0.8.10, kernel 3.10.0-327.4.4.el7.x86_6):

dbdctl setup-snapshot /dev/vda1 /.datto 0
mount /dev/datto0 /mnt/snap
EXT4-fs (datto0): INFO: recovery required on readonly filesystem
EXT4-fs (datto0): write access unavailable, cannot proceed

Expected: Filesystem on block device is mountable without "norecovery" flag, because it does not need recovery
Is: Dattobd does not flush the filesystem journal before creating the snapshot

You are already using freeze_bdev when the block device is transitioned from snapshot to tracing. Couldn't you call that during snapshot creation as well?

Documentation

How are you recording the incremental snapshots ? Is there any documentation for that ?

I am interested in this project.

insmod of dattobd driver causes BUG: unable to handle kernel paging request at ffff88009d5e3000

insmoding v0.9.2 of dattobd causes the following kernel BUG:

Mar 12 14:07:43 omegafw kernel: [  267.752739] BUG: unable to handle kernel paging request at ffff88009d5e3000
Mar 12 14:07:43 omegafw kernel: [  267.761826] IP: [<ffffffffa0a42101>] agent_init+0x101/0x1000 [dattobd]
Mar 12 14:07:43 omegafw kernel: [  267.770829] PGD 1d24067 PUD 13efff067 PMD 13effe067 PTE 0
Mar 12 14:07:43 omegafw kernel: [  267.779723] Oops: 0000 [#1] SMP 
Mar 12 14:07:43 omegafw kernel: [  267.788439] Modules linked in: dattobd(O+) nf_log_ipv4 nf_log_common xt_LOG xt_TCPMSS xt_conntrack xt_nat xt_multiport nf_conntrack_pptp nf_conntrack_proto_gre iptable_raw xt_CT xt_tcpudp iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables x_tables sch_fq_codel sch_prio sch_htb nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc pppoe pppox ppp_generic slhc aufs(O) dm_crypt dm_mod snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi amd_freq_sensitivity kvm_amd kvm crct10dif_pclmul crc32_pclmul snd_hda_intel sha256_ssse3 sha256_generic snd_hda_codec hmac snd_hda_core drbg ansi_cprng aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hwdep amdkfd snd_pcm evdev fam15h_power edac_mce_amd 8250_fintek radeon edac_core ttm sg pcspkr serio_raw k10temp wmi tpm_infineon snd_timer drm_kms_helper snd soundcore drm i2c_algo_bit i2c_piix4 shpchp button acpi_cpufreq tpm_tis tpm processor ses enclosure ext4 crc16 mbcache jbd2 btrfs xor raid6_pq sr_mod sd_mod cdrom uas usb_storage ohci_pci crc32c_intel ahci libahci xhci_pci libata ohci_hcd ehci_pci xhci_hcd ehci_hcd scsi_mod usbcore usb_common r8169 mii
Mar 12 14:07:43 omegafw kernel: [  267.877730] CPU: 3 PID: 3005 Comm: insmod Tainted: G           O    4.3.0-1-amd64 #1 Debian 4.3.5-1
Mar 12 14:07:43 omegafw kernel: [  267.886493] Hardware name: MSI MS-7865/AM1I (MS-7865), BIOS V10.2 12/24/2014
Mar 12 14:07:43 omegafw kernel: [  267.895168] task: ffff88008ee62f00 ti: ffff88008dfc0000 task.ti: ffff88008dfc0000
Mar 12 14:07:43 omegafw kernel: [  267.903777] RIP: 0010:[<ffffffffa0a42101>]  [<ffffffffa0a42101>] agent_init+0x101/0x1000 [dattobd]
Mar 12 14:07:43 omegafw kernel: [  267.912381] RSP: 0018:ffff88008dfc3ca8  EFLAGS: 00010286
Mar 12 14:07:43 omegafw kernel: [  267.920836] RAX: ffff88009d5e2ad8 RBX: 0000000000000000 RCX: 0000000000000000
Mar 12 14:07:43 omegafw kernel: [  267.929235] RDX: ffffffffa07ab3d8 RSI: ffffffff81abc9d0 RDI: ffffffff81abc9a0
Mar 12 14:07:43 omegafw kernel: [  267.937525] RBP: ffff880134dd2160 R08: ffffffffa07ab3d8 R09: ffffffff812e1160
Mar 12 14:07:43 omegafw kernel: [  267.945689] R10: ffffffff8182b796 R11: 0000000000000000 R12: ffffffffa0a42000
Mar 12 14:07:43 omegafw kernel: [  267.953742] R13: 0000000000000000 R14: ffffffffa0a3f2c0 R15: 0000000000000001
Mar 12 14:07:43 omegafw kernel: [  267.961695] FS:  00007ff6c580a700(0000) GS:ffff88013ed80000(0000) knlGS:0000000000000000
Mar 12 14:07:43 omegafw kernel: [  267.969576] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 12 14:07:43 omegafw kernel: [  267.977335] CR2: ffff88009d5e3000 CR3: 000000008dc1b000 CR4: 00000000000406e0
Mar 12 14:07:43 omegafw kernel: [  267.985029] Stack:
Mar 12 14:07:43 omegafw kernel: [  267.992546]  ffffffff81a13080 ffffffff81002122 ffffea0004d39e00 02ffff8000000000
Mar 12 14:07:43 omegafw kernel: [  268.000117]  0000000000000001 ffffea0004d39e00 0000000000000286 ffff88013eff8d80
Mar 12 14:07:43 omegafw kernel: [  268.007597]  0000000000000282 0000000000000286 ffffffffa0a3f2c0 00000000500430c5
Mar 12 14:07:43 omegafw kernel: [  268.014967] Call Trace:
Mar 12 14:07:43 omegafw kernel: [  268.022154]  [<ffffffff81002122>] ? do_one_initcall+0xb2/0x200
Mar 12 14:07:43 omegafw kernel: [  268.029299]  [<ffffffff8115e0c3>] ? do_init_module+0x5b/0x1dc
Mar 12 14:07:43 omegafw kernel: [  268.036297]  [<ffffffff810f0fb3>] ? load_module+0x2173/0x2780
Mar 12 14:07:43 omegafw kernel: [  268.043160]  [<ffffffff810ed730>] ? __symbol_put+0x60/0x60
Mar 12 14:07:43 omegafw kernel: [  268.049889]  [<ffffffff811d2ccb>] ? kernel_read+0x4b/0x70
Mar 12 14:07:43 omegafw kernel: [  268.056475]  [<ffffffff810f17ee>] ? SyS_finit_module+0xae/0xe0
Mar 12 14:07:43 omegafw kernel: [  268.062941]  [<ffffffff81586432>] ? system_call_fast_compare_end+0xc/0x67
Mar 12 14:07:43 omegafw kernel: [  268.069302] Code: c6 48 c7 c7 c8 dc a3 a0 e8 90 bb 71 e0 e9 b7 00 00 00 31 c0 80 3d a7 d1 ff ff 00 0f 84 af 00 00 00 48 b8 00 00 00 00 00 88 ff ff <48> 81 b8 28 05 00 00 f0 eb 1e 81 75 0d 48 81 b8 30 05 00 00 30 
Mar 12 14:07:43 omegafw kernel: [  268.082545] RIP  [<ffffffffa0a42101>] agent_init+0x101/0x1000 [dattobd]
Mar 12 14:07:43 omegafw kernel: [  268.088790]  RSP <ffff88008dfc3ca8>
Mar 12 14:07:43 omegafw kernel: [  268.094859] CR2: ffff88009d5e3000
Mar 12 14:07:43 omegafw kernel: [  268.100783] ---[ end trace 79e495319cebb6e2 ]---
Mar 12 14:07:43 omegafw kernel: [  268.179642] datto: device specified does not exist: -22
Mar 12 14:07:43 omegafw kernel: [  268.181916] datto: error during destroy ioctl handler: -22

rmmod then fails due to the driver apparently being 'in use' (by what??). v0.8.14 works fine.

Uptodate Debian Testing machine, 'Linux omegafw 4.3.0-1-amd64 #1 SMP Debian 4.3.5-1 (2016-02-06) x86_64 GNU/Linux'.

Fails to build on Linux 2.6.35

Attempting to build on Fedora Core 14 (2.6.35.14-106.fc14.x86_64).

[root@segv-fc14 dattobd]# make driver
make -C src
make[1]: Entering directory `/root/dattobd/src'
if [ ! -f kernel-config.h ] || tail -1 kernel-config.h | grep -qv '#endif'; then ./genconfig.sh 2.6.35.14-106.fc14.x86_64; fi;
make -C /lib/modules/2.6.35.14-106.fc14.x86_64/build M=/root/dattobd/src modules
make[2]: Entering directory `/usr/src/kernels/2.6.35.14-106.fc14.x86_64'
  CC [M]  /root/dattobd/src/dattobd.o
/root/dattobd/src/dattobd.c: In function ‘submit_bio_wait’:
/root/dattobd/src/dattobd.c:286:14: error: ‘BIO_RW_SYNC’ undeclared (first use in this function)
/root/dattobd/src/dattobd.c:286:14: note: each undeclared identifier is reported only once for each function it appears in
/root/dattobd/src/dattobd.c: In function ‘snap_handle_read_bio’:
/root/dattobd/src/dattobd.c:2369:46: error: ‘BIO_RW_SYNC’ undeclared (first use in this function)
make[3]: *** [/root/dattobd/src/dattobd.o] Error 1
make[2]: *** [_module_/root/dattobd/src] Error 2
make[2]: Leaving directory `/usr/src/kernels/2.6.35.14-106.fc14.x86_64'
make[1]: *** [default] Error 2
make[1]: Leaving directory 

redundant read problem when reading from snapshot

when read request comes to snapshot, dattobd firstly read all Bios from the base device and then it replace the bios in cow. Bios of read request that also in cow are read twice, may it be less efficent?especially for cow data is large.
how about first check whether the read bio is in cow, if it is, read it just from cow. if not, read it from base device?

CFQ: Snapshot floods dmesg: "request aux data allocation failed"

Taking a snapshot when using CFQ will flood dmesg with the following line:

__get_request: dev 8:0: request aux data allocation failed, iosched may be disturbed

This appears to only happens on partitions (/dev/sda2); I do not see the issue with dm volumes (/dev/mapper/centos-root).

Steps to reproduce:

echo cfq > /sys/block/sda/queue/scheduler
dbdctl setup-snapshot /dev/sda1 /cow.snap 1
dd if=/dev/urandom of=/urand bs=1M count=32
sync

update-img: Warnings building on Debian 6

Building on Debian 6 produces:

make[1]: Entering directory `/root/dattobd/utils'
gcc -Wall -std=gnu99 -o update-img update-img.c /root/dattobd//lib/libdattobd.a
update-img.c: In function ‘copy_block’:
update-img.c:38: warning: implicit declaration of function ‘pread’
update-img.c:46: warning: implicit declaration of function ‘pwrite’
make[1]: Leaving directory `/root/dattobd/utils'