vrtmrz/livesync-bridge

Hello everyone!

I am Vorotamoroz, a believer in progress and harmony for mankind. (You know, this is the theme of EXPO '70). May the world be improved with all our hands!

Some of non-trivial informations will be posted in here: https://mozilla.social/@vrtmrz
Also, I might post also some miscellaneous 'busy' reason reports here.

Excuses

You may have noticed a decline in the frequency of my updates. This, I regret to inform, is due to a spot of misfortune that has befallen my dear loved family.
However, you need not fret! I have lost neither my passion nor my resolve. Though my responses may be fragmented for a time, rest assured, I shall make a full return.
Updates would be done properly!

livesync-bridge's Issues

Hidden files show up with leading characters, headless sync not working.

First of all, thank you for these tools and all the work you put into them. I'm still working on my setup but am excited to get it working.

I have selfhosted-livesync up and running and want to set up livesync-bridge so that I can use a script to edit .md files directly on my server and have the changes reflected in Obsidian. However, I only got this to work for a minute and have had weird problems since then, even after deleting the database and re-installing both livesync tools.

Currently I am getting messages like this:
bridge-1 | 2/13/2024, 7:37:27 PM 10 WATCH: PROCESS DONE: i.obsidian/workspace.json
bridge-1 | 2/13/2024, 7:37:27 PM 1 WATCH: PROCESSING: i.obsidian/workspace.json

And then the files show up in my CouchDB:

and in Obsidian and File Explorer:

Whenever I open the vault, I get a bunch of messages which only happens after starting livesync-bridge:

I edited the docker-compose to include the directory where I save my vaults (not sure where they're expected to be located by default):

version: "2.1"
services:
  bridge:
    build: .
    volumes:
      - ./data:/app/data
      - ./dat:/app/dat
      - "G:/obsidian/vaults:/obsidian/vaults"
    restart: unless-stopped

Here's my config.json:

{
    "peers": [
        {
            "type": "couchdb",
            "name": "testname",
            "group": "test",
            "database": "livesynctest",
            "username": "myusername",
            "password": "My-CoUcH-Db-PaSsW0Rd",
            "url": "https://my.couchdb.url",
            "customChunkSize": 100,
            "minimumChunkSize": 20,
            "passphrase": "",
            "obfuscatePassphrase": "",
            "baseDir": ""
        },
        {
            "type": "storage",
            "group": "test",
            "name": "storage-test",
            "baseDir": "/obsidian/vaults/livesynctest/",
            "scanOfflineChanges": true
        }   
    ]
}

Recommendation: Support database/folder synchronization in different warehouses

Hi, please ask vrtmrz:

I want to synchronize multiple Databases (Vaults) to multiple folders one by one for backup of original files without opening Obsidian. For example:

1. DB_A --> Storage_A
2. DB_B --> Storage_B
   ...

Judging from the code and documentation, LiveSync-Bridge currently supports the synchronization of a single warehouse (referring to the contents of the same folder). The main purpose is for sharing, but there is no "one-to-one" synchronization (used for backup and folders export) instance.

Please help to see if it is currently supported? And how to support? grateful!

Finally, a big thank you to contributors such as vrtmrz for making great Obsidian plug-ins, solving synchronization issues, and continuing to update them. Keep up the good work!

Too many open files (os error 24)

Livesync-bridge crashes when there are 89-90 or more files.

This is the error I'm getting:

LiveSync Bridge is now starting...
LiveSync Bridge is now started!
4/19/2024, 12:06:56 AM  1       Cache initialized 300 / 10000000000000
4/19/2024, 12:06:56 AM  1       Cache initialized 300 / 50000000
4/19/2024, 12:06:56 AM  1       Cache initialized 300 / 10000000000000
4/19/2024, 12:06:56 AM  -1      Requesting ... get http://localhost:5984/obsdb/_local%2Fobsydian_livesync_milestone
4/19/2024, 12:06:56 AM  10      [server-local-storage] Scan offline changes: Disabled
error: Uncaught Error: Too many open files (os error 24)
    at new FsWatcher (ext:runtime/40_fs_events.js:31:17)
    at Object.watchFs (ext:runtime/40_fs_events.js:93:10)
    at ext:deno_node/_fs/_fs_watch.ts:51:21
    at eventLoopTick (ext:core/01_core.js:203:13)

I wrote a script to delete files until the program didn't crash and about 90 files seems to be the number at which the program crashes. If there are less than 90 files in the storage baseDir, it will download files until it reaches 90 and then crashes.

My current setup is a linux machine, trying to replicate to a folder on the same server. I'm not running in a docker instance, just directly from my terminal.

I did the obvious things first. I've set my file handle limit all the way up:

> ulimit -n
65536

And just to double check:

> lsof -u laharah | wc -l
4795

Here's the current config I'm using:

{
    "peers": [
        {
            "type": "couchdb",
            "name": "server-database",
            "database": "obsdb",
            "url": "http://localhost:5984",
            "username": "obsidian",
            "password": "[REDACTED]",
            "baseDir": ""
        },
        {
            "type": "storage",
            "name": "server-local-storage",
            "scanOfflineChanges": false,
            "baseDir": "/home/laharah/test_obs"
        }
    ]
}

Here also is the relevent portion of the stack trace where the error originates.

15370 <... sched_yield resumed> )       = 0
15521 sigaltstack({ss_sp=0x7f49a00d7000, ss_flags=0, ss_size=8192},  <unfinished ...>
15370 sched_yield( <unfinished ...>
15521 <... sigaltstack resumed> NULL)   = 0
15370 <... sched_yield resumed> )       = 0
15370 sched_yield( <unfinished ...>
15521 prctl(PR_SET_NAME, "notify-rs inoti"... <unfinished ...>
15370 <... sched_yield resumed> )       = 0
15521 <... prctl resumed> )             = 0
15370 futex(0x55c6e6745998, FUTEX_WAIT_BITSET_PRIVATE, 4294967295, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
15521 mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f4750000000
15521 munmap(0x7f4754000000, 67108864)  = 0
15521 mprotect(0x7f4750000000, 135168, PROT_READ|PROT_WRITE) = 0
15521 sched_getaffinity(15521, 32, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]) = 32
15521 epoll_wait(326, [{EPOLLIN, {u32=1, u64=1}}], 16, -1) = 1
15521 inotify_add_watch(325, "/home/laharah/test_obs/.blog/_assets/js", IN_MODIFY|IN_ATTRIB|IN_CL
OSE_WRITE|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF) = 1
15521 futex(0x55c6e6745998, FUTEX_WAKE_PRIVATE, 1) = 1
15370 <... futex resumed> )             = 0
15521 epoll_wait(326,  <unfinished ...>
15370 inotify_init()                    = -1 EMFILE (Too many open files)
15370 brk(0x55c6e6bdc000)               = 0x55c6e6bdc000
15370 gettid()                          = 15370
15370 mmap(0x14cc10200000, 528384, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x14cc10200000
15370 munmap(0x14cc10242000, 258048)    = 0
15370 mprotect(0x14cc10200000, 270336, PROT_READ|PROT_WRITE) = 0
15370 epoll_ctl(5, EPOLL_CTL_DEL, 321, NULL) = 0
15370 close(321)                        = 0
15370 stat("/home/laharah/.cache/deno/location_data/398eebd359bc9e171ffbcfd87c28f16aefe4ea41490b8d61bf1b53cb2cf1c064/local_storage", {st_mode=S_IFREG|0644, st_size=12288, ...}) = 0
15370 fcntl(14, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=1073741826, l_len=510}) = 0 

My current best guess is that chokidar has some kind of error. Reading into some issues it looks like chokidar is supposed to combine inotify calls to reduce open handles, but it doesn't seem to be working for some reason. My TS isn't good enough for me to implement a workaround, and skimming the code, nothing jumps out at me as a cause.

I suspect that there's something wrong with my server's configuration, since I'd imagine that I'd see another issue from someone else by now.

Please let me know if there's anything you can think of. I'm planning to try running livesync-bridge in a docker container, just to check, but I don't think it'll help since the underlying volume will be served from my filesystem. If the error originates below the container, it shouldn't matter if it's containerized or not. Still I'll give it a try in the morning just in case.

Repeatedly seeing `WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...`

Hey, thanks for this, it's really hard to find a solution to syncing my obsidian vault with my blog, so I appreciate you sharing this code.

When I run this on my server all I see is this; any idea what I'm doing wrong?

...
meout=100000&heartbeat=5000
4/12/2023, 11:09:30 am	10	WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...
4/12/2023, 11:09:40 am	10	WATCH: START: (since:now)
4/12/2023, 11:09:40 am	-1	Requesting ... get http://127.0.0.1:5984/brightscroll_posts/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=now&feed=continuous&timeout=100000&heartbeat=5000
4/12/2023, 11:09:40 am	10	WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...
4/12/2023, 11:09:50 am	10	WATCH: START: (since:now)
4/12/2023, 11:09:50 am	-1	Requesting ... get http://127.0.0.1:5984/brightscroll_posts/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=now&feed=continuous&timeout=100000&heartbeat=5000
4/12/2023, 11:09:50 am	10	WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...
4/12/2023, 11:10:00 am	10	WATCH: START: (since:now)
4/12/2023, 11:10:00 am	-1	Requesting ... get http://127.0.0.1:5984/brightscroll_posts/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=now&feed=continuous&timeout=100000&heartbeat=5000
4/12/2023, 11:10:00 am	10	WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...

Here's my config:

{
    "peers": [
        {
            "type": "couchdb",
            "name": "brightscroll",
            "database": "brightscroll_posts",
            "username": "brightscroll",
            "password": <omitted>,
            "url": "http://127.0.0.1:5984",
            "customChunkSize": 100,
            "minimumChunkSize": 20,
            "baseDir": "/home/thomas/posts"
        }
    ]
}

InitialScan option?

I don't know how to initialize the filesystem storage.

How does Storage work?

This may be a misunderstanding, but I assume the storage functionality houses and syncs all vault files on a server?

Could you provide a Docker file structure example? This quote from the read me is what I'm trying to understand

storage folder have to start with "data/" to be in the volume

Livesync between Obisidian Clients

I've gotten the live sync working between clients (both connected to the same self hosted CouchDB). Works very well to sync both from phone and PC when editing notes.

When I spin up this livesync-bridge, I'm not sure how or what works.

Config

I'm running this as a container on my Server

With this config I assume the server will get a copy and sync the same vault files as my PC and Phone.

{
    "peers": [
        {
            "type": "couchdb",
            "name": "obsidianDB",
            "database": "obsidianDB",
            "username": "admin",
            "password": "***",
            "url": "https://DOMAIN",
            "customChunkSize": 100,
            "minimumChunkSize": 20,
            "passphrase": "",
            "obfuscatePassphrase": "",
            "baseDir": ""
        },
        {
          "type": "storage",
          "name": "obsidian-storage",
          "baseDir": "./vault/",
          "scanOfflineChanges": true
        }
    ]
}

logs

LiveSync Bridge is now starting...
LiveSync Bridge is now started!
2/10/2024, 8:11:50 PM	1	Cache initialized 300 / 10000000000000
2/10/2024, 8:11:50 PM	1	Cache initialized 300 / 50000000
2/10/2024, 8:11:50 PM	1	Cache initialized 300 / 10000000000000
2/10/2024, 8:11:50 PM	-1	Requesting ... get https://DOMAIN/obsidianDB/_local%2Fobsydian_livesync_milestone
2/10/2024, 8:11:50 PM	10	[obsidian-storage] Scan offline changes: Enabled, now starting...
2/10/2024, 8:11:51 PM	10	[pywriter] Watch starting from 2161-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYGBRT8XKMZuYGpkbGRsia4ehwl5LECSoQFI_UcY5AQ2KDnZ1MTIJBVdWxYAx2Iqlw
2/10/2024, 8:11:51 PM	10	WATCH: START: (since:2161-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYGBRT8XKMZuYGpkbGRsia4ehwl5LECSoQFI_UcY5AQ2KDnZ1MTIJBVdWxYAx2Iqlw)
2/10/2024, 8:11:51 PM	-1	Requesting ... get https://DOMAIN/obsidianDB/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=2161-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYGBRT8XKMZuYGpkbGRsia4ehwl5LECSoQFI_UcY5AQ2KDnZ1MTIJBVdWxYAx2Iqlw&feed=continuous&timeout=100000&heartbeat=5000
2/10/2024, 8:11:51 PM	10	[obsidian-storage] Scan offline changes: Finished

My Server repo

Here is my folder structure. The vault folder is a clone of my obsidian vault root that's on my PC and Phone

Whenever I make a change on PC or Phone Obsidian, this change does not reflect to the Server side vault. Is this a wrong assumption of what livesync-bridge can do?

I even manually copied over files from the obsidian into the Server's vault folder to see if that would help. Also, I remember on the older version of livesync I could run docker compose restart and that would kick the sync back into gear and get the server files up to date.

WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...

Thank you very much for this project, but I seem unable to achieve the expected results. I have set up CouchDB using Docker, and both the PC and mobile versions of Obsidian can sync properly. Now, I have a requirement to sync the inote database from CouchDB to the local directory /volume2/docker-data/bt/www/wwwroot/static.hsbt.xyz/ via livesync-bridge.

I ran it using the following method:

Simply run
Clone the GitHub Repository
git clone --recursive https://github.com/vrtmrz/livesync-bridge
Open the config file dat/config.sample.json, edit and save to dat/config.json. (You do not have to worry, the sample is in the following section).
Simply run like this.
$ deno run -A main.ts
Note: If you want to scan all storage and databases from the beginning, please run with --reset

My configuration file is as follows:

    "peers": [
        {
            "type": "storage",
            "group": "main",
            "name": "HS-BT-inote",
            "baseDir": "/volume2/docker-data/bt/www/wwwroot/static.hsbt.xyz/",
            "syncFromRemote": true,
            "syncToRemote": true,
            "scanOfflineChanges": true
        },
        {
            "type": "couchdb",
            "name": "HomeStation-couchDB-iNote",
            "group": "main",
            "database": "inote",
            "username": "admin",
            "password": "xxxx",
            "url": "http://192.168.10.107:17888",
            "customChunkSize": 100,
            "minimumChunkSize": 20,
            "passphrase": "",
            "obfuscatePassphrase": "",
            "baseDir": "",
            "syncFromRemote": true,
            "syncToRemote": true,
            "scanOfflineChanges": true
        }
    ]
}

The execution log is:

root@HomeStation:/volume2/docker-data/couchdb/livesync-bridge# deno run -A main.ts
LiveSync Bridge is now starting...
LiveSync Bridge is now started!
7/16/2024, 9:59:09 AM	1	Cache initialized 300 / 10000000000000
7/16/2024, 9:59:09 AM	1	Cache initialized 300 / 10000000000000
7/16/2024, 9:59:09 AM	1	Cache initialized 300 / 50000000
7/16/2024, 9:59:09 AM	10	[HS-BT-inote] Scan offline changes: Enabled, now starting...
7/16/2024, 9:59:09 AM	-1	Requesting ... get http://192.168.10.107:17888/inote/_local%2Fobsydian_livesync_milestone
7/16/2024, 9:59:09 AM	10	[HomeStation-couchDB-iNote] Watch starting from now
7/16/2024, 9:59:09 AM	10	WATCH: START: (since:now)
7/16/2024, 9:59:09 AM	-1	Requesting ... get http://192.168.10.107:17888/inote/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=now&feed=continuous&timeout=100000&heartbeat=5000
7/16/2024, 9:59:09 AM	10	WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...
7/16/2024, 9:59:19 AM	10	WATCH: START: (since:now)
7/16/2024, 9:59:19 AM	-1	Requesting ... get http://192.168.10.107:17888/inote/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=now&feed=continuous&timeout=100000&heartbeat=5000
7/16/2024, 9:59:19 AM	10	WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...
7/16/2024, 9:59:29 AM	10	WATCH: START: (since:now)
7/16/2024, 9:59:29 AM	-1	Requesting ... get http://192.168.10.107:17888/inote/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=now&feed=continuous&timeout=100000&heartbeat=5000
7/16/2024, 9:59:29 AM	10	WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...

End to End Encryption breaks CouchDB <-> Storage Peers

I successfully created a bridge between a couchdb and storage type peer of a very simple Obsidian vault. However, when I tried to then use end to end encryption, it resulted in errors. All my other vaults have end to end encryption and I wasn't able to get a bridge setup for those at all. This is why I tried a simple vault without encryption. I believe encryption is somehow not supported? If not, would it be added at some point?

My config.json and error log are below to review. Thanks for this project, it's the last piece that makes Obsidian able to be self-hosted with a headless sync to a file system that is backed up via traditional techniques.

config.json:

{
    "peers": [
        {
            "type": "couchdb",
            "name": "livesync-bridge-couchdb",
            "database": "mysync",
            "username": "deprecated",
            "password": "deprecated",
            "url": "http://192.168.1.1:1234",
            "passphrase": "deprecated",
            "customChunkSize": 100,
            "minimumChunkSize": 20,
            "obfuscatePassphrase": "",
            "baseDir": ""
        },
        {
            "type": "storage",
            "name": "livesync-bridge-storage",
            "baseDir": "/livesync-bridge/mysync",
            "scanOfflineChanges": true
        }
    ]
}

error.log

2/23/2024, 10:50:37 PM	1	WATCH: SKIP obsydian_livesync_version: Not a document
2/23/2024, 10:50:37 PM	1	Couldn't decode! You should wrong the passphrases
Error: Encrypted data corrupted!
    at decrypt (file:///livesync-bridge/repo/livesync-bridge/lib/src/e2ee_v2.ts:232:19)
    at DirectFileManipulator.decryptDocumentPath (file:///livesync-bridge/repo/livesync-bridge/lib/src/DirectFileManipulator.ts:156:26)
    at DirectFileManipulator.processJSONL (file:///livesync-bridge/repo/livesync-bridge/lib/src/DirectFileManipulator.ts:463:45)
    at DirectFileManipulator.beginWatch (file:///livesync-bridge/repo/livesync-bridge/lib/src/DirectFileManipulator.ts:510:36)
    at eventLoopTick (ext:core/01_core.js:153:7)
2/23/2024, 10:50:37 PM	1	Error:Encrypted data corrupted!
2/23/2024, 10:50:37 PM	1	WATCH: SOMETHING WENT WRONG ON EACH PROCESS
Error: Encrypted data corrupted!
    at decrypt (file:///livesync-bridge/repo/livesync-bridge/lib/src/e2ee_v2.ts:232:19)
    at DirectFileManipulator.decryptDocumentPath (file:///livesync-bridge/repo/livesync-bridge/lib/src/DirectFileManipulator.ts:156:26)
    at DirectFileManipulator.processJSONL (file:///livesync-bridge/repo/livesync-bridge/lib/src/DirectFileManipulator.ts:463:45)
    at DirectFileManipulator.beginWatch (file:///livesync-bridge/repo/livesync-bridge/lib/src/DirectFileManipulator.ts:510:36)
    at eventLoopTick (ext:core/01_core.js:153:7)
2/23/2024, 10:50:37 PM	1	Error:Encrypted data corrupted!
2/23/2024, 10:50:37 PM	1	{"seq":"4-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5kTkXKMBumpKYampqgK4Yh_Y8FiDJ0ACk_kNNYYSYYmRknpJkiK4nCwBQhip-","id":"Test1.md","changes":[{"rev":"1-00b247a18d6e284b85b416b20dca256e"}],"doc":{"_id":"Test1.md","_rev":"1-00b247a18d6e284b85b416b20dca256e","children":["h:+1sv5w1hjhek03"],"path":"Test1.md","ctime":1708714883053,"mtime":1708714883053,"size":20,"type":"plain"}}
2/23/2024, 10:50:37 PM	1	WATCH: SKIP _design/replicate: Not a document
2/23/2024, 10:51:16 PM	1	Couldn't decode! You should wrong the passphrases
Error: Encrypted data corrupted!

Multiples problems with Livesync-bridge

I'm not sure what to write in this issue, because the behavior of the program changes quite often.

In one case it launches like this:

PS [...]/livesync-bridge> deno run -A main.ts
LiveSync Bridge is now starting...
LiveSync Bridge is now started!
29.01.2024 22:21:21     1       Cache initialized 300 / 10000000000000
29.01.2024 22:21:21     1       Cache initialized 300 / 50000000
29.01.2024 22:21:21     1       Cache initialized 300 / 10000000000000
29.01.2024 22:21:21     -1      Requesting ... get https://[***]/obsidian/_local%2Fobsydian_livesync_milestone
29.01.2024 22:21:21     10      [storage-obsidian] Scan offline changes: Enabled, now starting...
29.01.2024 22:21:21     10      [storage-obsidian] Scan offline changes: Finished
29.01.2024 22:21:22     10      [obsidian] Watch starting from 4185-g1AAAACheJzLYWBgYMpgTmEQTM4vTc5I[***]wxyQVCJDUv3___-zMpiTGBg4vucCxdhTzY1NTJOTsenBY1IeC5BkaABS_-E[***]TA1MUrDpjULACVMKYw
29.01.2024 22:21:22     10      WATCH: START: (since:4185-g1AAAACheJzLYWBgYMpgTmEQTM4vTc5I[***]wxyQVCJDUv3___-zMpiTGBg4vucCxdhTzY1NTJOTsenBY1IeC5BkaABS_-E[***]TA1MUrDpjULACVMKYw)
29.01.2024 22:21:22     -1      Requesting ... get https://obsidian.canardporation.com//obsidian/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=4185-g1AAAACheJzLYWBgYMpgTmEQTM4vTc5I[***]wxyQVCJDUv3___-zMpiTGBg4vucCxdhTzY1NTJOTsenBY1IeC5BkaABS_-E[***]TA1MUrDpjULACVMKYw&feed=continuous&timeout=100000&heartbeat=5000

And it stays stuck like that indefinitely (I left it running for 20 minutes) without doing anything.

In another case it seems to work, it replies? well the files, but there are a lot of files that I assume are temporary in the storage, which pollutes the storage folder a lot.

Files and folders look like this: %6c69cb467b42342ead6f240746dbd7414103147efddc038c3f32a762643a3a2570rjA1AbrQpWh76pXMfNOkAtdedcviwobjUbym0=

And yet another case, the process works, there are my files (always with a lot of temporary files) and it ends up doing:

error: Uncaught (in promise) TypeError: error sending request for url (https://[***]//obsidian/f%3Ac278c2dbe815b3f3c9546866c851b8dd9ae61f84351dbaeb83c2ac94ce1793ed): connection error: stream closed because of a broken pipe
        return await fetch(requestURI, opt);
               ^
    at async mainFetch (ext:deno_fetch/26_fetch.js:170:12)
    at async fetch (ext:deno_fetch/26_fetch.js:391:7)
    at async DirectFileManipulator._fetch (file:///[***]/livesync-bridge/lib/src/DirectFileManipulator.ts:81:16)
    at async DirectFileManipulator._fetchJson (file:///[***]/livesync-bridge/lib/src/DirectFileManipulator.ts:85:23)
    at async DirectFileManipulator.getById (file:///[***]/livesync-bridge/lib/src/DirectFileManipulator.ts:199:26)
    at async DirectFileManipulator.get (file:///[***]/livesync-bridge/lib/src/DirectFileManipulator.ts:185:21)
    at async PeerCouchDB.put (file:///[***]/livesync-bridge/PeerCouchDB.ts:45:21)
    at async Hub.dispatch (file:///[***]/livesync-bridge/Hub.ts:41:27)
    at async file:///[***]/livesync-bridge/PeerStorage.ts:169:17
    at async _runWithLock (file:///[***]/livesync-bridge/lib/src/lock.ts:79:16)

And finally, it worked once, but only on the last file saved in the database, but it replicated the changes as needed:

LiveSync Bridge is now starting...
LiveSync Bridge is now started!
29.01.2024 22:16:25     1       Cache initialized 300 / 10000000000000
29.01.2024 22:16:25     1       Cache initialized 300 / 50000000
29.01.2024 22:16:25     1       Cache initialized 300 / 10000000000000
29.01.2024 22:16:25     -1      Requesting ... get https://[***]/_local%2Fobsydian_livesync_milestone
29.01.2024 22:16:25     10      [storage-obsidian] Scan offline changes: Enabled, now starting...
29.01.2024 22:16:25     10      [storage-obsidian] Scan offline changes: Finished
29.01.2024 22:16:25     10      [obsidian] Watch starting from 4176-g1AAAACheJzLYWBgY[***]wNDLXMwBCwxyQVCJDUv3___-zMpiTGBg4PuYCxdhTzY1NTJOTsenBY1IeC5BkaABS_-EGsseDDbQwNTA1MUrDpjULACI0KYM
29.01.2024 22:16:25     10      WATCH: START: (since:4176-g1AAAACheJzLYWBgY[***]wNDLXMwBCwxyQVCJDUv3___-zMpiTGBg4PuYCxdhTzY1NTJOTsenBY1IeC5BkaABS_-EGsseDDbQwNTA1MUrDpjULACI0KYM)
29.01.2024 22:16:25     -1      Requesting ... get https://[***]//obsidian/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=4176-g1AAAACheJzLYWBgY[***]wNDLXMwBCwxyQVCJDUv3___-zMpiTGBg4PuYCxdhTzY1NTJOTsenBY1IeC5BkaABS_-EGsseDDbQwNTA1MUrDpjULACI0KYM&feed=continuous&timeout=100000&heartbeat=5000
29.01.2024 22:16:25     1       WATCH: PROCESSING: 2024-01-28.md
29.01.2024 22:16:25     -1      Collecting chunks: 64
29.01.2024 22:16:25     -1      Requesting ... post https://[***]//obsidian/_all_docs?include_docs=true
29.01.2024 22:16:26     -1      Chunks retrieved (64 / 64)
29.01.2024 22:16:26     10      GET: DONE (META): 2024-01-28.md
29.01.2024 22:16:26     10      [obsidian] --> 2024-01-28.md change detected
29.01.2024 22:16:26     10      [storage-obsidian] <-- obsidian/2024-01-28.md saved
29.01.2024 22:16:26     10      WATCH: PROCESS DONE: 2024-01-28.md
29.01.2024 22:16:26     10      [storage-obsidian] --> 2024-01-28.md change detected
29.01.2024 22:16:41     1       WATCH: PROCESSING: 2024-01-28.md
29.01.2024 22:16:41     -1      Collecting chunks: 64
29.01.2024 22:16:41     -1      Requesting ... post https://[***]//obsidian/_all_docs?include_docs=true
29.01.2024 22:16:41     -1      Chunks retrieved (1 / 64)
29.01.2024 22:16:41     10      GET: DONE (META): 2024-01-28.md
29.01.2024 22:16:41     10      [obsidian] --> 2024-01-28.md change detected
29.01.2024 22:16:41     10      [storage-obsidian] <-- obsidian/2024-01-28.md saved
29.01.2024 22:16:41     10      WATCH: PROCESS DONE: 2024-01-28.md
29.01.2024 22:16:41     10      [storage-obsidian] --> 2024-01-28.md change detected
29.01.2024 22:16:44     1       WATCH: PROCESSING: 2024-01-28.md
29.01.2024 22:16:44     -1      Collecting chunks: 64
29.01.2024 22:16:44     -1      All chunks has been found on cache.
29.01.2024 22:16:44     10      GET: DONE (META): 2024-01-28.md
29.01.2024 22:16:44     10      [obsidian] --> 2024-01-28.md change detected
29.01.2024 22:16:44     10      [storage-obsidian] <-- obsidian/2024-01-28.md saved
29.01.2024 22:16:44     10      WATCH: PROCESS DONE: 2024-01-28.md
29.01.2024 22:16:44     10      [storage-obsidian] --> 2024-01-28.md change detected
29.01.2024 22:16:44     1       GET: START: 2024-01-28.md
29.01.2024 22:16:44     -1      Requesting ... get https://[***]//obsidian/f%3Aad890f0c8[***]b563b93e3490f00955b62ccca97a1ba1dc5ba7f026ba
29.01.2024 22:16:44     10      GET: DONE: 2024-01-28.md
29.01.2024 22:16:44     1       PUT: START: 2024-01-28.md
29.01.2024 22:16:44     -1      Collecting chunks: 1
29.01.2024 22:16:44     -1      Requesting ... post https://[***]//obsidian/_all_docs
29.01.2024 22:16:44     -1      Chunks retrieved (1 / 1)
29.01.2024 22:16:44     10      PUT: All chunks:1, Upload chunk: 1
29.01.2024 22:16:44     -1      Requesting ... post https://[***]//obsidian/_bulk_docs
29.01.2024 22:16:44     -1      Requesting ... get https://[***]//obsidian/f%3Aad890f0c8[***]b563b93e3490f00955b62ccca97a1ba1dc5ba7f026ba
29.01.2024 22:16:45     1       PUT: UPLOADING: 2024-01-28.md
29.01.2024 22:16:45     -1      Requesting ... put https://[***]//obsidian/f%3Aad890f0c8[***]b563b93e3490f00955b62ccca97a1ba1dc5ba7f026ba
29.01.2024 22:16:45     10      PUT: DONE: 2024-01-28.md
29.01.2024 22:16:45     10      [obsidian] <--  2024-01-28.md saved
29.01.2024 22:16:45     1       WATCH: PROCESSING: %6c69cb467b42342ead6f240746dbd741[***]a762643a3a2570rjA1AbrQpWh76pXMfNOkAtdedcviwobjUbym0=
29.01.2024 22:16:45     -1      Collecting chunks: 1
29.01.2024 22:16:45     -1      Requesting ... post https://[***]//obsidian/_all_docs?include_docs=true
29.01.2024 22:16:45     -1      Chunks retrieved (1 / 1)
29.01.2024 22:16:45     10      GET: DONE (META): %6c69cb467b42342ead6f240746dbd741[***]a762643a3a2570rjA1AbrQpWh76pXMfNOkAtdedcviwobjUbym0=
29.01.2024 22:16:45     10      [obsidian] --> %6c69cb467b42342ead6f240746dbd741[***]a762643a3a2570rjA1AbrQpWh76pXMfNOkAtdedcviwobjUbym0=change detected
29.01.2024 22:16:45     10      [storage-obsidian] <-- obsidian/%6c69cb467b42342ead6f240746dbd741[***]a762643a3a2570rjA1AbrQpWh76pXMfNOkAtdedcviwobjUbym0= saved
29.01.2024 22:16:45     10      WATCH: PROCESS DONE: %6c69cb467b42342ead6f240746dbd741[***]a762643a3a2570rjA1AbrQpWh76pXMfNOkAtdedcviwobjUbym0=
29.01.2024 22:16:45     10      [storage-obsidian] --> %6c69cb467b42342ead6f240746dbd741[***]a762643a3a2570rjA1AbrQpWh76pXMfNOkAtdedcviwobjUbym0= change detected

In all of these cases, it doesn't work "enough" to be usable. I tried to understand why but I couldn't figure it out. I would like to point out that I use Obsidian-LiveSync during all these tests and that it works wonderfully.

Specifications

CouchDB v3.3.3-1~bullseye

CouchDB's response to https://[***]/_all_dbs: ["obsidian"]

I'll say it again but replication on several devices with obsidian-livesync works flawlessly, so I would say that it's livesync-bridge which has a problem...

I'm on the main branch of this project for my tests.

The configuration of dat/config.json:

{
    "peers": [
        {
            "type": "couchdb",
            "name": "obsidian",
            "database": "obsidian",
            "username": "[***]",
            "password": "[***]",
            "url": "https://[***]/",
            "customChunkSize": 100,
            "minimumChunkSize": 20,
            "passphrase": "",
            "obfuscatePassphrase": "[***]",
            "baseDir": ""
        },
        {
          "type": "storage",
          "name": "storage-obsidian",
          "baseDir": "./obsidian/",
          "scanOfflineChanges": true
        }
    ]
}

Hope you can help me :)

Directory Folder rename breaks sync

When renaming a folder, example "developer" -> "Developer Notes" the connection between bridge and Obsidian sync clients seems to come to a halt.

Old directory is re Synce

Also the old directory likes to come back, or doesn't remove itself on the server side "storage" folder. Example, I rename "developer" -> "Developer Notes", but the server side storage folder will now have both "developer" and "Developer Notes" directories.

A later sync even brings back the old "developer" folder into my Obsidian App vault.

Bridge Docker Logs

Looks like when I edit a note I still get activity in Bridge's logs, but going to the Obsidian App running the Self Hosted Live Sync

2/19/2024, 9:18:03 PM	-1	Requesting ... post https://obsidian-db.MYDOMAIN.com/obsidian_db/_all_docs?include_docs=true
2/19/2024, 9:18:03 PM	-1	Chunks retrieved (2 / 2)
2/19/2024, 9:18:03 PM	10	GET: DONE (META): Recently edited.md
2/19/2024, 9:18:03 PM	10	[pywriter] --> New Note.md change detected
2/19/2024, 9:18:03 PM	10	WATCH: PROCESS DONE: New Note.md

Work around

It could be inpatients, but things seem to work after a few docker compose restart on the Bridge app and closing, restarting the Self Hosted Live Sync plugin.

Renaming a Directory

Is there any "best practice" for renaming a folder, especially if it nests many other sub directories so that Bridge and Live Sync behave nicely together? Should I stop the sync, rename file, and rebuild the remote db to match my local db anytime I need to rename a folder?

How should I proceed for my other Obsidian App devices (i.e. mobile)? Do I need to rebuild local from the remote db as well?