doomhammer/roam-to-git

Automatic RoamResearch backup to Git

Automatic RoamResearch backup

This script helps you backup your RoamResearch graphs!

This script automatically

• Downloads a markdown archive of your RoamResearch workspace
• Downloads a json archive of your RoamResearch workspace
• Unzips them to your git directory
• Commits and pushes the difference to Github

Demo

See it in action!. This repo is updated using roam-to-git.

Why to use it

• You have a backup if RoamResearch loses some of your data.
• You have a history of your notes.
• You can browse your Github repository easily with a mobile device

Use it with Github Actions (recommended)

Create a (private) Github repository for all your notes

With gh: gh repo create notes (yes, it's private)

Or manually

Configure Github secrets

• Go to github.com/your/repository/settings/secrets

Add 3 (separate) secrets where the names are

ROAMRESEARCH_USER

ROAMRESEARCH_PASSWORD

ROAMRESEARCH_DATABASE

• Refer to env.template for more information

• when inserting the information, there is no need for quotations or assignments

Add GitHub action

cd notes
mkdir -p .github/workflows/
curl https://raw.githubusercontent.com/MatthieuBizien/roam-to-git-demo/master/.github/workflows/main.yml > \
    .github/workflows/main.yml
git add .github/workflows/main.yml
git commit -m "Add github/workflows/main.yml"
git push --set-upstream origin master

Check that the Github Action works

• Go to github.com/your/repository/actions
• Your CI job should start in a few seconds

Note:

If the backup does not automatically start, try pushing to the repository again

Use it locally

Note: if your file system is not case-sensitive, you will not backup notes that have the same name in different cases

Install Roam-To-Git

With pipx (if you don't know pipx, you should look at it, it's wonderful!)

pipx install git+https://github.com/MatthieuBizien/roam-to-git.git

Create a (private) Github repository for all your notes

With gh: gh repo create notes (yes, it's private)

Or manually

Then run git push --set-upstream origin master

Configure environment variables

• curl https://raw.githubusercontent.com/MatthieuBizien/roam-to-git/master/env.template > notes/.env
• Fill the .env file: vi .env
• Ignore it: echo .env > notes/.gitignore; cd notes; git add .gitignore; git commit -m "Initial commit"

Manual backup

• Run the script: roam-to-git notes/
• Check your Github repository, it should be filled with your notes :)

Automatic backup

One-liner to run it with a cron every hours: echo "0 * * * * '$(which roam-to-git)' '$(pwd)/notes'" | crontab -

NB: there are issues on Mac with a crontab.

Debug

Making roam-to-git foolproof is hard, as it depends on Roam, on Github Action or the local environment, on software not very stable (pyppeteer we still love you 😉 ) and on the correct user configuration.

For debugging, please try the following:

• Check that the environment variables ROAMRESEARCH_USER, ROAMRESEARCH_PASSWORD, ROAMRESEARCH_DATABASE are correctly setup
• Login into Roam using the username and the password. You may want to ask a new password if you have enabled Google Login, as it solved some user problems.
• Run roam-to-git --debug to check the authentification and download work
• Look at the traceback
• Look for similar issues
• If nothing else work, create a new issue with as many details as possible. I will try my best to understand and help you, no SLA promised 😇

Task list

Backup all RoamResearch data

• Download automatically from RoamResearch
• Create Cron
• Write detailed README
• Publish the repository on Github
• Download images (they currently visible in Github, but not in the archive so not saved in the repository 😕)

Format the backup to have a good UI

Link formatting to be compatible with Github markdown

• Format [[links]]
• Format #links
• Format attribute::
• Format [[ [[link 1]] [[link 2]] ]]
• Format ((link))

Backlink formatting

• Add backlinks reference to the notes files
• Integrate the context into the backlink
• Manage / in file names

Other formatting

• Format {{TODO}} to be compatible with Github markdown
• Format `{{query}}``

Make it for others

• Push it to Github
• Add example repository
• Make the backup directory configurable
• Publicize it
  • RoamResearch Slack thread
  • RoamResearch Reddit
  • Twitter

Some ideas, I don't need it, but PR welcome 😀

• Test it/make it work on Windows
• Pre-configure a CI server so it can run every hour without a computer Thanks @Stvad for #4!

roam-to-git's Issues

Github: Roam Research backup: All jobs have failed May 2023

Describe the bug
Hi, the workflow was working till now with no problems for over a year. But suddenly for the last 2 weeks, it's not updating the GitHub anymore. No other changes were made to any parts. GitHub actions says getting requirements to build wheel did not run successfully, the process completed with exit code 1.

To Reproduce
Steps to reproduce the behavior:
No other changes to the workflow

Traceback

Please complete the following information:

• OS: Windows
• Do you use Github Action?
• Yes
• Do you use multiple Roam databases?
• No
• Does roam-to-git use to work for you? When precisely did it stopped to work?
• Was workin for more than a year, stopped 2 weeks ago.
• Does some backup runs are still working?
• No

Additional context

Run pip install git+https://github.com/DoomHammer/roam-to-git.git@roam-to-garden
Collecting git+https://github.com/DoomHammer/roam-to-git.git@roam-to-garden
Cloning https://github.com/DoomHammer/roam-to-git.git (to revision roam-to-garden) to /tmp/pip-req-build-6dmq91cz
Running command git clone --filter=blob:none --quiet https://github.com/DoomHammer/roam-to-git.git /tmp/pip-req-build-6dmq91cz
Running command git checkout -b roam-to-garden --track origin/roam-to-garden
Switched to a new branch 'roam-to-garden'
branch 'roam-to-garden' set up to track 'origin/roam-to-garden'.
Resolved https://github.com/DoomHammer/roam-to-git.git to commit 384dd55
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'error'
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [43 lines of output]
Traceback (most recent call last):
File "/tmp/pip-build-env-i7yonrzu/overlay/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 35, in init
parsed = parse_requirement(requirement_string)
File "/tmp/pip-build-env-i7yonrzu/overlay/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/_parser.py", line 64, in parse_requirement
return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
File "/tmp/pip-build-env-i7yonrzu/overlay/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/_parser.py", line 82, in _parse_requirement
url, specifier, marker = _parse_requirement_details(tokenizer)
File "/tmp/pip-build-env-i7yonrzu/overlay/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/_parser.py", line 126, in _parse_requirement_details
marker = _parse_requirement_marker(
File "/tmp/pip-build-env-i7yonrzu/overlay/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/_parser.py", line 147, in _parse_requirement_marker
tokenizer.raise_syntax_error(
File "/tmp/pip-build-env-i7yonrzu/overlay/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/_tokenizer.py", line 163, in raise_syntax_error
raise ParserSyntaxError(
pkg_resources.extern.packaging._tokenizer.ParserSyntaxError: Expected end or semicolon (after version specifier)
gitpython>=3.1.*
~~~~~^

  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-i7yonrzu/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-i7yonrzu/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-i7yonrzu/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 487, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/tmp/pip-build-env-i7yonrzu/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 16, in <module>
    File "<string>", line 16, in <listcomp>
    File "/tmp/pip-build-env-i7yonrzu/overlay/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3173, in __init__
      super(Requirement, self).__init__(requirement_string)
    File "/tmp/pip-build-env-i7yonrzu/overlay/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 37, in __init__
      raise InvalidRequirement(str(e)) from e
  pkg_resources.extern.packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier)
      gitpython>=3.1.*
               ~~~~~^
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Notice: A new release of pip is available: 22.0.4 -> 23.1.2
Notice: To update, run: pip install --upgrade pip
Error: Process completed with exit code 1.

Note titles still contain quotation marks and commas, which makes builds and "bundle" fail on Windows

Describe the bug
I'm running into trouble trying to clone the repo locally to run "bundle", to be able to add dependencies (in this case nokogiri but irrelevant). There's a "error: invalid path" message, which is caused by the fact that a filename "/_notes/activists-should..." contains a comma, and quotation marks, which is forbidden on Windows. It only worked once I removed both from the note title manually.

Expected behavior
Have Windows-forbidden characters removed from note titles automatically.

Please complete the following information:

• OS: Windows 10
• Do you use Github Action? Yes
• Do you use multiple Roam databases? No, only one in this repo
• Does roam-to-git use to work for you? When precisely did it stopped to work?

GitHub says "Roam Research backup: All jobs have failed" after any .md file changed via GitHub

Describe the bug
Within one minute after saving a change to a Markdown file in _pages in GitHub, an error email from GitHub comes:

Clicking through to GitHub, see this:

Some 6 minutes later, (in the past, within a minute) get another email:

This has occured multiple times, whenever I changed any Markdown file via GitHub.

To Reproduce
Steps to reproduce the behavior:

1. Change index.md or about.md directly within the GitHub site via a browser (Firefox).
2. Check live site using Maxime Vaillancourt's template after a minute: Changes were published successfully.
3. Get an email from GitHub notifications saying "[kolanuts/digital-garden] Run failed: Roam Research backup - main (f1558ec)"

Expected behavior
I'd expect there to be no failure message, and for the backup to succeed.

Traceback

Run roam-to-git --debug notes/ and report what you get.
It should open a Chrome front-end, an do the scrapping. The repository content will not be modified. If applicable, add screenshots to help explain your problem.

• OS: Win10
• Do you use Github Action? Yes
• Do you use multiple Roam databases? Yes, but only 1 is connected with my repo of course
• Does roam-to-git use to work for you? (This should be "Did roam-to-git work for you previously?") I think this has happened every time, but the updates are still pushed live!
• Does some backup runs are still working? (This should be "Are some backup runs still working?") I think... not? Not sure how to see this. But screenshot under Traceback suggests backups are not working.

Working Github actions starts failing now with AttributeError, error in MainThread

Describe the bug
I was using the workflow for about 3 weeks, but suddenly from 3 days back, all the jobs have been failing. They have the same error. I tried resetting the GitHub secrets but does not work.
To Reproduce
Steps to reproduce the behavior:
Don't know what happened, I did not change any settings before the jobs started failing.

Traceback

Run roam-to-git --skip-fetch --skip-git .
roam-to-git --skip-fetch --skip-git .
shell: /usr/bin/bash -e {0}
env:
pythonLocation: /opt/hostedtoolcache/Python/3.8.11/x64
ROAMRESEARCH_USER: ***
ROAMRESEARCH_PASSWORD: ***
ROAMRESEARCH_DATABASE: ***
2021-08-18 17:16:35.729 | ERROR | main::33 - An error has been caught in function '', process 'MainProcess' (1912), thread 'MainThread' (139748815165248):
Traceback (most recent call last):

File "/opt/hostedtoolcache/Python/3.8.11/x64/bin/roam-to-git", line 33, in
sys.exit(load_entry_point('roam-to-git==0.1', 'console_scripts', 'roam-to-git')())
│ │ └ <function importlib_load_entry_point at 0x7f19ce30d4c0>
│ └
└ <module 'sys' (built-in)>
File "/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/roam_to_git/main.py", line 76, in main
patch_pyppeteer()
└ <function patch_pyppeteer at 0x7f19cd4823a0>
File "/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/roam_to_git/scrapping.py", line 18, in patch_pyppeteer
original_method = pyppeteer.connection.websockets.client.connect
│ │ └ <module 'websockets' from '/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/websockets/init.py'>
│ └ <module 'pyppeteer.connection' from '/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/pyppeteer/connection....
└ <module 'pyppeteer' from '/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/pyppeteer/init.py'>
File "/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/websockets/imports.py", line 96, in getattr
raise AttributeError(f"module {package!r} has no attribute {name!r}")

AttributeError: module 'websockets' has no attribute 'client'
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.8.11/x64/bin/roam-to-git", line 33, in
sys.exit(load_entry_point('roam-to-git==0.1', 'console_scripts', 'roam-to-git')())
File "/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/loguru/_logger.py", line 1149, in catch_wrapper
return function(*args, **kwargs)
File "/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/roam_to_git/main.py", line 76, in main
patch_pyppeteer()
File "/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/roam_to_git/scrapping.py", line 18, in patch_pyppeteer
original_method = pyppeteer.connection.websockets.client.connect
File "/opt/hostedtoolcache/Python/3.8.11/x64/lib/python3.8/site-packages/websockets/imports.py", line 96, in getattr
raise AttributeError(f"module {package!r} has no attribute {name!r}")
AttributeError: module 'websockets' has no attribute 'client'
Error: Process completed with exit code 1.

Run roam-to-git --debug notes/ and report what you get.
Don't know how to run this, where should I run the command?

Please complete the following information:

• OS: [Win 10]
• Do you use Github Action? Yes
• Do you use multiple Roam databases? only one
• Does roam-to-git use to work for you? When precisely did it stopped to work? was working for 3 weeks, stopped on 16 Aug 2021
• Does some backup runs are still working? No

Additional context
Add any other context about the problem here.

Needed ability to publish all notes if “Public.md” is not found

roam-to-git/roam_to_git/formatter.py

Line 79 in 384dd55

def get_allowed_notes(dir: Path) -> List[str]:

I Wish few lines of codes added in the above referred function to give option whether to publish all the notes or just publish only notes that is referred in Public.md

Is it possible to publish whole notes if I don’t have a Public.md file ?