{"id":13726530,"url":"https://github.com/MatthieuBizien/roam-to-git","last_synced_at":"2025-05-07T21:33:03.450Z","repository":{"id":45872744,"uuid":"243395909","full_name":"MatthieuBizien/roam-to-git","owner":"MatthieuBizien","description":"Automatic RoamResearch backup to Git","archived":false,"fork":false,"pushed_at":"2024-10-31T17:51:36.000Z","size":469,"stargazers_count":550,"open_issues_count":21,"forks_count":133,"subscribers_count":20,"default_branch":"master","last_synced_at":"2024-10-31T18:31:38.498Z","etag":null,"topics":["backlink","backup","cron","export","github","roam","roamresearch","zettelkasten"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MatthieuBizien.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-02-27T00:33:46.000Z","updated_at":"2024-10-31T17:51:42.000Z","dependencies_parsed_at":"2024-01-10T19:10:46.287Z","dependency_job_id":null,"html_url":"https://github.com/MatthieuBizien/roam-to-git","commit_stats":{"total_commits":91,"total_committers":15,"mean_commits":6.066666666666666,"dds":0.4065934065934066,"last_synced_commit":"3878e8fef6a418f039521bae194f3f3a1f39a835"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatthieuBizien%2Froam-to-git","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatthieuBizien%2Froam-to-git/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatthieuBizien%2Froam-to-git/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatthieuBizien%2Froam-to-git/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MatthieuBizien","download_url":"https://codeload.github.com/MatthieuBizien/roam-to-git/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224654308,"owners_count":17347721,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backlink","backup","cron","export","github","roam","roamresearch","zettelkasten"],"created_at":"2024-08-03T01:03:10.981Z","updated_at":"2024-11-14T16:34:03.256Z","avatar_url":"https://github.com/MatthieuBizien.png","language":"Python","funding_links":[],"categories":["Python","Misc"],"sub_categories":[],"readme":"# Automatic RoamResearch backup\n\n[![Roam Research backup](https://github.com/MatthieuBizien/roam-to-git-demo/workflows/Roam%20Research%20backup/badge.svg)](https://github.com/MatthieuBizien/roam-to-git-demo/actions)\n[![roam-to-git tests.py](https://github.com/MatthieuBizien/roam-to-git/workflows/roam-to-git%20tests.py/badge.svg)](https://github.com/MatthieuBizien/roam-to-git/actions)\n\nThis script helps you backup your [RoamResearch](https://roamresearch.com/) graphs!\n\nThis script automatically\n- Downloads a **markdown archive** of your RoamResearch workspace\n- Downloads a **json archive** of your RoamResearch workspace\n- Download the full **EDN** of your RoamResearch workspace\n- Unzips them to your git directory\n- Format your markdown, including with **backlinks**\n- **Commits and push** the difference to GitHub\n\n# What's new\n\n**V.02:** \n- Use Selenium library, and roam-to-git seems to be much faster and stable 🔥\n- Download the EDN archive\n\n# Demo\n[See it in action!](https://github.com/MatthieuBizien/roam-to-git-demo). This repo is updated using roam-to-git.\n\n🚀🚀 **NEW** 🚀🚀 : The [Unofficial backup](https://github.com/MatthieuBizien/RoamResearch-offical-help/) \nof the official [RoamResearch Help Database](https://roamresearch.com/#/app/help)\n\n# Why to use it\n\n- You have a **backup** if RoamResearch loses some of your data.\n- You have a **history** of your notes.\n- You can **browse** your GitHub repository easily with a mobile device\n\n\n# Use it with GitHub Actions (recommended)\n\n**Note**: [Erik Newhard's guide](https://eriknewhard.com/blog/backup-roam-in-github) shows an easy way of setting up GitHub Actions without using the CLI.\n\n##  Create a (private) GitHub repository for all your notes\n\nWith [gh](https://github.com/cli/cli): `gh repo create notes` (yes, it's private)\n\nOr [manually](https://help.github.com/en/github/getting-started-with-github/create-a-repo)\n\n## Configure GitHub secrets \n\n- Go to github.com/your/repository/settings/secrets \n\n\n##### Regarding Google Account Authorization\n\nDue to the limitations of OAuth and complexities with tokens, we are unable to snapshot accounts that are set up with the *Login with Google* option as of now.\n\nTo set up backup in this case, you will need to *create*(not exactly) a native account from your old Google Account, which is as simple as using the reset password link found in Roam. \n\n![image](https://user-images.githubusercontent.com/46789005/99179188-24482f00-2741-11eb-9c24-df7bb8707709.png)\n\nOnce you've reset your password, use the following steps to finish setting up your backup!\n\n\n### Configuring GitHub Secrets\n\nAdd 3 (separate) secrets where the names are \n\n`ROAMRESEARCH_USER`\n\n`ROAMRESEARCH_PASSWORD`\n\n`ROAMRESEARCH_DATABASE`\n\n- Refer to [env.template](env.template) for more information\n\n- when inserting the information, there is no need for quotations or assignments\n\n![image](https://user-images.githubusercontent.com/173090/90904133-2cf1c900-e3cf-11ea-960d-71d0543b8158.png)\n\n\n## Add GitHub action\n\n```\ncd notes\nmkdir -p .github/workflows/\ncurl https://raw.githubusercontent.com/MatthieuBizien/roam-to-git-demo/master/.github/workflows/main.yml \u003e \\\n    .github/workflows/main.yml\ngit add .github/workflows/main.yml\ngit commit -m \"Add github/workflows/main.yml\"\ngit push --set-upstream origin master\n```\n\n## Check that the GitHub Action works\n\n- Go to github.com/your/repository/actions\n- Your CI job should start in a few seconds\n\n### Note:\n\nIf the backup does not automatically start, try pushing to the repository again\n\n# Use with GitLab CI\n\nThis section is based on this article on GitLab blog:\nhttps://about.gitlab.com/blog/2017/11/02/automating-boring-git-operations-gitlab-ci/\n\n## Create a project\n\nCreate a project for your notes. We will refer to it as\n`YOUR_USER/YOUR_PROJECT`.\n\n## Create key pair for pushing commits\n\nGenerate a new key pair that will be used by the CI job to push the new commits.\n\n```bash\n$ ssh-keygen -f gitlab-ci-commit\nGenerating public/private rsa key pair.\nEnter passphrase (empty for no passphrase):\nEnter same passphrase again:\nYour identification has been saved in gitlab-ci-commit-test\nYour public key has been saved in gitlab-ci-commit-test.pub\nThe key fingerprint is:\nSHA256:HoQUcbUPJU2Ur78EineqA6IVljk8ZD9XIxiGFUrBues agentydragon@pop-os\nThe key's randomart image is:\n+---[RSA 3072]----+\n|   .o=O*..o++.   |\n|   .*+.o. o+o    |\n|   +.=. .oo. .   |\n|    X o..  o  .  |\n|   . = oS   o.   |\n|    + .. o ...   |\n|   + . .o o ...  |\n|  . E   .. o ..  |\n|        .o.   .. |\n+----[SHA256]-----+\n```\n\nDO NOT commit the private key (`gitlab-ci-commit`).\n\n## Add the public key as a deploy key\n\nThis step allows the CI job to push when identified by the public key.\n\nGo to Project Settings \u0026rarr; Repository (`https://gitlab.com/YOUR_USER/YOUR_PROJECT/-/settings/repository`) \u0026rarr; Deploy Keys.\n\nPaste the content of the public key file (`gitlab-ci-commit-test.pub`), and\nenable write access for the public key.\n\n![](img/gitlab-add-key.png)\n\nClick \"Add Key\".\n\n## Add the private key as a CI variable\n\nThis step gives the CI job the private key so it can authenticate against\nGitLab.\n\nGo to Project Settings \u0026rarr; Pipelines (`https://gitlab.com/YOUR_USER/YOUR_PROJECT/-/settings/ci_cd`) \u0026rarr; Variables.\n\nClick \"Add variable\", with name `GIT_SSH_PRIV_KEY`, and paste in the content\nof the private key file (`gitlab-ci-commit`). You probably want to mark\n\"Protect\". You might want to look up GitLab docs on protected branches.\n\n![](img/gitlab-add-variable.png)\n\nClick \"Add variable\".\n\nAlso add the following variables with appropriate values:\n\n* `ROAMRESEARCH_USER`\n* `ROAMRESEARCH_PASSWORD`\n* `ROAMRESEARCH_DATABASE`\n\n## Create a `gitlab_known_hosts` file\n\nIn your repo, create and commit a `gitlab_known_hosts` file containing the\nneeded SSH `known_hosts` entry/entries for the GitLab instance. It will be used\nby the CI job to check that it's talking to the right server.\n\nThis should work as of 2021-01-04:\n\n```\n# Generated by @agentydragon at 2021-01-04 at by `git fetch`ing a GitLab repo\n# with an empty ~/.ssh/known_hosts file.\n\n|1|zIQlCRxv+s9xhVCAfGL2nvaZqdY=|jbPpD9GNaS/9Z4iJzE9gw2XCo20= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=\n|1|uj60xYhsW2vAM8BpQ+xZz51ZarQ=|BNIJlvu4rNcmrxd60fkqpChrf9A= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=\n```\n\n## Create `.gitlab-ci.yml`\n\nCreate a `.gitlab-ci.yml` file:\n\n```yaml\nbackup:\n  when: manual\n  before_script:\n    - 'which ssh-agent || ( apt-get update -y \u0026\u0026 apt-get install openssh-client -y )'\n    - eval $(ssh-agent -s)\n    - ssh-add \u003c(echo \"$GIT_SSH_PRIV_KEY\")\n    - git config --global user.email \"roam-to-git@example.org\"\n    - git config --global user.name \"roam-to-git automated backup\"\n    - mkdir -p ~/.ssh\n    - cat gitlab_known_hosts \u003e\u003e ~/.ssh/known_hosts\n\n    # (Taken from: https://github.com/buildkite/docker-puppeteer/blob/master/Dockerfile)\n    # We install Chrome to get all the OS level dependencies, but Chrome itself\n    # is not actually used as it's packaged in the pyppeteer library.\n    # Alternatively, we could include the entire dep list ourselves\n    # (https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md#chrome-headless-doesnt-launch-on-unix)\n    # but that seems too easy to get out of date.\n    - apt-get install -y wget gnupg ca-certificates\n    - wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -\n    - echo \"deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main\" \u003e\u003e /etc/apt/sources.list.d/google.list\n    - apt-get update\n    - apt-get install google-chrome-stable libxss1 python3-pip -y\n\n    - pip3 install git+https://github.com/MatthieuBizien/roam-to-git.git\n\n    # TODO(agentydragon): Create and publish Docker image with all deps already\n    # installed.\n  script:\n    # Need to clone the repo again over SSH, since by default GitLab clones\n    # the repo for CI over HTTPS, for which we cannot authenticate pushes via\n    # pubkey.\n    - git clone --depth=1 git@gitlab.com:YOUR_USER/YOUR_PROJECT\n    - cd YOUR_PROJECT\n\n    # --no-sandbox needed because Chrome refuses to run as root without it.\n    - roam-to-git --browser-arg=--no-sandbox .\n```\n\nCommit and push.\n\n## To run the pipeline\n\nGo to `https://gitlab.com/YOUR_USER/YOUR_PROJECT/-/jobs`,\npress the Play button:\n\n![](img/gitlab-start-job.png)\n\nShortly after you should see an automated commit added to master.\n\n![](img/gitlab-done.png)\n\n## Scheduled backups\n\nTo run the script, say, every hour, go to the project's\nCI / CD \u0026rarr; Schedules (`https://gitlab.com/YOUR_USER/YOUR_PROJECT/-/pipeline_schedules`).\nClick \"New schedule\", fill out and submit (like this for every 15 minutes):\n\n![](img/gitlab-schedule.png)\n\n# Use it locally\n\n**Note**: if your file system is not case-sensitive, you will not backup notes that have the same name in different \ncases\n\n## Install Roam-To-Git\nWith [pipx](https://github.com/pipxproject/pipx) \n(if you don't know pipx, you should look at it, it's wonderful!)\n\n`pipx install git+https://github.com/MatthieuBizien/roam-to-git.git`\n\n## Create a (private) GitHub repository for all your notes\n\nWith [gh](https://github.com/cli/cli): `gh repo create notes` (yes, it's private)\n\nOr [manually](https://help.github.com/en/github/getting-started-with-github/create-a-repo)\n\nThen run `git push --set-upstream origin master`\n\n## Configure environment variables\n\n- `curl https://raw.githubusercontent.com/MatthieuBizien/roam-to-git/master/env.template \u003e notes/.env`\n- Fill the .env file: `vi .env`\n- Ignore it: `echo .env \u003e notes/.gitignore; cd notes; git add .gitignore; git commit -m \"Initial commit\"`\n\n## Manual backup\n\n- Run the script: `roam-to-git notes/`\n- Check your GitHub repository, it should be filled with your notes :)\n\n## Automatic backup\n\nOne-liner to run it with a [cron](https://en.wikipedia.org/wiki/Cron) every hour: \n`echo \"0 *  *  *  *  '$(which roam-to-git)' '$(pwd)/notes'\" | crontab -`\n\nNB: there are [issues](https://github.com/MatthieuBizien/roam-to-git/issues/43) on Mac with a crontab.\n\n# Debug\n\nMaking `roam-to-git` foolproof is hard, as it depends on Roam, on GitHub Action or the local environment, \non software not very stable (`pyppeteer` we still love you 😉 )\nand on the correct user configuration.\n\nFor debugging, please try the following:\n\n- Check that the environment variables `ROAMRESEARCH_USER`, `ROAMRESEARCH_PASSWORD`, `ROAMRESEARCH_DATABASE` are correctly setup\n- Login into Roam using the username and the password. \nYou may want to ask a new password if you have enabled Google Login, as it solved some user problems.\n- Run `roam-to-git --debug` to check the authentication and download work\n- Look at the traceback\n- Look for similar issues\n- If nothing else work, create a new issue with as many details as possible. \nI will try my best to understand and help you, no SLA promised 😇\n\n# Task list\n\n## Backup all RoamResearch data\n\n- [x] Download automatically from RoamResearch\n- [x] Create Cron\n- [x] Write detailed README\n- [x] Publish the repository on GitHub\n- [ ] Download images (they currently visible in GitHub, but not in the archive so not saved in the repository 😕)\n\n## Format the backup to have a good UI\n\n### Link formatting to be compatible with GitHub markdown\n- [x] Format `[[links]]`\n- [x] Format `#links`\n- [x] Format `attribute::`\n- [ ] Format `[[ [[link 1]] [[link 2]] ]]` \n- [ ] Format `((link))`\n\n### Backlink formatting\n- [x] Add backlinks reference to the notes files\n- [x] Integrate the context into the backlink\n- [x] Manage `/` in file names\n\n### Other formatting\n- [x] Format `{{TODO}}` to be compatible with GitHub markdown\n- [ ] Format `{{query}}``\n\n## Make it for others\n- [x] Push it to GitHub\n- [x] Add example repository\n- [x] Make the backup directory configurable\n- [ ] Publicize it\n    - [x] [RoamResearch Slack](https://roamresearch.slack.com/) [thread](https://roamresearch.slack.com/archives/CN5MK4D2M/p1588670473431200)\n    - [ ] [RoamResearch Reddit](https://www.reddit.com/r/RoamResearch/)\n    - [ ] Twitter\n\n## Some ideas, I don't need it, but PR welcome 😀\n- [ ] Test it/make it work on Windows\n- [x] Pre-configure a CI server so it can run every hour without a computer\n    Thanks @Stvad for [#4](https://github.com/MatthieuBizien/roam-to-git/issues/4)!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMatthieuBizien%2Froam-to-git","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMatthieuBizien%2Froam-to-git","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMatthieuBizien%2Froam-to-git/lists"}