{"id":19279499,"url":"https://github.com/eeditiones/tuttle","last_synced_at":"2025-09-12T21:47:35.920Z","repository":{"id":38197635,"uuid":"424603568","full_name":"eeditiones/tuttle","owner":"eeditiones","description":null,"archived":false,"fork":false,"pushed_at":"2025-08-05T10:11:11.000Z","size":1728,"stargazers_count":10,"open_issues_count":9,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-08-05T12:10:59.679Z","etag":null,"topics":["exist-db","git","github","gitlab","library","sync","xar"],"latest_commit_sha":null,"homepage":"","language":"XQuery","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eeditiones.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"eeditiones","custom":"https://www.paypal.com/paypalme/eeditiones"}},"created_at":"2021-11-04T13:21:23.000Z","updated_at":"2025-08-05T10:11:14.000Z","dependencies_parsed_at":"2024-03-21T17:27:57.429Z","dependency_job_id":"324e7b79-d4f3-474f-a855-102382580cb7","html_url":"https://github.com/eeditiones/tuttle","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/eeditiones/tuttle","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eeditiones%2Ftuttle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eeditiones%2Ftuttle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eeditiones%2Ftuttle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eeditiones%2Ftuttle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eeditiones","download_url":"https://codeload.github.com/eeditiones/tuttle/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eeditiones%2Ftuttle/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274882087,"owners_count":25367398,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-12T02:00:09.324Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["exist-db","git","github","gitlab","library","sync","xar"],"created_at":"2024-11-09T21:15:24.453Z","updated_at":"2025-09-12T21:47:35.909Z","avatar_url":"https://github.com/eeditiones.png","language":"XQuery","readme":" \n# Tuttle - a Git-integration for eXist-db\n\nSynchronizes your data collection with GitHub and GitLab.\n\n## User Documentation\n\n[User Documentation](https://eeditiones.github.io/tuttle-doc/)\n\n## Functionality\n\n* Sync data collection from Git to DB\n* Deal with multiple repositories\n* Incremental updates\n* Works with private or public repositories\n* Works with self hosted instances\n* Extendable to other git services\n\n## Requirements\n\n-  [node](https://nodejs.org/en/): `v22`\n-  [exist-db](https://www.exist-db.org): `v5.5.1+ \u003c 7.0.0`\n\n## Installation\n\nPre-built packages are available\n- as [github-releases](https://github.com/eeditiones/tuttle/releases) \n  ```bash\n  xst package install github-release tuttle --owner eeditiones\n  ```\n\n- and on [exist-db's public package registry](https://exist-db.org/exist/apps/public-repo/packages/tuttle?eXist-db-min-version=5.5.1).\n  ```bash\n  xst package install from-registry tuttle\n  ```\n\n\n## Building from source\n\nTuttle uses Gulp as its build tool which itself builds on NPM. \nTo initialize the project and load dependencies run\n\n```\nnpm install\n```\n\n| Run | Description |\n|---------|-------------|\n|```npm run build``` | builds the Tuttle package |\n|```npm run deploy``` | build and install Tuttle in one go |\n\n\u003e Note: the `deploy` commands below assume that you have a local eXist-db running on port 8080. However the database connection can be configured (see gulp-exist documentation)\n\n## Testing\n\nTo run the local test suite you need\n\n* an instance of eXist running on `localhost:8080` and \n* `npm` to be available in your path\n* a GitHub personal access token with read access to public repositories\n* a gitlab personal access token with read access to public repositories\n\nIn CI these access tokens are read from environment variables.\nYou can do the same with\n```bash\nexport tuttle_token_tuttle_sample_data=\u003cGITHUB_PAT\u003e; \\ \nexport tuttle_token_gitlab_sample_data=\u003cGITLAB_PAT\u003e; \\ \npath/to/startup.sh\n```\n\nAlternatively, you can modify `/db/apps/tuttle/data/tuttle.xml` _and_ `test/fixtures/alt-tuttle.xml`, `test/fixtures/alt-repo-xml-tuttle.xml` to include your tokens. But remember to never commit them!\n\nRun tests with\n\n```\nnpm test\n```\n\n## Configuration\n\nTuttle is configured in `data/tuttle.xml`. \n\nNew with version 2.0.0:\n\nA commented example configuration is available `data/tuttle-example-config.xml`.\nIf you want to update tuttle your modified configuration file will be backed up to\n`/db/tuttle-backup/tuttle.xml` and restored on installation of the new version.\n\nOtherwise, when no back up of an existing config-file is found, the example configuration is copied to `data/tuttle.xml`.\n\n\u003e [!TIP]\n\u003e When migrating from an earlier version you can copy your existing configuration to the backup location:\n\u003e `xmldb:copy-resource('/db/apps/tuttle/data', 'tuttle.xml', '/db/tuttle-backup', 'tuttle.xml')`\n\n### Repository configuration \n\nThe repositories to keep in sync with a gitservice are all listed under the repos-element.\n\nThe name-attribute refers to the **destination collection** also known as the **target collection**.\n\n#### Collection\n\nAn example: `\u003ccollection name=\"tuttle-sample-data\"\u003e`\nThe collection `/db/apps/tuttle-sample-data` is now considered to be kept in sync with a git repository.\n\n```xml\n\u003ccollection name=\"tuttle-sample-data\"\u003e\n    \u003cdefault\u003etrue\u003c/default\u003e\n\n    \u003ctype\u003egithub\u003c/type\u003e\n    \u003cbaseurl\u003ehttps://api.github.com/\u003c/baseurl\u003e\n\n    \u003crepo\u003etuttle-sample-data\u003c/repo\u003e\n    \u003cowner\u003etuttle-sample-data\u003c/owner\u003e\n\n    \u003ctoken\u003ea-personal-access-token\u003c/token\u003e\n\n    \u003cref\u003ea-branch\u003c/ref\u003e\n\n    \u003chookuser\u003ea-exist-user\u003c/hookuser\u003e\n    \u003chookpasswd\u003ethat-users-password\u003c/hookpasswd\u003e\n\u003c/collection\u003e\n```\n\n#### type\n\n```xml\n\u003ctype\u003egitlab\u003c/type\u003e\n```\n\nThere are two supported git services at the moment `github` and `gitlab`\n\n#### baseurl\n\n```xml\n\u003cbaseurl\u003ehttps://api.server/\u003c/baseurl\u003e\n```\n\n* For github the baseurl is `https://api.github.com/` or your github-enterprise API endpoint\n* For gitlab the baseurl is `https://gitlab.com/api/v4/` but can also be your private gitlab server egg 'https://gitlab.existsolutions.com/api/v4/'\n\n#### repo, owner and project-id\n\n* For github you **have to** specify the owner and the repo\n* For gitlab you **have to** specify the project-id of the repository\n\n\n#### ref\n\n```xml\n\u003cref\u003emain\u003c/ref\u003e\n```\n\nDefines the branch you want to track. \n\n#### hookuser \u0026 hookpasswd\n\n#### token\n\nIf a token is specified Tuttle authenticates against GitHub or GitLab. When a token is not defined, Tuttle assumes a public repository without any authentication.\n\n\u003e [!NOTE]\n\u003e Be aware of the rate limits for unauthenticated requests\n\u003e GitHub allows 60 unauthenticated requests per hour but 5,000 for authenticated requests\n\n\u003e [!TIP]\n\u003e It is also possible to pass the token via an environment variable. The name of the variable have to be  `tuttle_token_ + collection` (all dashes must be replaces by underscore). Example: `tuttle_token_tuttle_sample_data`\n\n##### Create API-Keys for Github / Gitlab\n\nAt this stage of development, the API keys must be generated via the API endpoint `/git/apikey` or for a specific collection `/git/{collection}/apikey`. \n\nIn the configuration `tuttle.xml` the \"hookuser\" is used to define the dbuser which executes the update.\n\nExample configuration for GitHub:\n * 'Payload URL': https://existdb:8443/exist/apps/tuttle/git/hook\n * 'Content type': application/json\n\nExample configuration for GitLab:\n * 'URL' : https://46.23.86.66:8443/exist/apps/tuttle/git/hook\n\n\n## Dashboard\n\nThe dashboard lists all configured collections showing the health\nof all of them at a glance.\nHere, you can trigger a full deployment or an incremental update for each collection.\n\nFull deployment clones the repository from git at ref and installs it as a `.xar` file or just moves the staging collection.\nThis is a way to get to a known state in case you encounter issues.\nAn incremental update only applies those changes to the target collection that happened in the repository after the last synchronization.\n\n\u003e [!NOTE]\n\u003e Tuttle is built to keep track of **data collections**\n\n\u003e [!NOTE]\n\u003e Tuttle does not run pre- or post install scripts nor change the index configuration on incremental updates!\n\n### Let's start\n\n1) customize the configuration (`data/tuttle.xml`)\n2) login to the dashboard\n2) click on 'full' to trigger a full deployment from git to existdb\n3) now you can update your collection with a click on 'incremental'\n\nRepositories from which a valid XAR (existing `expath-pkg.xml`) package can be generated are installed as a package, all others are created purely on the DB.\n\n\u003e [!NOTE]\n\u003e Note that there may be index problems if a collection is not installed as a package.\n\n## API\n\nThe page below is reachable via [api.html](api.html) in your installed tuttle app. \n\n![Tuttle](doc/Tuttle-OpenAPI.png)\n\n### API endpoint description\n\nCalling the API without {collection} ``config:default-collection()`` is chosen. \n\n#### Fetch to staging collection\n\n`` GET ~/tuttle/{collection}/git``\n\nWith this most basic endpoint the complete data repository is pulled from the gitservice.\nThe data will not directly update the target collection but be stored in a staging\ncollection. \n\nTo update the target collection use another POST request to `/tuttle/git`.\n\nThe data collection is stored in `/db/app/sample-collection-staging`.\n\n#### Deploy the collection\n\n`` POST ~/tuttle/{collection}/git``\n\nThe staging collection `/db/app/sample-collection-staging` is deployed to `/db/app/sample-collection`. All permissions are set and a pre-install function is called if needed.\n\n#### Incremental update\n\n`` POST ~/tuttle/{collection}/git``\n\nAll commits since the last update are applied.To ensure the integrity of the collection, all commits are deployed individually.\n\n#### Get the repository hashed\n\n`` GET ~/tuttle/{collection}/hash``\n\nReports the GIT hashed of all participating collections and the hash of the remote repository.\n\n#### Get Commits\n\n`` GET ~/tuttle/{collection}/commits``\n\nDisplays all commits with commit message of the repository. \n\n#### Hook Trigger\n\n`` GET ~/tuttle/{collection}/hook``\n\nThe webhook is usually triggered by GitHub or GitLab.\nAn incremental update is triggered.\nAuthentication is done by APIKey. The APIKey must be set in the header of the request.\n\n#### Example für GitLab\n``` curl --header 'X-Gitlab-Token: RajWFNCILBuQ8SWRfAAAJr7pHxo7WIF8Fe70SGV2Ah' http://127.0.0.1:8080/exist/apps/tuttle/git/hook```\n\n### Generate the APIKey\n\n`` GET ~/tuttle/{collection}/apikey``\n\nThe APIKey is generated and displayed once. If forgotten, it must be generated again.\n\n\n### Display the Repository configuration and status\n\n`` GET ~/tuttle/config ``\n\nDisplays the configuration and the state of the git repository.\n\nStates:\n - uptodate: Collection is up to date with GIT\n - behind: Collection is behind GIT and need an update\n - new: Collection is not a tuttle collection, full deployment is needed\n\n```xml\n\u003ctuttle\u003e\n  \u003cdefault\u003esample-collection-github\u003c/default\u003e\n  \u003crepos\u003e\n    \u003crepo type=\"github\" url=\"https://github.com/Jinntec/tuttle-demo\" ref=\"master\" collection=\"sample-collection-github\" status=\"uptodate\"/\u003e\n    \u003crepo type=\"gitlab\" url=\"https://gitlab.com/tuttle-test/tuttle-demo.git\" ref=\"master\" collection=\"sample-collection-gitlab\" status=\"uptodate\"/\u003e\n  \u003c/repos\u003e\n\u003c/tuttle\u003e\n```\n\n### Remove Lockfile\n\n`` POST ~/tuttle/{collection}/lockfile ``\n\nRemove lockfile after anything goes wrong.\n\n#### Print Lockfile\n\n`` GET ~/tuttle/{collection}/lockfile ``\n\nThe running task is stored in the lockfile. It ensures that two tasks do not run at the same time.\n\n\n## Access token for gitservice (incomplete)\n\nTo talk to the configured gitservice Tuttle needs an access token. These can\nbe obtained from the respective service.\n\n* see [Creating a personal access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) for github\n* see [Personal access tokens](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) for Gitlab\n\nThe key for the gitservice must be configured in Gitservice configuration as shown above.\n\n## Roadmap\n\n- [ ] DB to Git\n\n## Honorable mentions:\n\n![Horace Parnell Tuttle](src/resources/images/HPTuttle-1866.png)\n\n[Horace Parnell Tuttle - American astronomer](http://www.klima-luft.de/steinicke/ngcic/persons/tuttle.htm)\n\n[Archibald \"Harry\" Tuttle - Robert de Niro in Terry Gilliams' 'Brazil'](https://en.wikipedia.org/wiki/Brazil_(1985_film))\n","funding_links":["https://github.com/sponsors/eeditiones","https://www.paypal.com/paypalme/eeditiones"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feeditiones%2Ftuttle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feeditiones%2Ftuttle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feeditiones%2Ftuttle/lists"}