Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/eeditiones/tuttle
https://github.com/eeditiones/tuttle
exist-db git github gitlab library sync xar
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/eeditiones/tuttle
- Owner: eeditiones
- Created: 2021-11-04T13:21:23.000Z (about 3 years ago)
- Default Branch: master
- Last Pushed: 2024-04-24T14:25:26.000Z (8 months ago)
- Last Synced: 2024-05-01T09:52:39.828Z (8 months ago)
- Topics: exist-db, git, github, gitlab, library, sync, xar
- Language: XQuery
- Homepage:
- Size: 2.17 MB
- Stars: 8
- Watchers: 6
- Forks: 3
- Open Issues: 17
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Tuttle - a Git-integration for eXist-db
Synchronizes your data collection with GitHub and GitLab.
## User Documentation
[User Documentation](https://eeditiones.github.io/tuttle-doc/)
## Functionality
* Sync data collection from Git to DB
* Deal with multiple repositories
* Incremental updates
* Works with private or public repositories
## Requirements
- [node](https://nodejs.org/en/): `v18`
- [gulp](https://gulpjs.com): `v4.x` (for building)
- [exist-db](https://www.exist-db.org): `v5.5.1+ < 7.0.0`
## Building and Installation
Tuttle uses Gulp as its build tool which itself builds on NPM.
To initialize the project and load dependencies run
```npm i```
> Note: the `install` commands below assume that you have a local eXist-db running on port 8080. However the database connection can be modified in `.existdb.json.`
| Run | Description |
|---------|-------------|
|```gulp build```|to just build Tuttle. |
|```gulp install```|To build and install Tuttle in one go|
The resulting xar(s) are found in the root of the project.
## Testing
To run the local test suite you need an instance of eXist running on `localhost:8080` and `npm` to be available in your path.
Run tests with ```npm test```
## Configuration
Tuttle is configured in `data/tuttle.xml`.
### Gitservice configuration
@name is always the name of the destination collection. It will be configured in `data/tuttle.xml`
An example:
```xml
true
github
https://api.github.com/
tuttle-sample-data
tuttle-sample-data
master
admin
gitlab
https://gitlab.com/api/v4/
tuttle-sample-data
XXX
master
admin
```
#### type
Gitserver type: 'github' or 'gitlab'
#### baseurl
* For github the baseurl is always api.github.com
* For gitlab the url can also be your private gitlab server egg 'https://gitlab.existsolutions.com/api/v4/'
#### repo, owner and project-id
* For github you have to specify the owner and the repo
* For gitlab you have to specify the project-id of the repository
#### ref
Define the working branch of the git repository
#### hookuser & hookpasswd
#### token
If a token is specified Tuttle authenticates against GitHub or GitLab. When a token is not defined, Tuttle assumes a public repository without any authentication.
It is also possible to pass the token via an environment variable. The name of the variable have to be `tuttle_token_ + collection` (all dashes must be replaces by underscore). Example: `tuttle_token_tuttle_sample_data`
Be aware of the rate limits
* GitHub:
* 60 unauthenticated requests per hour
* 5,000 authenticated requests per hour
##### Create API-Keys for Github / Gitlab
At this stage of development, the API keys must be generated via the API endpoint `/git/apikey` or for a specific collection `/git/{collection}/apikey`.
In the configuration `tuttle.xml` the "hookuser" is used to define the dbuser which executes the update.
Example configuration for GitHub:
* 'Payload URL': https://existdb:8443/exist/apps/tuttle/git/hook
* 'Content type': application/json
Example configuration for GitLab:
* 'URL' : https://46.23.86.66:8443/exist/apps/tuttle/git/hook
## Dashboard
The dashboard can trigger a full deployment or an incremental update.
Full deployment clones the repository from git and install it as a `.xar` file.
With incremental update only the changes to the database collection are applied.
### Lets start
1) customize the configuration (`modules/config.xql`)
2) click on 'full' to trigger a full deployment from git to existdb
3) now you can update your collection with a click on 'incremental'
Repositories from which a valid XAR (existing `expath-pkg.xml` and `repo.xml`) package can be generated are installed as a package, all others are created purely on the DB.
**REMARK: Note that there may be index problems if a collection is not installed as a package.**
**REMARK: Only use it with data collections**
## API
The page below is reachable via [api.html](api.html) in your installed tuttle app.
### API endpoint description
Calling the API without {collection} ``config:default-collection()`` is chosen.
#### Fetch to staging collection
`` GET ~/tuttle/{collection}/git``
With this most basic endpoint the complete data repository is pulled from the gitservice.
The data will not directly update the target collection but be stored in a staging
collection.
To update the target collection use another POST request to `/tuttle/git`.
The data collection is stored in `/db/app/sample-collection-staging`.
#### Deploy the collection
`` POST ~/tuttle/{collection}/git``
The staging collection `/db/app/sample-collection-staging` is deployed to `/db/app/sample-collection`. All permissions are set and a pre-install function is called if needed.
#### Incremental update
`` POST ~/tuttle/{collection}/git``
All commits since the last update are applied.To ensure the integrity of the collection, all commits are deployed individually.
#### Get the repository hashed
`` GET ~/tuttle/{collection}/hash``
Reports the GIT hashed of all participating collections and the hash of the remote repository.
#### Get Commits
`` GET ~/tuttle/{collection}/commits``
Displays all commits with commit message of the repository.
#### Hook Trigger
`` GET ~/tuttle/{collection}/hook``
The webhook is usually triggered by GitHub or GitLab.
An incremental update is triggered.
Authentication is done by APIKey. The APIKey must be set in the header of the request.
#### Example für GitLab
``` curl --header 'X-Gitlab-Token: RajWFNCILBuQ8SWRfAAAJr7pHxo7WIF8Fe70SGV2Ah' http://127.0.0.1:8080/exist/apps/tuttle/git/hook```
### Generate the APIKey
`` GET ~/tuttle/{collection}/apikey``
The APIKey is generated and displayed once. If forgotten, it must be generated again.
### Display the Repository configuration and status
`` GET ~/tuttle/config ``
Displays the configuration and the state of the git repository.
States:
- uptodate: Collection is up to date with GIT
- behind: Collection is behind GIT and need an update
- new: Collection is not a tuttle collection, full deployment is needed
```xml
sample-collection-github
```
### Remove Lockfile
`` POST ~/tuttle/{collection}/lockfile ``
Remove lockfile after anything goes wrong.
#### Print Lockfile
`` GET ~/tuttle/{collection}/lockfile ``
The running task is stored in the lockfile. It ensures that two tasks do not run at the same time.
## Access token for gitservice (incomplete)
To talk to the configured gitservice Tuttle needs an access token. These can
be obtained from the respective service.
* see [Creating a personal access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) for github
* see [Personal access tokens](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) for Gitlab
The key for the gitservice must be configured in Gitservice configuration as shown above.
## DB to Git
Will be implemented in release 2.0.0
## Honorable mentions:
[Horace Parnell Tuttle - American astronomer](http://www.klima-luft.de/steinicke/ngcic/persons/tuttle.htm)
[Archibald "Harry" Tuttle - Robert de Niro in Terry Gilliams' 'Brazil'](https://en.wikipedia.org/wiki/Brazil_(1985_film))