{"id":21192176,"url":"https://github.com/mu-semtech/mu-migrations-service","last_synced_at":"2025-08-08T17:37:05.942Z","repository":{"id":45231443,"uuid":"77613889","full_name":"mu-semtech/mu-migrations-service","owner":"mu-semtech","description":"Service for running database migrations","archived":false,"fork":false,"pushed_at":"2024-09-03T15:17:34.000Z","size":57,"stargazers_count":2,"open_issues_count":5,"forks_count":7,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-06-06T09:42:00.335Z","etag":null,"topics":["microservice","migrations","mu-service","musemtech","sparql"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mu-semtech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-12-29T13:25:27.000Z","updated_at":"2021-12-29T19:07:40.000Z","dependencies_parsed_at":"2023-02-15T07:30:56.803Z","dependency_job_id":"0fcbad7f-7d33-4bbc-a59e-9a84fb2cdbc7","html_url":"https://github.com/mu-semtech/mu-migrations-service","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/mu-semtech/mu-migrations-service","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fmu-migrations-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fmu-migrations-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fmu-migrations-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fmu-migrations-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mu-semtech","download_url":"https://codeload.github.com/mu-semtech/mu-migrations-service/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fmu-migrations-service/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264521172,"owners_count":23622096,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["microservice","migrations","mu-service","musemtech","sparql"],"created_at":"2024-11-20T19:07:49.305Z","updated_at":"2025-07-10T03:31:24.327Z","avatar_url":"https://github.com/mu-semtech.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mu-migrations-service\n\nThe mu-migrations-service runs migrations on the database.  This currently includes SPARQL queries (`*.sparql`) and Turtle files (`*.ttl`). We intend more formats to be supported in the future.\n\n## Tutorials\n\n### Add the migrations service to a stack\nTo install the migrations service in your project, add the migrations-service to\nthe `docker-compose.yml` file of your mu-project by adding the following snippet:\n\n```\n    migrations:\n      image: semtech/mu-migrations-service:0.9.0\n      links:\n        - triplestore:database\n      volumes:\n        - ./config/migrations:/data/migrations\n```\n\n`triplestore` is the service name of the database (probably a Virtuoso instance) running in your stack.\n\nStart your stack using `docker-compose up -d`. The migrations service will be created.\n\nExecute `docker-compose logs -ft migrations` to inspect the logs of the service. You will see the migrations service started up successfully. No migrations are executed since we didn't define one yet. Let's go to the next step and create our first migration!\n\n\n### Writing a migration to update a predicate in your dataset\nWe're going to define a migration that will change all predicates `schema:name` in our dataset to `foaf:name`.\n\nFirst, create a new migration file `./config/migrations/20200329140538-replace-schema-name-with-foaf-name.sparql`.\nNext, insert the SPARQL query to execute the change in the file.\n\n```\n    PREFIX schema: \u003chttp://schema.org/\u003e\n    PREFIX foaf: \u003chttp://xmlns.com/foaf/0.1/\u003e\n\n    DELETE {\n      GRAPH ?g { ?s schema:name ?o . }\n    } INSERT {\n      GRAPH ?g { ?s foaf:name ?o . }\n    } WHERE {\n      GRAPH ?g { ?s schema:name ?o . }\n    }\n```\n\nRestart the migrations service by running `docker-compose restart migrations`. Inspect the logs using `docker-compose logs -ft migrations`. You will see the migration gets executed and the success status is printed in the migrations summary in the logs.\n\n## How-to guides\n\n### Manipulating data using a SPARQL query\n\nSpecify the migration in a file, like\n`./config/migrations/20160808225103-statuses.sparql` containing a SPARQL\nquery like:\n\n```\n    PREFIX dct: \u003chttp://purl.org/dc/terms/\u003e\n    PREFIX tac: \u003chttp://tasks-at-hand.com/vocabularies/core/\u003e\n    PREFIX ext: \u003chttp://mu.semte.ch/vocabularies/ext/\u003e\n    PREFIX rm: \u003chttp://mu.semte.ch/vocabularies/logical-delete/\u003e\n    PREFIX typedLiterals: \u003chttp://mu.semte.ch/vocabularies/typed-literals/\u003e\n    PREFIX mu: \u003chttp://mu.semte.ch/vocabularies/core/\u003e\n    PREFIX xsd: \u003chttp://www.w3.org/2001/XMLSchema#\u003e\n    PREFIX app: \u003chttp://mu.semte.ch/app/\u003e\n    PREFIX owl: \u003chttp://www.w3.org/2002/07/owl#\u003e\n    PREFIX rdf: \u003chttp://www.w3.org/1999/02/22-rdf-syntax-ns#\u003e\n    INSERT DATA {\n      GRAPH \u003chttp://mu.semte.ch/application\u003e {\n        \u003chttp://tasks-at-hand.com/resources/statuses/not_started\u003e\n          a tac:Status;\n          mu:uuid \"wellknown-status-not_started\";\n          dct:title \"not started\".\n        \u003chttp://tasks-at-hand.com/resources/statuses/ongoing\u003e\n          a tac:Status;\n          mu:uuid \"wellknown-status-ongoing\";\n          dct:title \"ongoing\".\n        \u003chttp://tasks-at-hand.com/resources/statuses/done\u003e\n          a tac:Status;\n          mu:uuid \"wellknown-status-done\";\n          dct:title \"done\".\n      }\n    }\n```\n\n### Inserting data in the default graph using Turtle files\nSpecify the migration in a file, like\n`./config/migrations/20160808225103-statuses.ttl` containing triples in Turtle format like:\n\n```\n    @prefix dct: \u003chttp://purl.org/dc/terms/\u003e .\n    @prefix tac: \u003chttp://tasks-at-hand.com/vocabularies/core/\u003e .\n    @prefix ext: \u003chttp://mu.semte.ch/vocabularies/ext/\u003e .\n    @prefix rm: \u003chttp://mu.semte.ch/vocabularies/logical-delete/\u003e .\n    @prefix typedLiterals: \u003chttp://mu.semte.ch/vocabularies/typed-literals/\u003e .\n    @prefix mu: \u003chttp://mu.semte.ch/vocabularies/core/\u003e .\n    @prefix xsd: \u003chttp://www.w3.org/2001/XMLSchema#\u003e .\n    @prefix app: \u003chttp://mu.semte.ch/app/\u003e .\n    @prefix owl: \u003chttp://www.w3.org/2002/07/owl#\u003e .\n    @prefix rdf: \u003chttp://www.w3.org/1999/02/22-rdf-syntax-ns#\u003e .\n\n    \u003chttp://tasks-at-hand.com/resources/statuses/not_started\u003e\n          a tac:Status;\n          mu:uuid \"wellknown-status-not_started\";\n          dct:title \"not started\".\n    \u003chttp://tasks-at-hand.com/resources/statuses/ongoing\u003e\n          a tac:Status;\n          mu:uuid \"wellknown-status-ongoing\";\n          dct:title \"ongoing\".\n    \u003chttp://tasks-at-hand.com/resources/statuses/done\u003e\n          a tac:Status;\n          mu:uuid \"wellknown-status-done\";\n          dct:title \"done\".\n```\n\nBy default, the Turtle files will be imported in the `\u003chttp://mu.semte.ch/application\u003e` graph.\n\n### Inserting data in a specific graph using Turtle files (experimental)\nTo insert data in a specific graph instead of the default graph, create a `.graph`-file by the same name as the `.ttl`-file. E.g. `./config/migrations/20160808225103-statuses.graph` for a Turtle migration named `./config/migrations/20160808225103-statuses.ttl`.\n\nThe `.graph` file contains only one line, specifying the graph name to insert the data in:\n\n```\nhttp://mu.semte.ch/graphs/custom-graph\n```\n\n## Reference\n### Naming and organizing the migrations\nMigrations are specified in files, to be executed in the order of their filename. All the files need to be available in `/data/migrations` inside the docker container, but the may be organized in subfolders.\n\nThe name of a migration file must always start with a number and be unique across all migrations (also migrations stored in other folders!).\nIt is advised to use the unix system time as the basis for the filename of your migration, postfixed with a short name of what the migration performs. E.g. `20200329140538-replace-schema-name-with-foaf-name.sparql`.\n\nSince the execution state of a migration is determined only be the filename and not the full file path, subfolders may be reorganized after execution of a migration. This allows for example to archive migrations in subfolders per year while the project progresses.\n\n### Execution guarantees\nThe migrations service provides the following guarantees of execution:\n- Migrations are run sequentially in order of the first number in the filename, in ascending order.\n- A migration has to complete successfully in order for the next migration to start, there is no concurrent execution of migrations.\n- If a migration fails to run, no subsequent migrations will be attempted.\n- A migration that has been marked as completed will not be started again nor will a migration with the same filename be executed.\n\n### Migration management in the database\nThe completion of a migration is stored in the database in the `MU_APPLICATION_GRAPH` (default: `\u003chttp://mu.semte.ch/application\u003e`).\n\nEach successfully executed migration is represented by a resource of type `muMigr:Migration` with the following properties:\n- `muMigr:filename`: name of the migration file\n- `muMigr:executedAt`: datetime when the migration successfully finished\n\nUsed prefix: `muMigr: \u003chttp://mu.semte.ch/vocabularies/migrations/\u003e`\n\n### Configuration\nThe migration service supports configuration via environment variables.\n- `BATCH_SIZE`: amount of triples to insert in one go for a Turtle migration (default: 12000)\n- `MINIMUM_BATCH_SIZE`: if the batch size drops below this number the service will stop with an error. (default: 100)\n- `COUNT_BATCH_SIZE`: number of executed migrations to retrieve from the database in one go (default: 10000). \n*NOTE*: Make sure this is lower or equal to the maximum number of rows returned by the SPARQL endpoint.\n\nThis microservice is based on the [mu-ruby template](https://github.com/mu-semtech/mu-ruby-template) and supports the environment variables documented in its [README](https://github.com/mu-semtech/mu-ruby-template#configuration).\n\n## Discussions\n### Large datasets and batch size\nTriple stores typically can only handle a certain amount of triples to be ingested per request. The migration service supports batching to split of large datasets in multiple requests. This can be configured with the `BATCH_SIZE` environment variable. If an error occurs during batch ingestion the batch size will be halved and the request retried until `MINIMUM_BATCH_SIZE` is reached. At this point an error will be thrown.\n\nTo make sure a dataset is loaded completely it will first be ingested into a temporary graph, on success the contents will be added to the target graph with a SPARQL Graph query.\n\n### Working with mu-authorization\nExperimental: You can hook the migrations service onto mu-authorization.  The migrations service will add the `mu-auth-sudo` header and execute migrations with elevated priviledges.  Support is experimental and we'd love to hear about your experience with this feature so we can harden it.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmu-semtech%2Fmu-migrations-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmu-semtech%2Fmu-migrations-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmu-semtech%2Fmu-migrations-service/lists"}