{"id":21192167,"url":"https://github.com/mu-semtech/validation-service","last_synced_at":"2025-03-14T21:13:15.169Z","repository":{"id":151390314,"uuid":"130366947","full_name":"mu-semtech/validation-service","owner":"mu-semtech","description":"Microservice to execute async data validations using configurable in code validations","archived":false,"fork":false,"pushed_at":"2018-06-27T09:08:43.000Z","size":23,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-01-21T13:43:58.585Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mu-semtech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-20T13:37:28.000Z","updated_at":"2018-06-27T09:08:24.000Z","dependencies_parsed_at":null,"dependency_job_id":"1764c1b8-d53e-4a7a-b2f9-3144cfbd5aa9","html_url":"https://github.com/mu-semtech/validation-service","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fvalidation-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fvalidation-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fvalidation-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fvalidation-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mu-semtech","download_url":"https://codeload.github.com/mu-semtech/validation-service/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243646667,"owners_count":20324586,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T19:07:48.098Z","updated_at":"2025-03-14T21:13:15.153Z","avatar_url":"https://github.com/mu-semtech.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# validation-service\nMicroservice to execute async data validations using configurable in code validations. The validation execution's progress and validation errors are written to the store. This microservice only provides endpoints (1) to trigger an async execution of a validation set and (2) to monitor the status of a single execution. Endpoints to get all executions, validations and errors should be configured using [mu-cl-resources](https://github.com/mu-semtech/mu-cl-resources).\n\n## Installation\nTo add the service to your stack, add the following snippet to `docker-compose.yml`:\n```yaml\nservices:\n  validation:\n    image: semtech/mu-validation-service:0.3.0\n    volumes:\n      - ./config/validations:/config\n```\n\nDon't forget to update the dispatcher configuration to route requests to the validation service.\n\n## Configuration\nValidations are provided in code as an array of javascript objects exported in `/config/index.js`.\n\nEach validation object should have the following properties:\n* name [string]: name of the validation rule\n* description [string]: description of the validation rule\n* validationSets [array[uri]]: list of URIs defining the sets to which the validation rule belongs\n* validate [(execution) =\u003e [object]]: function that executes the validation and returns an array of validation errors. An empty array means the validation succeeded. The function receives the current execution as parameter. Validation errors must be written to the store in this function. The helpers functions described below may be of help to implement this function.\n\nNote: the application graph is available through `process.env.MU_APPLICATION_GRAPH`.\n\n## Helper functions\nThe validation service provides helpers functions to implement the validate function of a validation rule. Currently the helpers functions listed below are available. They can be imported from `/app/helpers`.\n\nE.g. `import { validateSparqlSelect } from '/app/helpers';`\n\n### Validations using a SPARQL query\n#### validateSparqlSelect(sparqlQuery)\nHelper function to make a validation rule using a SPARQL SELECT query. The validation is considered invalid if the query returns any result. Each entry in the result set is stored as a validation error in the store.\n\nThe error message per entry is constructed by calling the validation's message function passing the result bindings of the SELECT query as a params object. E.g. `{ \"s\": \"http://data.lblod.info/id/mandataris/123\", \"start\": \"01-12-2018\" }`. These values can then be used to construct the error message.\n\nExample validation:\n```javascript\n  {\n    name: 'my-validation',\n    description: 'Start date must fall before end date',\n    validationSets: [\n      'http://data.lblod.info/id/validation-set/mandatendatabank'\n    ],\n    message: function(params) {\n      return `Mandataris ${params['s']}: start date ${params['start']} is later than end date ${params['einde']}`;\n    },\n    validate: validateSparqlSelect(`\n        PREFIX mandaat: \u003chttp://data.vlaanderen.be/ns/mandaat#\u003e\n        PREFIX mu: \u003chttp://mu.semte.ch/vocabularies/core/#\u003e\n        SELECT ?s ?uuid ?start ?einde\n        FROM \u003c${process.env.MU_APPLICATION_GRAPH}\u003e\n        WHERE {\n          ?s a mandaat:Mandataris ;\n             mandaat:start ?start ;\n             mandaat:einde ?einde .\n          OPTIONAL { ?s mu:uuid ?uuid }\n          FILTER (?einde \u003c ?start)\n        }`)\n  }\n```\n\n#### validateSparqlAsk(sparqlQuery)\nHelper function to make a validation rule using a SPARQL ASK query. The validation is considered invalid if the query returns 'false'. In that case one validation error is written to the store. The error message of the validation may be a static string or a parameterless function.\n\nExample validation:\n```javascript\n  {\n    name: 'my-validation',\n    description: 'At least 1 person',\n    validationSets: [\n      'http://data.lblod.info/id/validation-set/mandatendatabank'\n    ],\n    message: 'At least 1 persoon',\n    validate: validateSparqlAsk(`\n        PREFIX persoon: \u003chttp://data.vlaanderen.be/ns/persoon#\u003e\n        ASK {\n          GRAPH \u003c${process.env.MU_APPLICATION_GRAPH}\u003e {\n            ?s a persoon:Persoon .\n          }\n        }`)\n  }\n```\n\n\n### Validation errors\n#### insertNewError(executionUri, validationUri, message)\nHelper function to write a validation error to the store.\n\nParameters:\n* executionUri [string]: URI of the execution that produced the validation error\n* validationUri [string]: URI of the validation that failed\n* message [string]: error message of the validation\n\nThe function returns a `ValidationError` object with a `uri` property such that the user can enrich the data stored about the validation error afterwards.\n\n#### insertNewErrors(errors)\nHelper function to write multiple validation errors in bulk to the store.\n\nParameters:\n* errors [array]: Array of validation error objects. Each error object must contain the following properties:\n** executionUri [string]: URI of the execution that produced the validation error\n** validationUri [string]: URI of the validation that failed\n** message [string]: error message of the validation\n\nThe function returns an array of `ValidationError` objects with a `uri` property such that the user can enrich the data stored about the validation errors afterwards.\n\n## API\n### POST /executions\nTrigger an async execution of a validation set.\n\nRequest body may optionally define a validation set. If no validation set is speficied all validations will be executed.\nE.g.\n```javascript\n{\n  \"validation-set\": \"http://data.lblod.info/id/validation-set/mandatendatabank\"\n}\n```\n\n### GET /executions/:id\nMonitor the status of a single execution. Status is one of `ongoing`, `done`, `failed`, `canceled`.\n\nExample\n```javascript\n{\n    \"data\": {\n        \"type\": \"executions\",\n        \"id\": \"d1a1d430-43dc-11e8-a5fd-9b3cd5f0fe08\",\n        \"attributes\": {\n            \"uri\": \"http://mu.semte.ch/services/validation-service/executions/d1a1d430-43dc-11e8-a5fd-9b3cd5f0fe08\",\n            \"status\": \"failed\",\n            \"created\": \"2018-04-19T14:20:32.371Z\"\n        }\n    }\n}\n```\n\n## Retrieving executions, validations and errors using mu-cl-resources\nThis microservice only provides endpoints (1) to trigger an async execution of a validation set and (2) to monitor the status of a single execution. Endpoints to get all executions, validations and errors should be configured using [mu-cl-resources](https://github.com/mu-semtech/mu-cl-resources).\n\nAdd the following prefixes to your `repository.lisp`\n```lisp\n(add-prefix \"validation\" \"http://mu.semte.ch/vocabularies/validation/\")\n(add-prefix \"dct\" \"http://purl.org/dc/terms/\")\n```\n\nAdd the following configuration to your `domain.lisp`\n```lisp\n(define-resource validation-execution ()\n  :class (s-prefix \"validation:Execution\")\n  :properties `((:status :string ,(s-prefix \"validation:status\"))\n                (:created :datetime ,(s-prefix \"dct:created\")))\n  :has-many `((validation-error :via ,(s-prefix \"validation:generatedBy\")\n                                :inverse t\n                                :as \"errors\")\n              (validation :via ,(s-prefix \"validation:performsValidation\")\n                       :as \"validations\"))\n  :resource-base (s-url \"http://mu.semte.ch/services/validation-service/executions/\")\n  :features '(include-uri)\n  :on-path \"validation-executions\"\n)\n\n(define-resource validation ()\n  :class (s-prefix \"validation:Validation\")\n  :properties `((:name :string ,(s-prefix \"validation:name\"))\n                (:description :string ,(s-prefix \"validation:description\"))\n                (:status :string ,(s-prefix \"validation:status\")))\n  :has-one `((validation-execution :via ,(s-prefix \"validation:performsValidation\")\n                                   :inverse t\n                                   :as \"execution\"))\n  :has-many `((validation-error :via ,(s-prefix \"validation:validation\")\n                                   :inverse t\n                                   :as \"errors\"))\n  :resource-base (s-url \"http://mu.semte.ch/services/validation-service/validations/\")\n  :features '(include-uri)\n  :on-path \"validations\"\n)\n\n\n(define-resource validation-error ()\n  :class (s-prefix \"validation:Error\")\n  :properties `((:message :string ,(s-prefix \"validation:message\")))\n  :has-one `((validation-execution :via ,(s-prefix \"validation:producedBy\")\n                                   :as \"execution\")\n             (validation :via ,(s-prefix \"validation:validation\")\n                                   :as \"validation\"))\n  :resource-base (s-url \"http://mu.semte.ch/services/validation-service/validation-errors/\")\n  :features '(include-uri)\n  :on-path \"validation-errors\"\n)\n```\n\nAdd the following dispatcher rules in `dispatcher.ex`\n```erlang\n  get \"/validation-executions/*path\" do\n    Proxy.forward conn, path, \"http://resource/validation-executions/\"\n  end\n  get \"/validations/*path\" do\n    Proxy.forward conn, path, \"http://resource/validations/\"\n  end\n  get \"/validation-errors/*path\" do\n    Proxy.forward conn, path, \"http://resource/validation-errors/\"\n  end\n```\n\nThe latest validation execution of a specific set can be retrieved via\n```\nGET /validation-executions?sort=-created\u0026filter[status]=done\u0026filter[validation-set]=http://data.lblod.info/id/validation-set/mandatendatabank\u0026page[size]=1\n```\n\n## Development\nAdd the following snippet to your stack during development:\n```yaml\nservices:\n  validation:\n    image: semtech/mu-javascript-template:1.3.1\n    ports:\n      - 8888:80\n    environment:\n      NODE_ENV: \"development\"\n      MU_APPLICATION_GRAPH: \"http://mu.semte.ch/graphs/public\"\n    volumes:\n      - /path/to/your/code:/app/\n      - ./config/validations:/config\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmu-semtech%2Fvalidation-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmu-semtech%2Fvalidation-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmu-semtech%2Fvalidation-service/lists"}