{"id":19338321,"url":"https://github.com/gbv/validation-server","last_synced_at":"2026-02-26T01:33:28.860Z","repository":{"id":57390347,"uuid":"455117290","full_name":"gbv/validation-server","owner":"gbv","description":"Web service to validate data with support of multiple schema languages","archived":false,"fork":false,"pushed_at":"2024-10-29T14:00:18.000Z","size":1373,"stargazers_count":2,"open_issues_count":42,"forks_count":1,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-04-14T15:49:58.074Z","etag":null,"topics":["data-science","validation"],"latest_commit_sha":null,"homepage":"https://format.gbv.de/validate/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gbv.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-03T10:18:15.000Z","updated_at":"2023-05-15T12:55:16.000Z","dependencies_parsed_at":"2024-08-29T14:47:37.827Z","dependency_job_id":"784a0782-459b-4f3a-95bc-63bfd2972825","html_url":"https://github.com/gbv/validation-server","commit_stats":{"total_commits":235,"total_committers":2,"mean_commits":117.5,"dds":"0.012765957446808529","last_synced_commit":"0bbcf27fff2ae2f39ce33926bcdf5203f5bfc2b4"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbv%2Fvalidation-server","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbv%2Fvalidation-server/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbv%2Fvalidation-server/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gbv%2Fvalidation-server/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gbv","download_url":"https://codeload.github.com/gbv/validation-server/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250352289,"owners_count":21416464,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","validation"],"created_at":"2024-11-10T03:17:02.234Z","updated_at":"2026-02-26T01:33:23.833Z","avatar_url":"https://github.com/gbv.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Validation-Server\n\n[![Test](https://github.com/gbv/validation-server/actions/workflows/test.yml/badge.svg?branch=dev)](https://github.com/gbv/validation-server/actions/workflows/test.yml)\n[![NPM package version](https://img.shields.io/npm/v/validation-server.svg)](https://www.npmjs.com/package/validation-server)\n[![standard-readme compliant](https://img.shields.io/badge/readme%20style-standard-brightgreen.svg)](https://github.com/RichardLitt/standard-readme)\n\n\u003e Web service to validate data with support of multiple schema languages\n\nLarge parts of practical data science or other data processing work is spent by bringing dirty data into shape. **Data formats** define desired shapes of data. To check whether data conforms to a data format, it must be **validated**. This application helps to validate data against data formats. [Validation errors](#validation-errors) show if, where and how a format is violated so data can be cleaned or rejected. Data formats can be configured with **schemas** in multiple [schema languages](#schema-languages).\n\n## Table of Contents\n\n- [Background](#background)\n  - [Data Formats](#data-formats)\n  - [Schema Languages](#schema-languages)\n  - [Locator Languages](#locator-languages)\n  - [See Also](#see-also)\n- [Install](#install)\n  - [From GitHub](#from-github)\n  - [Configuration](#configuration)\n- [Usage](#usage)\n  - [Run Server](#run-server)\n  - [Deployment](#deployment)\n  - [Updates](#updates)\n  - [Use as Module](#use-as-module)\n  - [Command Line Interface](#command-line-interface)\n  - [Run Tests](#run-tests)\n- [API](#api)\n  - [GET /validate](#get-validate)\n  - [POST /{format}](#post-format)\n  - [GET /formats](#get-formats)\n  - [GET /languages](#get-languages)\n  - [GET /schema](#get-schema)\n  - [Validation Errors](#validation-errors)\n  - [API Errors](#api-errors)\n- [Maintainers](#maintainers)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Background\n\n### Data Formats\n\nData formats supported by validation server are described as JSON Object with:\n\n- `id` mandatory local format identifier\n- `title` optional title\n- `short` optional short title (abbreviation or acronym)\n- `description` optional short textual description or definition\n- `url` optional URL with information about the format\n- `wikidata` optional Wikidata identifier of the format\n- `mimetypes` optional array of content types\n- `base` optional identifier of a base format (e.g. `json` for JSON-based formats)\n- `versions` an optional object with format versions, the keys used as version identifier (is `default` if unknown) and each values an object with:\n    - `schemas` an optional array with schemas, each an object with:\n        - `type` mandatory schema type (identifier of a schema language, e.g. `json-schema`)\n        - `url` optional URL to retrieve the schema file from\n        - `value` schema as string (mandatory if no `url` given)\n\nThis meta-format is defined as **Data About Data Formats** with a JSON Schema at \u003chttps://format.gbv.de/validate/format-schema.json\u003e.\n\nAPI endpoint [/formats](#get-formats) or [command line argument](#command-line-interface) `--list` return data formats supported by an instance of validation server.\n\n### Schema Languages\n\nSchema languages (also known as schema formats or schema types) are data formats used to define other data formats. Formats defined by a schema language all share a common base format. For instance JSON Schema is a schema language to define JSON-based formats, XML Schema is a schema language to define XML-based formats, and regular expressions can be used as schema language to describe character-based formats.\n\nSchema languages supported by validation server are described as [data formats](#data-formats) with additional mandatory keys `restricts` referencing the common base format(s).\n\nThe following schema languages are supported for validation of other formats. The list is available via API endpoint [/languages](#get-languages):\n\n- JSON Schema (`json-schema`)\n  - Supports `draft-04`, `draft-06`, and `draft-07`\n  - Supports format keywords from \u003chttps://github.com/ajv-validator/ajv-formats\u003e and \u003chttps://github.com/luzlab/ajv-formats-draft2019\u003e\n\n- XML Schema (`xsd`)\n  - Requires `xmllint` executable to be installed.\n\n- Regular Expressions (`regexp`)\n  - Supports ECMAScript variant with Unicode flag enabled automatically and flags other than `i`, `m`, `s` ignored.\n  - Can be given as plain pattern or in form `/${pattern}/${flags}`\n\n### Locator Languages\n\nLocator languages such as XPath and JSON Pointer are used to reference parts of a document. Validation server supports the following languages to locate [validation errors](#validation-errors) and/or to select which parts of a document to validate:\n\n- [JSON Pointer](https://datatracker.ietf.org/doc/html/rfc6901) (`jsonpointer`) for error locations in JSON documents\n\n- Line \u0026 Column (`linecol`) for error locations in documents: e.g. `3:7` for line 3 column 7\n\n- Character position (`rfc5147`) for error locations given by character position as defined in [RFC 5147](https://datatracker.ietf.org/doc/html/rfc5147): e.g. `char=23`\n\n- JSONPath (`jsonpath`) limited to value `$.*` to select to validate elements of a JSON array\n\n### See Also\n\nThe format registry \u003chttp://format.gbv.de/\u003e (mostly German) lists data formats relevant to cultural heritage institutions. The thesis described at \u003chttp://aboutdata.org\u003e includes some theoretical background.\n\n## Install\n\nRequires at least Node v14.8.0.\n\nSupport of XML Schema based formats requires `xmllint` to be installed:\n\n```bash\nsudo apt install libxml2-utils\n```\n\n### From GitHub\n\n```bash\ngit clone https://github.com/gbv/jskos-server.git\ncd jskos-server\nnpm ci\n```\n\n### Configuration\n\nThe service must be customized via configuration files. By default, this configuration file resides in `config/config.json` (or `config/config.test.json` for tests). Missing keys are defaulted from `config/config.default.json`:\n\n```json\n{\n  \"title\": \"Validation Service\",\n  \"description\": \"Web service to validate data against schemas\",\n  \"version\": \"X.X.X\",\n  \"port\": 3700,\n  \"proxies\": [],\n  \"postLimit\": \"20MB\",\n  \"timeout\": 2000,\n  \"allErrors\": false\n  \"verbosity\": \"info\",\n  \"formats\": [],\n  \"cache\": \"../cache\",\n}\n```\n\nKeys `version` and `description` are defaulted to its value in `package.json`. In addition the environment variable `NODE_ENV` is respected with `development` as default. Alternative values are `production`, `test`, and `debug`.\n\nKey `formats` must contain an array of [data formats](#data-formats) or a file containing such array (JSON, NDJSON or YAML format). The list of formats is automatically extended by some hardcoded formats and schema languages.\n\nSome validators stop after finding the first error. Key `allErrors` configures validators to continue validation to find more errors.\n\nAdditional keys `provider` and `links` can be used to adjust the HTML footer.\n\nA JSON Schema to validate configuration is included at [`config/config-schema.json`](config/config-schema.json).\n\n## Usage\n\nThis node package is mainily designed to be run as web service but it can also be [used as module](#use-as-module). See [API Reference](#api) for usage of a running server instance instead.\n\n### Run Server\n\n```bash\n# Development server with hot reload and auto reconnect at localhost:3700 (default)\nnpm run start\n\n# To run the server in production, run this:\nNODE_ENV=production node ./server.js\n```\n\nOn startup all configured schemas are downloaded and cached in directory configured with `cache` (set to `false` to use a temporary directory). Addition and updates require to restart the server.\n\n### Deployment\n\nTo provide the service behind a nginx web server at path `/validate/` (like at \u003chttp://format.gbv.de/validate/\u003e), add this to nginx configuration file:\n\n```\nlocation /validate {\n    port_in_redirect off;\n    return /validate/;\n}\nlocation /validate/ {\n    proxy_pass http://127.0.0.1:3700/;\n}\n```\n\nWe recommend to use [PM2](https://pm2.keymetrics.io/) to start and update the server:\n\n```bash\npm2 start ecosystem.config.json\n```\n\n### Updates\n\nTo update an instance deployed with PM2:\n\n```bash\n# get updates from repository\ngit pull\n\n# install dependencies\nnpm ci\n\n# restart the process (adjust process name if needed)\npm2 restart validation-server\n```\n\nThe server needs to be restarted to reflect updates in [configuration](#configuration), including formats and schemas to be supported. Script `bin/update.js` can help updating formats and local schema files without breaking things.\n\n### Use as Module\n\n*The internal API is not stable yet!*\n\n```js\nconst { loadConfig, validateConfig, createService, knownFormats } = require(\"validation-server\")\n\nconst config = loadConfig()\n\ncreateService(config).then(service =\u003e {\n  const format = service.getFormat(\"json-schema\", { version: \"draft-07\" })\n\n  // asynchronous validation\n  format.valid(data)\n    .then(() =\u003e console.log(\"ok\"))\n    .catch(e =\u003e console.error(e.errors))\n\n  // validate a stream of records\n  const resultStream = inputStream.pipe(format.validateStream)\n})\n```\n\n#### loadConfig\n\nReturns a valid configuration object. Location of an optional configuration file can be given as argument (or by setting environment variable `NODE_ENV` to `debug` or `test`):\n\n```js\nconst config = loadConfig(\"./config.json\")\n```\n\nAn error is thrown on invalid configuration.\n\n#### validateConfig\n\nValidate a configuration object. Throws an error if configuration is not valid. Returns the passed configuration otherwise.\n\n#### createService\n\nGiven a [configuration](#configuration) object, `createService` returns a promise to an initialized service object with methods:\n\n- `getFormat` returns format (or `undefined`), queried by `format` identifier and optional `version` (set to `default` if not specified)\n\n- `listFormats` returns a list of formats, optionally filtered (see [GET /formats](#get-formats) for query parameters)\n\nSchemas of format objects can have an additional `validate` method to validate data in this format.\n\n#### knownFormats\n\nObject with predefined data formats.\n\n### Command Line Interface\n\nThe module includes an experimental command line interface:\n\n```bash\nnpm run validate -- --help  # run from repository\nvalidate --help             # if installed via npm\n```\n\n### Run Tests\n\nTests requires `xmlint` to be installed. Then:\n\n```bash\nnpm test\nnpm run debug    # test with logging enabled\nnpm run coverage # test with code coverage report\n```\n\n## API\n\nThe response status code should always be 200 (possibly including [validation errors](#validation-errors), unless there was an [API error](#api-errors) such as wrong request parameters or unexpected internal failures.\n\n### GET /validate\n\nEndpoint to validate records passed via query parameter or URL.\n\n**Query parameters**\n\n* `format` format identifier\n\n* `url` Optional URL to load data from\n\n* `data` Data to be validated. Ignored when parameter `url` is given.\n\n* `select` Optional selection of records within the posted data. Only supported for JSON-based formats with JSONPath `select=$` (default: data is one record) and `select=$.*` (records are array or object elements).\n\n* `encoding` Optional alternative encoding for (experimental). Supports `yaml` and `ndjson` for json-based formats, so far.\n\n**Response**\n\nArray of same length as the posted data and validation result formeach record.  An element is `true` when the object passed validation, or an array of [validation errors](#validation-errors) when the object failed validation.\n\n**Examples**\n\nCheck whether a simple string such as `{}` or `[x]` is valid or invalid JSON:\n\n```sh\ncurl -g 'http://format.gbv.de/validate/validate?format=json\u0026data={}'\n```\n\n```json\n[\n  true\n]\n```\n\n```sh\ncurl -g 'http://format.gbv.de/validate/validate?format=json\u0026data=[x]'\n```\n\n```json\n[\n  [\n    {\n      \"message\": \"Unexpected token x in JSON at position 1\",\n      \"position\": { \"rfc5147\": \"char=1\", \"linecol\": \"1:2\" }\n    }\n  ]\n]\n```\n\nJSON parsing errors are returned with [location](#locator-languages).\n\nThe service does not guarantee to return all validation errors but it may stop at the first error.\n\n### POST /{format}\n\nValidate records like [GET /validate](#validate) but data is send via HTTP POST payload or as `multipart/form-data`.\n\n**Query parameters or multipart form fields**\n\n* `format` format identifier. Can also be specified in the URL path, e.g. `/json` is identical to `/?format=json`.\n\n* `version` optional version identifier. Can also be given as part of format as `format@version`\n\n* `select` optional selection of records within the posted data.\n\n* `file` File to be validated (form data only)\n\n* `data` Data to be validated (form data only)\n\n* `encoding` Optional alternative encoding (experimental)\n\n**Response**\n\nSame as response of [GET /validate](#get-validate).\n\n**Examples**\n\nThis file `schema.json` contains valid JSON but not a valid JSON Schema:\n\n```\n{\n  \"properties\": []\n}\n```\n\n```sh\ncurl -X POST 'http://format.gbv.de/validate/json-schema' --data-binary @schema.json\n```\n\n```json\n[\n  {\n    \"message\": \"must be object\",\n    \"position\": {\n      \"jsonpointer\": \"/properties\"\n    }\n  }\n]\n```\n\nThis file `articles.json` contains two records in `vzg-article` format, one invalid and one valid:\n\n```json\n[\n  { },\n  {\n    \"primary_id\": {\n      \"id\": \"123\",\n      \"type\": \"unknown\"\n    },\n    \"title\": \"An example article\",\n    \"lang_code\": [ \"ger\" ],\n    \"journal\": {\n      \"title\": \"Journal of Examples\",\n      \"year\": \"2022\"\n    }\n  }\n]\n\n```\n\nTo validate both records in one query, parameter `select=$.*` must be added:\n\n```sh\ncurl -X POST 'http://format.gbv.de/validate/vzg-article?select=$.*' --data-binary @articles.json\n```\n\n```json\n[\n  [\n    {\n      \"message\": \"must have required property 'primary_id'\",\n      \"position\": {\n        \"jsonpointer\": \"\"\n      }\n    }\n  ],\n  true\n]\n```\n\n### GET /formats\n\nLists all [data formats](#data-formats), optionally filtered by identifier, version, and/or schema type.\n\n**Query Parameters**\n\n* `format=[id]` select format with given format identifier\n\n* `version=[id]` version to filter for\n\n* `type=[string]` schema type filter for\n\n**Success Response**\n\nJSON Array of format objects.\n\n### GET /languages\n\nList schema languages as array of [data formats](#data-formats). The result is a subset of [GET /formats](#get-formats) with same query parameters and response format.\n\n### GET /schema\n\nGet a schema file.\n\n**Query Parameters**\n\n* `format=[id]` format identifier\n\n* `version=[string]` optional version (set to `default` by default)\n\n* `type=[string]` optional schema type\n\n**Success Response**\n\nThe schema file is served with corresponding content type.\n\n**Error Resonse**\n\nAn [API error](#api-errors) with status code 404 is returned in no corresponding schema was found.\n\n### Validation Errors\n\nValidation results (see [GET /validate](#get-validate) and [POST /{format}](#post-format)) can include validation errors. Each error is a JSON object with\n\n* `message` mandatory error message\n* `type` optional type of error, given as IRI\n* `position` optional object mapping locator formats to [locators](#locator-languages) (e.g.`rfc5147` to locate character positions in a string or `jsonpointer` to reference elements in a JSON document)\n\nErrors may contain additional keys but these may change with future versions of the service!\n\nFor instance the following validation error indicates that value of JSON key `authors` was not given as array:\n\n```json\n{\n  \"message\": \"must be array\",\n  \"position\": {\n    \"jsonpointer\": \"/authors\"\n  }\n}\n```\n\n### API Errors\n\nNon-validation errors such as wrong request parameters or unexpected internal failures are returned as JSON object such as the following:\n\n```json\n{\n  \"error\": \"MalformedRequest\",\n  \"code\": 400,\n  \"message\": \"Missing query parameter: format\"\n}\n```\n\nAPI error types include `MalformedRequest` and `NotFound`.\n\nA stack trace is included in development mode.\n\n## Maintainers\n\n- [@nichtich](https://github.com/nichtich)\n\n## Contributing\n\nPRs accepted against the `dev` branch. Never directly work on the main branch.\n\nFor releases (maintainers only) make changes on `dev` and then run the release script:\n\n```bash\nnpm run release:patch # or minor or major\n```\n\n## License\n\nMIT © 2022 Verbundzentrale des GBV (VZG)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgbv%2Fvalidation-server","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgbv%2Fvalidation-server","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgbv%2Fvalidation-server/lists"}