{"id":15044669,"url":"https://github.com/elastic/elasticsearch-specification","last_synced_at":"2025-05-16T15:08:16.700Z","repository":{"id":37244712,"uuid":"52318322","full_name":"elastic/elasticsearch-specification","owner":"elastic","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-15T08:42:11.000Z","size":647114,"stargazers_count":135,"open_issues_count":133,"forks_count":97,"subscribers_count":319,"default_branch":"main","last_synced_at":"2025-05-15T09:38:00.014Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elastic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-02-23T01:00:27.000Z","updated_at":"2025-05-15T08:36:21.000Z","dependencies_parsed_at":"2024-01-22T11:44:56.350Z","dependency_job_id":"6aa4b4cc-9ac7-46ae-86be-2256887aa8d6","html_url":"https://github.com/elastic/elasticsearch-specification","commit_stats":{"total_commits":2163,"total_committers":88,"mean_commits":"24.579545454545453","dds":0.8358760980120203,"last_synced_commit":"e7ffee1ba8b2972cc4a43dd0a83b731225684150"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-specification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-specification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-specification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-specification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elastic","download_url":"https://codeload.github.com/elastic/elasticsearch-specification/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254553958,"owners_count":22090417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T20:50:53.033Z","updated_at":"2025-05-16T15:08:11.682Z","avatar_url":"https://github.com/elastic.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Elasticsearch API Specification\n\nThe **Elasticsearch API Specification** provides the contract for communication between client and server components within the Elasticsearch stack.\nWith almost 500 API endpoints and around 3000 data types across the entire API surface, this project is a vitally important part of sustaining our engineering efforts at scale.\n\nThe repository has the following structure:\n\n| Path | Description |\n| -------- | ------- |\n| [`api-design-guidelines/`](api-design-guidelines/) | Knowledge base of best practices for API design. |\n| [`compiler/`](compiler/) | TypeScript compiler for specification definition to JSON. |\n| [`compiler-rs/`](compiler-rs/) | |\n| [`docs/`](docs/) | |\n| [`output/`](output/) | |\n| [`specification/`](specification/) | Elasticsearch request/response definitions in TypeScript. |\n| [`typescript-generator/`](typescript-generator/) | |\n\nThis JSON representation is formally defined by [a set of TypeScript definitions (a meta-model)](./compiler/src/model/metamodel.ts)\nthat also explains the various properties and their values.\n\n\n## Prepare the environment\n\nFor generating the JSON representation and running the validation code you need\nto install and configure Node.js in your development environment.\n\nYou can install Node.js with [`nvm`](https://github.com/nvm-sh/nvm):\n\n```sh\ncurl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash\n```\n\nOnce the installation is completed, install Node.js with `nvm`:\n\n```sh\n# this command will install the version configured in .nvmrc\nnvm install\n```\n\n## How to generate the JSON representation\n\n```\n# clone the project\n$ git clone https://github.com/elastic/elasticsearch-specification.git\n\n# install the dependencies\n$ make setup\n\n# generate the JSON representation\n$ make generate\n\n# the generated output can be found in ./output/schema/schema.json\n$ cat output/schema/schema.json\n```\n\n## How to generate the OpenAPI representation\n\nFollow the steps to generate the JSON representation, then:\n\n```\n# Generate the OpenAPI representation\n$ make transform-to-openapi\n\n# Apply fixes\n$ make overlay-docs\n\n# The generated output can be found in ./output/openapi/\n```\n\n## Contributing\n\nSee [CONTRIBUTING.md](./CONTRIBUTING.md)\n\n## Make Targets\n\n```\nUsage:\n  make \u003ctarget\u003e\n  validate         Validate a given endpoint request or response\n  validate-no-cache  Validate a given endpoint request or response without local cache\n  generate         Generate the output spec\n  compile          Compile the specification\n  license-check    Add the license headers to the files\n  license-add      Add the license headers to the files\n  spec-format-check  Check specification formatting rules\n  spec-format-fix  Format/fix the specification according to the formatting rules\n  spec-dangling-types  Generate the dangling types rreport\n  setup            Install dependencies for contrib target\n  clean-dep        Clean npm dependencies\n  transform-expand-generics  Create a new schema with all generics expanded\n  transform-to-openapi  Generate the OpenAPI definition from the compiled schema\n  filter-for-serverless  Generate the serverless version from the compiled schema\n  dump-routes      Create a new schema with all generics expanded\n  contrib          Pre contribution target\n  lint-docs        Lint the OpenAPI documents\n  lint-docs-serverless  Lint only the serverless OpenAPI document\n  help             Display help\n```\n\n### Structure of the JSON representation\n\nThe JSON representation is [formally defined as TypeScript definitions](./compiler/src/model/metamodel.ts).\nRefer to them for the full details. It is an object with two top level keys:\n\n```jsonc\n{\n  \"types\": [...],\n  \"endpoints\": [...]\n}\n```\n\nThe first one, `types`, contains all the type definitions from the specification, such as\n`IndexRequest` or `MainError`, while the second one, `endpoints`, contains every\nendpoint of Elasticsearch and the respective type mapping. For example:\n\n```jsonc\n{\n  \"types\": [    {\n    \"attachedBehaviors\": [\n      \"CommonQueryParameters\"\n    ],\n    \"body\": {\n      \"kind\": \"value\",\n      \"value\": {\n        \"kind\": \"instance_of\",\n        \"type\": {\n          \"name\": \"TDocument\",\n          \"namespace\": \"_global.index\"\n        }\n      }\n    },\n    \"generics\": [\n      {\n        \"name\": \"TDocument\",\n        \"namespace\": \"_global.index\"\n      }\n    ],\n    \"inherits\": {\n      \"type\": {\n        \"name\": \"RequestBase\",\n        \"namespace\": \"_types\"\n      }\n    },\n    \"kind\": \"request\",\n    \"name\": {\n      \"name\": \"Request\",\n      \"namespace\": \"_global.index\"\n    },\n    \"path\": [...],\n    \"query\": [...]\n  }, {\n    \"inherits\": {\n      \"type\": {\n        \"name\": \"WriteResponseBase\",\n        \"namespace\": \"_types\"\n      }\n    },\n    \"kind\": \"response\",\n    \"name\": {\n      \"name\": \"Response\",\n      \"namespace\": \"_global.index\"\n    }\n  }],\n  \"endpoints\": [{\n      \"accept\": [\n        \"application/json\"\n      ],\n      \"contentType\": [\n        \"application/json\"\n      ],\n      \"description\": \"Creates or updates a document in an index.\",\n      \"docUrl\": \"https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html\",\n      \"name\": \"index\",\n      \"request\": {\n        \"name\": \"Request\",\n        \"namespace\": \"_global.index\"\n      },\n      \"requestBodyRequired\": true,\n      \"response\": {\n        \"name\": \"Response\",\n        \"namespace\": \"_global.index\"\n      },\n      \"since\": \"0.0.0\",\n      \"stability\": \"stable\",\n      \"urls\": [...],\n      \"visibility\": \"public\"\n    }]\n}\n```\n\nThe example above represents the [index](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html)\nrequest, inside the `endpoints` array you can find the API name and the type mappings under `request.name`\nand `response.name`. The respective type definitons can be found inside the `types` array.\n\nIn some cases an endpoint might be defined, but there is no a type definition yet, in such case\nthe `request` and `response` value will be `null`.\n\n## How to validate the specification\n\nThe specification is validated daily by the [client-flight-recorder](https://github.com/elastic/clients-flight-recorder) project.\nThe validation result can be found [here](https://github.com/elastic/clients-flight-recorder/blob/main/recordings/types-validation/types-validation.md).\n\n### Validate the specification in your machine\n\nThe following step only apply if you don't have `~/.elastic/github.token` in place.\n\nCreate GitHub token to allow authentication with [Vault](https://www.vaultproject.io/).\n  * Go to https://github.com/settings/tokens.\n  * Click `Generate new token`.\n  * Give your token a name and make sure to click the `repo` and `read:org` scopes.\n  * Create a file at `~/.elastic/github.token` and paste the GitHub token into it.\n  * Change permissions on the file allow access only from the user.\n     `chmod 600 ~/.elastic/github.token`\n\nYou can see [here](https://github.com/elastic/infra/tree/master/docs/vault#github-auth)\nhow to generate a token.\n\nOnce you have configured the environment, run the following commands:\n\n```sh\ngit clone https://github.com/elastic/elasticsearch-specification.git\ngit clone https://github.com/elastic/clients-flight-recorder.git\n\ncd elasticsearch-specification\n# this will validate the xpack.info request type against the main branch of Elasticsearch\nmake validate api=xpack.info type=request branch=main\n\n# this will validate the xpack.info request and response types against the 8.15 branch\nmake validate api=xpack.info branch=8.15\n```\n\nThe last command above will install all the dependencies and run, download\nthe test recordings and finally validate the specification.\nIf you need to download the recordings again, run `make validate-no-cache api=xpack.info type=request branch=main`.\n\nOnce you see the errors, you can fix the original definition in `/specification`\nand then run the command again until the types validator does not trigger any new error.\nFinally open a pull request with your changes.\n\n## Documentation\n\n- [How to add a new API](./docs/add-new-api.md)\n- [Behaviors](./docs/behaviors.md)\n- [Compiler](./docs/compiler.md)\n- [Documenting the API specification](./docs/doc-comments-guide.md)\n- [Known issues](./docs/known-issues.md)\n- [Modeling Guide](./docs/modeling-guide.md)\n- [Schema structure](./docs/schema-structure.md)\n- [Specification structure](./docs/specification-structure.md)\n- [Style Guide](./docs/style-guide.md)\n- [Fixing a defintion, a complete story](./docs/validation-example.md)\n\n## FAQ\n\n### I want to see a report of how types and namespaces are being used.\n\nYou can find a report of the `main` branch [here](https://elastic.github.io/elasticsearch-specification/report.html).\n\n### A specific property is not always present, how do I define it?\n\nWhen you define a property the syntax is `propertyName: propertyType`.\nBy default a property is required to exist. If you know that a property will not\nalways be there, you can add a question mark just before the column:\n\n```ts\npropertyRequired: string\npropertyOptional?: string\n```\n\n### A definition is missing, how do I add it?\n\nSee [here](./docs/add-new-api.md).\n\n### A definition is not correct, how do I fix it?\n\nAll the definitons are inside `/specification` folder, search the bad defintion and update it,\nyou can find above how to run the validation of the spec.\n\n### An endpoint is missing, how do I add it?\n\nSee [here](./docs/add-new-api.md).\n\n### An endpoint definition is not correct, how do I fix it?\n\nAll the endpoint definitons are inside `/specification/_json_spec` folder, which contains a series of\nJSON files taken directly from the Elasticsearch rest-api-spec.\nYou should copy from there the updated endpoint defintion and change it here.\n\n### The validation in broken on GitHub but works on my machine!\n\nVery likely the recordings on your machine are stale, rerun the validation with the `validate-no-cache` make target.\n\nYou should pull the latest change from the `client-flight-recorder` as well.\n\n```sh\ncd client-flight-recorder\ngit pull\n```\n\n### Where do I find the generated test?\n\nEverytime you run `make validate` script, a series of test will be generated and dumped on disk.\nYou can find the failed tests in `clients-flight-recorder/scripts/types-validator/workbench`.\nThe content of this folder is a series of recorded responses from Elasticsearch wrapped inside an helper\nthat verifies if the type definiton is correct.\n\n### Which editor should I use?\n\nAny editor is fine, but to have a better development experience it should be configured\nto work with TypeScript. [Visual Studio Code](https://code.visualstudio.com/) and\n[IntelliJ IDEA](https://www.jetbrains.com/idea/) come with TypeScript support out of the box.\n\n### Is there a complete example of the process?\n\nYes, take a look [here](./docs/validation-example.md).\n\n### realpath: command not found\n\nThe validation script uses realpath which may be not present in your system.\nIf you are using MacOS, run the following command to fix the issue:\n\n```sh\nbrew install coreutils\n```\n\n### I need to modify the compiler, help!\n\nTake a look at the [compiler documentation](./docs/compiler.md).\n\n## BirdsEye overview\n\nThe work of several repositories come together in this repository.\nThis diagram aims to sketch an overview of how different pieces connect\n\n![overview.png](overview.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Felasticsearch-specification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felastic%2Felasticsearch-specification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Felasticsearch-specification/lists"}