{"id":23391450,"url":"https://github.com/ahrefs/esgg","last_synced_at":"2025-04-11T09:59:26.675Z","repository":{"id":46324829,"uuid":"176603784","full_name":"ahrefs/esgg","owner":"ahrefs","description":"ElasticSearch Guided (code) Generator","archived":false,"fork":false,"pushed_at":"2023-07-24T19:25:04.000Z","size":1580,"stargazers_count":10,"open_issues_count":1,"forks_count":2,"subscribers_count":34,"default_branch":"master","last_synced_at":"2025-04-03T00:17:03.539Z","etag":null,"topics":["elasticsearch","ocaml"],"latest_commit_sha":null,"homepage":"","language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ahrefs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-19T21:49:45.000Z","updated_at":"2025-02-06T15:59:12.000Z","dependencies_parsed_at":"2023-01-19T07:01:07.714Z","dependency_job_id":null,"html_url":"https://github.com/ahrefs/esgg","commit_stats":null,"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Fesgg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Fesgg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Fesgg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Fesgg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ahrefs","download_url":"https://codeload.github.com/ahrefs/esgg/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248371965,"owners_count":21093132,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","ocaml"],"created_at":"2024-12-22T04:17:29.237Z","updated_at":"2025-04-11T09:59:26.661Z","avatar_url":"https://github.com/ahrefs.png","language":"OCaml","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ElasticSearch Guided (code) Generator\n\n[![Build Status](https://travis-ci.org/ahrefs/esgg.svg?branch=master)](https://travis-ci.org/ahrefs/esgg)\n\n## Development\n\nInstall dependencies with `opam install --deps-only .`\n\nBuild with `make`\n\n## Mapping\n\n`esgg` takes as an input an ES [mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) (schema description)\nand actual query (with syntax for variables, described below). Often additional information is needed to map ES fields into proper OCaml types,\nthis is achieved by attaching `_meta` annotation object to the affected field (ES only supports `_meta` at root level, so these annotations make it\nimpossible to store extended mapping back into ES which is a pity), as follows:\n\n```json\n    \"counts\": {\n      \"_meta\": {\n        \"optional\": true\n      },\n      \"properties\": {\n        \"hash\": {\n          \"type\": \"long\",\n          \"_meta\": { \"repr\": \"int64\" }\n        },\n        \"value\": {\n          \"type\": \"long\"\n        }\n      }\n    },\n```\n\nSupported `_meta` attributes:\n\n* `{\"list\":true}` - property is an array (mapped to `list`)\n* `{\"list\":\"sometimes\"}` - property is either an array or single element (mapped to json with custom ocaml module wrap that will need to be provided in scope)\n* `{\"optional\":\u003ctrue|false\u003e}` - property may be missing (mapped to `option`)\n* `{\"ignore\":true}` - skip property altogether\n* `{\"fields_default_optional\":true}` - any subfield may be missing (can be overriden by per-field `optional:false`)\n* `{\"repr\":\"int64\"}` - override ES `type`, currently the only possible value is `\"int64\"` to ensure no bits are lost (by default `long` is mapped to OCaml `int`)\n\n### Host types mapping\n\nGenerated code allows to use application types for any fields. This is achieved by referencing specific type for each field in generated\ncode, instead of the primitive type from the mapping, allowing consumer of the code to map it onto custom type etc. For example the field\n`hash` in example above will have type `Counts.Hash.t` in generated code. In order to compile the generated code this type must be present\nin scope and mapped to something useful. Default mapping (which just maps everything to corresponding primitive types) can be generated\nwith `esgg reflect \u003cmapping name\u003e \u003cmapping.json\u003e`, e.g.:\n\n```bash\nesgg reflect hello_world src/mappings/hello_world.json \u003e\u003e src/mapping.ml\n```\n\nwill generate the following, which should be edited manually as needed, e.g. by making `Hash` a module with an abstract type\n\n```ocaml\n  module Counts = struct\n    module Hash = Id_(Int64_)\n    module Value = Id_(Long_)\n  end\n\n```\n\n## Variables\n\nSyntax for variables in template json files is as follows:\n\n  - `$var` for regular required variable\n  - `$var?` for optional variable (minimal surrounding scope is conditionally expunged)\n  - full form `$(var:hint)` where `hint` can be either `list` or `list?` currently\n\n## Reusing shared ATD definitions\n\nTo reuse shared definitions using the `-shared \u003cfile.atd\u003e` option, the `atd` file must have the `\u003cesgg from=\"...\"\u003e` annotation at the top of the file.\nThe value of the annotation must correspond to the OCaml module containing the shared definitions.\n\nExample:\n```\n# file.atd\n\n\u003cesgg from=\"Your_ocaml_module_name\"\u003e\n\n...atd type definitions...\n```\n\n\n## Elasticsearch features\n\nTODO document what is supported\n\nSome notes follow:\n\n### [aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html)\n\nThe following aggregation types are supported:\n\n#### [filters](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filters-aggregation.html)\n\n  - [x] named\n  - [x] anonymous\n  - [x] dynamic (i.e. a variable)\n  - [x] partial dynamic (i.e. containing variables)\n  - [x] other_bucket and other_bucket_key\n  - [ ] other_bucket with anonymous filters (ignored, user is responsible to treat last element of result specially)\n\nDynamic (defined at runtime) filters are supported, as follows `{ \"filters\": { \"filters\": $x } }`.\nIn this case corresponding part of output will be quite untyped. `$x` is assumed to be a dictionary and result will be represented with\ndictionaries. For anonymous filters (ie array of filters) use `$(x:list)`.\n\n#### date_histogram\n\n`key_as_string` is returned in output only when `format` is\n[explicitly specified](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/search-aggregations-bucket-datehistogram-aggregation.html#_keys),\nto discourage fragile code.\n\n#### [range](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-range-aggregation.html)\n\n  Keyed aggregation expects explicit `key` for each range. `from`/`to` fields in response are not extracted.\n\n#### [date_range](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-daterange-aggregation.html)\n\n  same as for range aggregation\n\n#### others\n\n  - min\n  - max\n  - avg\n  - sum\n  - value_count\n  - cardinality\n  - terms\n  - significant_terms\n  - significant_text\n  - histogram\n  - top_hits\n  - nested\n  - reverse_nested\n  - [bucket_sort](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-sort-aggregation.html)\n  - [cumulative_sum](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-cumulative-sum-aggregation.html)\n\n#### dynamic\n\n  Specifying aggregation as variable (`$var`) will lead to an untyped json in place of aggregation output, this can be used as temporary\n  workaround for unsupported aggregation types or for truly dynamic usecase (aggregation built at run-time).\n\n### [script](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-using.html)\n\nScripts are opaque, ie no type information is extracted and result is json.\n\n### [source filtering](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-source-filtering.html)\n\n- [x] exclude and include\n- [ ] wildcards\n- [x] dynamic (i.e. a variable) NB not implemented for get and mget\n\n## Tests\n\n`make test` runs regression tests in [test/](test/) verifying\nthat input and output atd generated from query stays unchanged.\nOnce there is an expected change in generated query - it should be committed.\nTests are easy to add and fast to run.\n\nTODO tests to verify that:\n\n  * code generated for query application of input variables does actually compile and produce correct query when run\n  * atd description of output (generated from query) can indeed unserialize ES output from that actual query\n\n## Conditions\n\nCopyright (c) 2018 Ahrefs \u003cgithub@ahrefs.com\u003e\n\nThis project is distributed under the terms of GPL Version 2. See LICENSE file for full license text.\n\nNB the output of esgg, i.e. the generated code, is all yours of course :)\n\n----\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahrefs%2Fesgg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahrefs%2Fesgg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahrefs%2Fesgg/lists"}