{"id":21192152,"url":"https://github.com/mu-semtech/mu-search","last_synced_at":"2026-02-09T15:15:14.096Z","repository":{"id":37964808,"uuid":"187207072","full_name":"mu-semtech/mu-search","owner":"mu-semtech","description":"Search facility for mu-semtech, powered by ElasticSearch","archived":false,"fork":false,"pushed_at":"2026-02-02T14:26:13.000Z","size":70592,"stargazers_count":0,"open_issues_count":19,"forks_count":11,"subscribers_count":5,"default_branch":"master","last_synced_at":"2026-02-03T03:21:33.840Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mu-semtech.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-05-17T11:46:44.000Z","updated_at":"2026-02-02T13:54:07.000Z","dependencies_parsed_at":"2024-01-16T18:29:38.483Z","dependency_job_id":"29ce5a38-d89c-4267-984f-e53fb3a75310","html_url":"https://github.com/mu-semtech/mu-search","commit_stats":null,"previous_names":[],"tags_count":45,"template":false,"template_full_name":null,"purl":"pkg:github/mu-semtech/mu-search","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fmu-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fmu-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fmu-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fmu-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mu-semtech","download_url":"https://codeload.github.com/mu-semtech/mu-search/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mu-semtech%2Fmu-search/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29270309,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-09T13:47:44.167Z","status":"ssl_error","status_checked_at":"2026-02-09T13:47:43.721Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T19:07:44.623Z","updated_at":"2026-02-09T15:15:14.088Z","avatar_url":"https://github.com/mu-semtech.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mu-search\n\nA component to integrate [authorization-aware](https://github.com/mu-semtech/mu-authorization) full-text search into a [mu.semte.ch stack](https://github.com/mu-semtech/mu-project) using [Elasticsearch](https://www.elastic.co/).\n\n## Tutorials\n### Add mu-search to a stack\nThe mu-search service uses Elasticsearch as a backend. Since the Elasticsearch docker image requires a lot of memory, increase the maximum on your system by executing the following command:\n\n```bash\nsysctl -w vm.max_map_count=262144\n```\n\nNext, add the mu-search and accompanying elasticsearch service to `docker-compose.yml`\n\n```yml\nservices:\n  search:\n    image: semtech/mu-search:0.11.0\n    links:\n      - db:database\n    volumes:\n      - ./config/search:/config\n  elasticsearch:\n    image: semtech/mu-search-elastic-backend:1.3.0\n    volumes:\n      - ./data/elasticsearch/:/usr/share/elasticsearch/data\n    environment:\n      - discovery.type=single-node\n\n```\n\nNote: because elasticsearch doesn't run as root in its container, it will mess up file permissions on the host system's mounted volumes. The current workaround is to add the directory to your app's git repo with a .gitkeep file and to set the permissions of the directory to 777, then committing this file to your repo.\n\nThe indices will be persisted in `./data/elasticsearch`. The `search` service needs to be linked to an instance of the [mu-authorization](https://github.com/mu-semtech/mu-authorization) service.\n\nCreate the `./config/search` directory and create a `config.json` with the following contents:\n\n```json\n{\n    \"types\" : [\n        {\n            \"type\" : \"document\",\n            \"on_path\" : \"documents\",\n            \"rdf_type\" : \"http://xmlns.com/foaf/0.1/Document\",\n            \"properties\" : {\n                \"title\" : \"http://purl.org/dc/elements/1.1/title\",\n                \"description\" : \"http://purl.org/dc/elements/1.1/description\"\n            }\n        },\n        {\n            \"type\" : \"user\",\n            \"on_path\" : \"users\",\n            \"rdf_type\" : \"http://xmlns.com/foaf/0.1/Person\",\n            \"properties\" : {\n                \"fullname\" : \"http://xmlns.com/foaf/0.1/name\"\n            }\n         }\n    ]\n}\n```\n\nFinally, add the following rules to your dispatcher configuration in `./config/dispatcher.ex` to make the search endpoint available:\n\n```elixir\n  define_accept_types [\n    json: [ \"application/json\", \"application/vnd.api+json\" ]\n  ]\n\n  @json %{ accept: %{ json: true } }\n\n  get \"/search/*path\", @json do\n    Proxy.forward conn, path, \"http://search/\"\n  end\n\n```\n\nRestart the dispatcher service to pick up the new configuration\n```bash\ndocker-compose restart dispatcher\n```\n\nRestart the stack using `docker-compose up -d`. The `elasticsearch` and `search` services will be created.\n\nSearch queries can now be sent to the `/search` endpoint. Make sure the user has access to the data according to the authorization rules.\n\n## How-to guides\n### How to persist indexes on restart\nBy default search indexes are deleted on (re)start of the mu-search service. This guide describes how to make sure search indexes are persisted on restart. Obviously, this configuration is recommended on production environments.\n\nFirst, make sure the search indexes are written to a mounted volume by specifying a bind mount to `/usr/share/elasticsearch/data` on the Elasticsearch container.\n\n```yml\nservices:\n  elasticsearch:\n    image: semtech/mu-search-elastic-backend:1.2.0\n    volumes:\n      - ./data/elasticsearch/:/usr/share/elasticsearch/data\n```\n\nRecreate the `elasticsearch` container by executing the following command\n\n```bash\ndocker-compose up -d\n```\n\nNext, enable the persistent indexes flag in the root of the search configuration file `./config/search/config.json` of your project.\n```javascript\n{\n  \"persist_indexes\": true,\n  \"types\": [\n    // index type specifications\n  ]\n}\n```\n\nRestart the `search` service to pick up the new configuration.\n\n```bash\ndocker-compose restart search\n```\n\nSearch indexes will be persisted in `./data/elasticsearch` folder and not be deleted on restart of the search service.\n\n### How to prepare a search index on startup\nThe search API provided by mu-search is authorization-aware. I.e. search results will only contain resources the user is allowed to access. To this end mu-search organises its search indexes per access right. Based on the user's allowed groups set on the incoming search requests, mu-search determines which indexes to search in.\n\nIndexes that don't exist yet will be created before the search operation is performed. Depending on the number of documents to index this may be a time-consuming operation.\n\nMu-search allows to configure authorization groups for which the indexes need to be created on startup already. This will save time at the moment the first search query for that profile arrives.\n\nConfiguration is done via the `eager_indexing_groups` in the search configuration file `./config/search/config.json`. The eager indexing groups are tightly related to the `GroupSpec`objects configured in [mu-authorization](https://github.com/mu-semtech/mu-authorization#groupspec).\n\nThe `eager_indexing_groups` is an array of group specifications. Each group specification is defined by an array of objects in which each object consists of:\n- **name**: name of the group specification (`GroupSpec`) in mu-authorization\n- **variables**: array of string values used to construct the graph URI for the group. These variables should match the possible result values of the `vars` in case of an `AccessByQuery` access rule in the `GroupSpec`. In case of an `AlwaysAccessible` access rule, this should be an empty array.\n\n#### Example: public data for unauthenticated users\nIf the application only provides public data for unauthenticated users in the graph `http://mu.semte.ch/graphs/public`, the following eager indexing groups must be configured:\n\n```javascript\n[\n  [ { \"name\": \"public\", \"variables\" : [] } ],\n  [ { \"name\": \"clean\", \"variables\": [] } ]\n]\n```\n\n#### Example: data per organization unit\nIf, next to the public data, data is organized per organization unit in graphs like `http://mu.semte.ch/graphs/\u003cunit-name\u003e`, the following eager indexing groups must be configured:\n\n```javascript\n[\n  [ { \"name\": \"public\", \"variables\" : [] }, { \"name\": \"organization-unit\", \"variables\" : [\"finance\"] } ],\n  [ { \"name\": \"public\", \"variables\" : [] }, { \"name\": \"organization-unit\", \"variables\" : [\"legal\"] } ],\n  [ { \"name\": \"clean\", \"variables\": [] } ]\n]\n```\n\nIn case a group contains a variable, an eager index must be configured for each possible value if you want all search indexes to be prepared upfront.\n\nEager indexes may be combined at search time to match the user's allowed groups. For example, if some users have access to the data of the finance department as well as the legal department, both indexes will be queried when the user performs a search operation.\n\n### How to integrate mu-seach with delta's to update search indexes\nThis how-to guide explains how to integrate mu-search with the delta-notification in order to automatically update search index entries when data in the triplestore is modified.\n\nThis guide assumes the [mu-authorization](https://github.com/mu-semtech/mu-authorization#add-mu-authorization-to-a-stack) and [delta-notifier](https://github.com/mu-semtech/delta-notifier) components have been added to your stack as explained in their respective installation guides.\n\nOpen the delta-notifier rules configuration `./config/delta/rules.js` and add the following rule:\n\n```javascript\n  {\n    match: {\n      // listen to all changes\n    },\n    callback: {\n      url: 'http://search/update',\n      method: 'POST'\n    },\n    options: {\n      resourceFormat: \"v0.0.1\",\n      gracePeriod: 10000,\n      ignoreFromSelf: true\n    }\n  }\n```\n\nEnable automatic index updates (not only invalidation) in mu-search by setting the `automatic_index_updates` flag at the root of `./config/search/config.json`.\n\n```javascript\n{\n  \"automatic_index_updates\": true,\n  \"types\": [\n     // definition of the indexed types\n  ]\n}\n```\n\nRestart the search and delta-notifier service.\n```bash\ndocker-compose restart search delta-notifier\n```\n\nAny change you make in your application will now trigger a request to the `/update` endpoint of mu-search. Depending on the indexed resources and properties, mu-search will update the appropriate search index entries.\n\n### How to specify a file's content as property\nThis guide explains how to make the content of files attached to a project resource searchable in the index.\n\nThis guide assumes you have already integrated mu-search in your application and configured an index for resources of type `schema:Project`.\n\nFor indexing files mu-search requires a Tika server to extract the content. Add the `tika` service next to the `search` and `elasticsearch` services in `docker-compose.yml`:\n```yml\nservices:\n  search:\n    ...\n  elasticsearch:\n    ...\n  tika:\n    image: semtech/mu-search-tika-backend:1.0.0\n```\n\nNext, add the following mounted volumes to the mu-search service in `docker-compose.yml`:\n- `/data`: folder containing the files to be indexed\n- `/cache`: folder to persist Tika's text extraction cache\n\n```yml\nservices:\n  search:\n    image: semtech/mu-search:0.11.0\n    volumes:\n      - ./config/search:/config\n      - ./data/files:/data\n      - ./data/search/cache:/cache\n```\n\nNext, add a property `files` in the `project` type index configuration. The property `files` will hold the content and metadata of the files.\n\n```javascript\n{\n    \"types\" : [\n        {\n            \"type\" : \"project\",\n            \"on_path\" : \"projects\",\n            \"rdf_type\" : \"http://schema.org/Project\",\n            \"properties\" : {\n                \"name\" : \"http://schema.org/name\",\n                \"files\" : {\n                   \"via\" : [\n                       \"http://purl.org/dc/terms/hasPart\",\n                       \"^http://www.semanticdesktop.org/ontologies/2007/01/19/nie#dataSource\"\n                   ],\n                   \"attachment_pipeline\" : \"attachment\"\n                 }\n            }\n        }\n    ]\n}\n```\n`via` expresses the path from the indexed resource to the file(s) having a URI like `\u003cshare://path/to/your/file.pdf\u003e`.\n\nRecreate the mu-search service using\n\n```bash\ndocker-compose up -d\n```\n\nAfter reindex has been completed, each indexed project will now contain a property `files` holding the content and metadata of the files linked to the project via `dct:hasPart/^nie:dataSource`.\n\nSearching the file's content is done using the nested property `content` on the defined field name, `files` in this case:\n\n```\nGET /documents/search?filter[files.content]=open-source\"\n```\n\n### How to inspect the content of a search index\nThe content of a search index can be inspected by running a [Kibana](https://www.elastic.co/kibana) dashboard on top of Elasticseach by adding the following snippet to your `docker-compose.override.yml`\n\n```yaml\nservices:\n  kibana:\n    image: docker.elastic.co/kibana/kibana:7.17.0\n    environment:\n      ELASTICSEARCH_HOSTS: \"http://elasticsearch:9200\"\n    ports:\n      - 127.0.0.1:5601:5601\n    user: root\n    command: |\n      sh -c \"/usr/local/bin/kibana-docker --allow-root;\"\n```\n\nStart the container\n```bash\ndocker-compose up -d kibana\n```\n\nOnce Kibana has started the dashboard is available at http://localhost:5601\n\nMake sure not to expose the Kibana dashboard in a production environment!\n\n### How to reset search indexes\n[To be completed...]\n\n## Reference\n### Search index configuration\nElasticsearch is used as a search engine. It indexes documents according to a specified configuration and provides a REST API to search documents. The mu-search service is a layer in front of Elasticsearch that allows to specify the mapping between RDF triples and the Elasticsearch documents/properties. It also integrates with [mu-authorization](https://github.com/mu-semtech/mu-authorization) making sure users can only search for documents they're allowed to access.\n\nThis section describes how to configure the resources and properties to be indexed and how to pass Elasticsearch specific configurations and mapping in the mu-search configuration file.\n\n#### Indexed resource types and properties\nThis section describes how to mapping between RDF triples and Elasticsearch documents can be specified in the mounted `/config/config.json` configuration file.\n\nThe `config.json` file contains a JSON object with a property `types`. This property contains an array of objects, one per document type that must be searchable.\n\n```javascript\n{\n  \"types\": [\n    // object per searchable document type\n  ]\n}\n```\n\nNote that these types do not map one-on-one with the search indexes in Elasticsearch. For each document type in the list a search index will be created **per** authorization group.\n\nEach type object in the `types` array consists of the following properties:\n- **type** : name of the type\n- **on_path** : path on which the search endpoint will be published\n- **rdf_type** : URI of the rdf:Class of the documents to index\n- **properties** : mapping of RDF predicates to document properties\n- **settings** : [type specific Elasticsearch settings](#elasticsearch-settings)\n- **mappings** : [type specific Elasticsearch mapping](#elasticsearch-mappings)\n\n`properties` contains a JSON object with a key per property in the resulting Elasticsearch document. These are the properties that will be searchable via the search API for the given resource type. The value of each key defines the mapping to RDF predicates starting from the root resource.\n\n**WARNING**: there are two protected fields that should not be used as property keys: `uuid` and `uri`. Both are used internally by the mu-search service to store the uuid and URI of the root resource.\n\n##### Simple properties\nIn the simplest scenario, the properties that need to be searchable map one-by-one on a predicate (path) of the resource.\n\nIn the example below, a search index per user group will be created for documents and users. The documents index contains resources of type `foaf:Document`s with a `title` and `description`. The users index contains `foaf:Person`s with only `fullname` as searchable property.\n\n```javascript\n{\n    \"types\" : [\n        {\n            \"type\" : \"document\",\n            \"on_path\" : \"documents\",\n            \"rdf_type\" : \"http://xmlns.com/foaf/0.1/Document\",\n            \"properties\" : {\n                \"title\" : \"http://purl.org/dc/elements/1.1/title\",\n                \"description\" : \"http://purl.org/dc/elements/1.1/description\"\n            }\n        },\n        {\n            \"type\" : \"user\",\n            \"on_path\" : \"users\",\n            \"rdf_type\" : \"http://xmlns.com/foaf/0.1/Person\",\n            \"properties\" : {\n                \"fullname\" : \"http://xmlns.com/foaf/0.1/name\"\n            }\n         }\n    ]\n}\n```\n\nIf multiple values are found in the triplestore for a given predicate, the resulting value for the property in the search document will be an array of all values.\n\nA property of the search document may also map to an inverse predicate. I.e. resource to be indexed is the object instead of the subject of the triple. An inverse predicate can be indicated in the mapping by prefixing the predicate URI with `^` as done in a SPARQL query.\n\nIn the example below the users index contains a property `group` that maps to the inverse predicate `foaf:member` relating a group to a user.\n\n```javascript\n{\n    \"types\" : [\n        {\n            \"type\" : \"user\",\n            \"on_path\" : \"users\",\n            \"rdf_type\" : \"http://xmlns.com/foaf/0.1/Person\",\n            \"properties\" : {\n                \"fullname\" : \"http://xmlns.com/foaf/0.1/name\",\n                \"group\": \"^http://xmlns.com/foaf/0.1/member\"\n            }\n         }\n    ]\n}\n```\n\nProperties can also be mapped to lists of predicates, corresponding to a property path in RDF. In this case, the property value is an array of strings. One string per path segment. The array starts from the indexed resource and may also include [inverse predicate URIs](#inverse-properties).\n\nIn the example below the documents index contains a property `topics` that maps to the label of the document's primary topic and a property `publishers` that maps to the names of the publishers via the inverse `foaf:publications` predicate.\n\n```javascript\n{\n    \"types\" : [\n        {\n            \"type\" : \"document\",\n            \"on_path\" : \"documents\",\n            \"rdf_type\" : \"http://xmlns.com/foaf/0.1/Document\",\n            \"properties\" : {\n                \"title\" : \"http://purl.org/dc/elements/1.1/title\",\n                \"description\" : \"http://purl.org/dc/elements/1.1/description\",\n                \"topics\" : [\n                  \"http://xmlns.com/foaf/0.1/primaryTopic\",\n                  \"http://www.w3.org/2004/02/skos/core#prefLabel\"\n                ],\n                \"publishers\": [\n                  \"^http://xmlns.com/foaf/0.1/publications\",\n                  \"http://xmlns.com/foaf/0.1/name\"\n                ]\n            }\n        }\n    ]\n}\n```\n\n\n##### File content property\nTo make the content of a file searchable, it needs to be indexed as a property in a search index. Basic indexing of PDF, Word etc. files is provided using a local [Apache Tika](https://tika.apache.org/) instance. A default ingest pipeline named `attachment` is created on startup of the mu-search service. Note that this is under development and liable to change.\n\nDefining a property to index the content of a file requires the following keys:\n- **via** : mapping of the RDF predicate (path) that relates the resource with the file(s) to index. The file URI the predicate path leads to must have a URI starting with `share://` indicating the location of the file. E.g. `\u003cshare://path/to/your/file.pdf\u003e`.\n- **attachment_pipeline** : attachment pipeline to use for indexing the files. Set to `attachment` to use the default ingest pipeline.\n\nThe example below adds a property `files` in the `project` type index configuration. The property `files` will hold the contents of the files related to the project via `dct:hasPart/^nie:dataSource`.\n\n```javascript\n{\n    \"types\" : [\n        {\n            \"type\" : \"project\",\n            \"on_path\" : \"projects\",\n            \"rdf_type\" : \"http://schema.org/Project\",\n            \"properties\" : {\n                \"name\" : \"http://schema.org/name\",\n                \"files\" : {\n                   \"via\" : [\n                       \"http://purl.org/dc/terms/hasPart\",\n                       \"^http://www.semanticdesktop.org/ontologies/2007/01/19/nie#dataSource\"\n                   ],\n                   \"attachment_pipeline\" : \"attachment\"\n                 }\n            }\n        }\n    ]\n}\n```\n\nFor each file retrieved through the `via`-definition, the Tika-processing results in an object containing the extracted text (as `content`), as well as other extracted metadata (in the future). Such object may look like this:\n\n```javascript\n{\n  content: \"Extracted text here\"\n}\n```\n\nThese objects are structured in the same way as the `attachment` objects resulting from the [Elasticsearch's Ingest Attachment Processor Plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/ingest-attachment.html). Keep in mind that this implies you need to specify the path to a specific property of the attachment object when defining an Elasticsearch mapping. E.g. mapping the file's content for the `files` field from the example above may look as follows:\n\n```javascript\n{\n  \"types\": [\n    {\n      \"type\": \"project\",\n      \"on_path\": \"projects\",\n      ...\n      \"mappings\" : {\n        \"properties\": {\n          \"name\" : { \"type\" : \"text\" },\n          \"files.content\" : { \"type\" : \"text\" }\n        }\n      }\n    },\n    // other type definitions\n  ]\n}\n```\n\nCurrently, only indexing of local files is supported. The files' logical path as well as other metadata is expected to be in the format specified by the [file-service](https://github.com/mu-semtech/file-service#data-model). Files must be present in the Docker volume `/data` inside the container.\n\nAttachments processed by Tika are cached in the directory `/cache` (by SHA256 of the file contents). This must be defined as a shared volume for the cache to be persistent.\n\nSee also \"How to specify a file's content as property\".\n\n##### [Experimental] Combining resources of multiple types into one index\nIt's possible to map several resources of different rdf classes onto one index where that makes sense, e.g. if they share the same properties.\n\nin config.json:\n```\n      \"rdf_type\": [\n        \"http://data.vlaanderen.be/ns/besluit#Bestuurseenheid\",\n        \"http://data.lblod.info/vocabularies/erediensten/CentraalBestuurVanDeEredienst\",\n        \"http://data.lblod.info/vocabularies/erediensten/BestuurVanDeEredienst\",\n        \"http://data.lblod.info/vocabularies/erediensten/RepresentatiefOrgaan\"\n      ],\n```\n\nNote that this is different from a composite index, where each type has its own index, as well as being indexed in the composite index. Another difference is that the composite index allows mapping different properties from the sub indexes onto one property in the composite index.\n\n##### Nested objects\nA search document can contain nested objects up to an arbitrary depth. For example for a person you can nest the address object as a property of the person search document.\n\nA nested object is defined by the following properties:\n- **via** : mapping of the RDF predicate that relates the resource with the nested object. May also be an inverse URI, or a list of predicate (a property path) as in non-nested properties\n- **rdf_type** : URI of the rdf:Class of the nested object\n- **properties** : mapping of RDF predicates to properties for the nested object\n\nObjects can be nested to arbitrary depth. The properties object is defined in the same way as the properties of the root document, but the properties of a nested object **cannot** contain file attachments.\n\n[Elasticsearch mappings](#elasticsearch-mappings) for nested objects must be specified in the `mappings` object at the root type using a path expression as key.\n\nIn the example below the document's creator is nested in the `author` property of the search document. The nested person object contains properties `fullname` and the current project's title as `project`.\n\n```json\n{\n    \"types\" : [\n        {\n            \"type\" : \"document\",\n            \"on_path\" : \"documents\",\n            \"rdf_type\" : \"http://xmlns.com/foaf/0.1/Document\",\n            \"properties\" : {\n                \"title\" : \"http://purl.org/dc/elements/1.1/title\",\n                \"description\" : \"http://purl.org/dc/elements/1.1/description\",\n                \"author\" : {\n                    \"via\" : \"http://purl.org/dc/elements/1.1/creator\",\n                    \"rdf_type\" : \"http://xmlns.com/foaf/0.1/Person\",\n                    \"properties\" : {\n                        \"fullname\" : \"http://xmlns.com/foaf/0.1/name\",\n                        \"project\": [\n                            \"http://xmlns.com/foaf/0.1/currentProject\",\n                            \"http://purl.org/dc/elements/1.1/title\"\n                        ]\n                    }\n                }\n            },\n            \"mappings\": {\n              \"properties\": {\n                \"title\" : { \"type\" : \"text\" },\n                \"author.fullname\": { \"type\" : \"text\" }\n              }\n            }\n        }\n    ]\n}\n```\n\n*NOTE*: currently mu-search does not take the rdf_type of the nested object into account. In the above example, any resource linked via the dct:creator predicate would be included in the elasticsearch document.\n\n##### [Experimental] Multilingual properties\nMu-search has experimental support for multilingual values. This can be done by setting the type of a property to `language-string`. Background on this feature can be found in [rfcs/multi-language-search.md](rfcs/multi-language-search.md)\n\nFor example:\n```json\n{\n    \"types\" : [\n        {\n            \"type\" : \"document\",\n            \"on_path\" : \"documents\",\n            \"rdf_type\" : \"http://xmlns.com/foaf/0.1/Document\",\n            \"properties\" : {\n                \"title\" : {\n                  \"via\": \"http://purl.org/dc/elements/1.1/title\",\n                  \"type\": \"language-string\"\n                }\n            },\n            \"mappings\": {\n              \"properties\": {\n                \"title\": {\n                  \"properties\": {\n                    \"default\" : { \"type\" : \"text\" },\n                    \"en\": { \"type\" : \"text\" }\n                  }\n                }\n              }\n            }\n         }\n      ]\n}\n```\n\nWhen setting a property type to language-string, mu-search will include the language tag of the literal in the search index. In the above example the title field would be expanded to a language container in the document:\n```json\n{\n  \"title\": {\n    \"en\": [\"the english title\"],\n    \"default\": [\"this literal had no language tag\"]\n  }\n}\n```\nLiterals without a language string are mapped onto the \"default\" field.\n\nFor searching, make sure to either specify the appropriate field (`filter[title.en]=xyz` or make use of a wildcard: `filter[title.*]=xyz`.\n\nIt's often advised to configure language specific analyzers for each language, this can be done in the mappings sections of the configuration.\n\n##### [Experimental] Composite types\nA search index can contain documents of different types. E.g. documents (`foaf:Document`) as well as creative works (`schema:CreativeWork`). Currently, each simple type the composite index is constituted of must be defined separately in the index configuration as well.\n\nA definition of a composite type index consists of the following properties:\n- **type** : name of the composite type\n- **composite_types** : list of simple type names that constitute the index\n- **on_path** : path on which the search endpoint will be published\n- **properties** : mapping of RDF predicates to document properties for each simple type\n\nIn contrast to the `properties` of a simple index, the `properties` of a composite index is an array. Each entry in the array is an object with the folliwng properties:\n- **name** : name of property of the search document\n- **mappings** : mapping to the simple type property per simple type. If the mapping for a simple type is absent, the same property name as the composite document is assumed.\n\nThe example below contains 2 simple indexes for documents and creative works, and a composite index `dossier` containing both simple index types. The composite index contains (1) a property `name` mapping to the document's `title` and creative work's `name` property respectively, and (2) a property `description` mapping to the `description` property for both simple types.\n\n```javascript\n{\n    \"types\" : [\n        {\n            \"type\" : \"document\",\n            \"on_path\" : \"documents\",\n            \"rdf_type\" : \"http://xmlns.com/foaf/0.1/Document\",\n            \"properties\" : {\n                \"title\" : \"http://purl.org/dc/elements/1.1/title\",\n                \"description\" : \"http://purl.org/dc/elements/1.1/description\"\n            }\n        },\n        {\n            \"type\" : \"creative-work\",\n            \"on_path\" : \"creative-works\",\n            \"rdf_type\" : \"http://schema.org/CreativeWork\",\n            \"properties\" : {\n                \"name\": \"http://schema.org/name\",\n                \"description\": \"http://schema.org/description\"\n            }\n         },\n         {\n            \"type\" : \"dossier\",\n            \"composite_types\" : [\"document\", \"creative-work\"],\n            \"on_path\" : \"dossiers\",\n            \"properties\" : [\n                {\n                    \"name\" : \"name\",\n                    \"mappings\" : {\n                        \"document\" : \"title\",\n                        \"creative-work\" : \"name\"\n                    }\n                },\n                {\n                    \"name\" : \"description\",\n                    \"mappings\" : {\n                        \"document\" : \"description\"\n                        // mapping for 'creative-work' is missing, hence same property name 'description' is assumed\n                    }\n                }\n            ]\n         }\n    ]\n}\n```\n#### Using Prefixes\nTo make the configuration more concise and maintainable, you can define prefixes for commonly used URI namespaces. Prefixes are defined at the root level of the configuration using the `prefixes` property:\n\n```json\n{\n  \"prefixes\": {\n    \"foaf\": \"http://xmlns.com/foaf/0.1/\",\n    \"dct\": \"http://purl.org/dc/terms/\",\n    \"skos\": \"http://www.w3.org/2004/02/skos/core#\"\n  },\n  \"types\": [\n    {\n      \"type\": \"document\",\n      \"on_path\": \"documents\",\n      \"rdf_type\": \"foaf:Document\",\n      \"properties\": {\n        \"title\": \"dct:title\",\n        \"label\": \"skos:prefLabel\",\n        \"creator\": \"^foaf:made\"\n      }\n    }\n  ]\n}\n```\n\n#### Elasticsearch settings\nElasticsearch provides a lot of [index configuration settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html) for analysis, logging, etc. Mu-search allows to provide this configuration for the whole domain and/or to be overridden (currently not merged!) on a per-type basis.\n\nTo specify Elasticsearch settings for all indexes, use `default_settings` next to the `types` specification:\n\n```javascript\n  \"types\" : [\n     // definition of the indexed types\n  ],\n  \"default_settings\" : {\n    \"analysis\": {\n      \"analyzer\": {\n        \"dutchanalyzer\": {\n          \"tokenizer\": \"standard\",\n          \"filter\": [\"lowercase\", \"asciifolding\", \"dutchstemmer\"]\n        }\n      },\n      \"filter\": {\n        \"dutchstemmer\": {\n          \"type\": \"stemmer\",\n          \"name\": \"dutch\"\n        }\n      }\n    }\n  }\n```\n\nThe content of the `default_settings` object is not elaborated here but can be found in the official [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html). All settings provided in `settings` in the Elasticsearch configuration can be used verbatim in the `default_settings` of mu-search.\n\nTo specify Elasticsearch settings for a single type, use `settings` on the type index specification:\n\n```javascript\n{\n  \"types\": [\n    {\n      \"type\": \"document\",\n      \"on_path\": \"documents\",\n      ...\n      \"settings\" : {\n        \"analysis\": {\n          \"analyzer\": {\n            \"dutchanalyzer\": {\n              \"tokenizer\": \"standard\",\n              \"filter\": [\"lowercase\", \"asciifolding\", \"dutchstemmer\"]\n            }\n          },\n          \"filter\": {\n            \"dutchstemmer\": {\n              \"type\": \"stemmer\",\n              \"name\": \"dutch\"\n            }\n          }\n      }\n    },\n    // other type definitions\n  ]\n}\n```\n\n#### Elasticsearch mappings\nElasticsearch provides the option to configure [a mapping per index](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) to specify how the properties of a document are stored and indexed. E.g. the type of the property value (string, date, boolean, ...), text-analysis to be applied on the value, etc.\n\nIn the mu-search configuration the Elasticsearch mappings can be passed via the `mappings` property per index type specification.\n```javascript\n{\n  \"types\": [\n    {\n      \"type\": \"document\",\n      \"on_path\": \"documents\",\n      ...\n      \"mappings\" : {\n        \"properties\": {\n          \"title\" : { \"type\" : \"text\" },\n          \"description\" : { \"type\" : \"text\" }\n        }\n      }\n    },\n    // other type definitions\n  ]\n}\n```\n\nThe content of the `mappings` object is not elaborated here but can be found in the official [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html). All settings provided in `mappings.properties` in the Elasticsearch configuration can be used verbatim in the `es_settings` of mu-search.\n\n### Index options\nIn the base scenario, indexes are created on an as-needed basis, whenever a new search profile (authorization rights and data type) is received. The first search query for a new search profile may therefore take more time to complete, because the index still needs to be built. Indexes can be manually re-indexed by triggering the `POST /:type/index` endpoint (see [below](#api)).\n\n#### Index metadata in the triple store\nWhen an index is created, it is registered in the triplestore in the `\u003chttp://mu.semte.ch/authorization\u003e` graph.\n\n[To be completed... describe used model in the triplestore]\n\n#### Persistent indexes\nBy default, on startup or restart of mu-search, all existing indexes are deleted, since data might have changed in the meantime. However, for sure in production environments, regenerating indexes might be a costly operation.\n\nPersistence of indexes can be enabled via the `persist_indexes` flag at the root of the mu-search configuration file:\n```javascript\n{\n  \"persist_indexes\": true,\n  \"types\": [\n    // index type specifications\n  ]\n}\n```\n\nPossible values are `true` and `false`. Defaults to `false`.\n\nNote that if set to `true`, the indexes may be out-of-date if data has changed in the application while mu-search was down.\n\n#### Eager indexes\nConfigure indexes to be pre-built when the application starts. For each user search profile for which the indexes needs to be prepared, the authorization group names and their corresponding variables needs to be passed.\n\n```javascript\n{\n  \"eager_indexing_groups\": [\n    [\n      { \"variables\": [\"company-x\"], \"name\": \"organization-read\" },\n      { \"variables\": [\"company-x\"], \"name\": \"organization-write\" },\n      { \"variables\": [], \"name\": \"public\" }\n    ],\n    [\n      { \"variables\": [\"company-y\"], \"name\": \"organization-read\" },\n      { \"variables\": [], \"name\": \"public\" }\n    ],\n    [\n      { \"variables\": [], \"name\": \"clean\" }\n    ]\n  ],\n  \"types\": [\n    // index type specifications\n  ]\n}\n```\n\nNote that if you want to prepare indexes for all user profiles in your application, you will have to provide an entry in the `eager_indexing_groups` list for **each** possible variable value. For example, if you have an authorization group defining a user can only access the data of his company (hence, the company name is a variable of the authorization group), you will need to define an eager index group for each of the possible companies in your application.\n\n#### Additive index access rights\nAdditive indexes are indexes that may be combined to respond to a search query in order to fully match the user's authorization groups. If a user is granted access to multiple groups, indexes will be combined to calculate the response. Therefore, it's strongly adviced the indexes contain non-overlapping data. Otherwise the result set may contain duplicates (see also: [removing duplicate results](https://github.com/mu-semtech/mu-search?tab=readme-ov-file#removing-duplicate-results)).\n\nOnly indexes that are defined in the `eager_indexing_groups` will be used in combinations. If no combination can be found that fully matches the user's authorization group a single index will be created for the request's authorization groups.\n\nIf data that is needed to build documents of a search index is stored across different authorization groups (e.g. public and an organization specific group), these groups need to be specified together in an eager group and not seperately. Otherwise the search index will only contain 'partial' documents.\n\nAssume your application contains a company-specific user group in the authorization configuration; 2 companies: company X and company Y; and mu-search contains one search index definition for documents. A search index will be generated for documents of company X and another index will be generated for documents of company Y. If a user is granted access to documents of company X as well as for documents of company Y, a search query performed by this user will be effectuated by combining both search indexes.\n\nA typical group to be specified as a single `eager_indexing_group` is `{ \"variables\": [], \"name\": \"clean\" }`. The index will not contain any data, but will be used in the combination to fully match the user's allowed groups.\n\n#### [Experimental] Ignoring allowed groups\nIn some cases you may search to ignore certain allowed groups when looking for matching indexes. Typically because they will not relate to data that has to be indexed and you want to avoid having many empty indexes. In this case you will have to provide an entry in the `ignored_allowed_groups` list for each group, currently this means including each possible variable value.\nFor example the clean group can be added to `ignored_allowed_groups` by adding `{ \"variables\": [], \"name\": \"clean\" }` to the list.\n\n#### [Experimental] Dynamic allowed group variables\nIn some cases you may encounter variables which are not known up front.  The `\"variables\"` array accepts a `\"*\"` to indicate a wildcard for an attribute.  This is currently supported in `ignored_allowed_groups`.  In `eager_indexing_groups` this is supported, but only if the `eager_indexing_group` array contains a single group.  Within `eager_indexing_groups` this allows us to create a dynamic index for an access right whilst still indicating this index does not impact other indexes.  For example, you may want to index the user's message history (`[{ \"name\": \"user\", \"variables\": [\"*\"] }]` which does not impact the index of the code-lists in public `[{ \"name\": \"public\", \"variables\": [] }].` An example for ignored groups may be to ignore all of the anonymous sessions' information which could be done as: `ignored_allowed_groups\": [ { \"name\": \"anonymous-session\", \"variables\": [\"*\"] } ]`.\n\n### Delta integration\nMu-search integrates with the delta's generated by [mu-authorization](https://github.com/mu-semtech/mu-authorization) and dispatched by the [delta-notifier](https://github.com/mu-semtech/delta-notifier).\n\nFollow the \"How to integrate mu-seach with delta's to update search indexes\" guide to setup delta notification handling for mu-search. Deltas are expected in the [v0.0.1 format](https://github.com/mu-semtech/delta-notifier/#v001) of the delta notifier.\n\n#### Full index invalidation\nBy default, when a delta notification is received by mu-search, all indexes containing data related to the changes are invalidated. The index will be rebuilt the next time it is searched.\n\nNote that a change on one resource may trigger the invalidation of multiple indexes depending on the authorization groups.\n\n#### Partial index updates\nAlternate to full index invalidation, indexes can be dynamically updated on a per-document basis according to received deltas. When a delta is received, the document corresponding to the delta is updated (or deleted) in every index corresponding to the delta. This update is not a blocking operation: an update will not lock the index, so that a simultaneously received search request might be run on the un-updated index.\n\nNote that a change on one resource may trigger the update of multiple indexes depending on the authorization groups.\n\nPartial index updates are enabled by setting the `automatic_index_updates` flag at the root of the search configuation:\n\n```javascript\n{\n  \"automatic_index_updates\": true,\n  \"types\": [\n     // definition of the indexed types\n  ]\n}\n```\n\n#### Update batching and queueing\nWhen a delta notification is handled, the update to be performed is pushed on the update queue. By default the queue is processed every minute. This timeout can be configured via `update_wait_interval_minutes` in the root of the search configuration:\n\n```javascript\n{\n  \"automatic_index_updates\": true,\n  \"update_wait_interval_minutes\": 8,\n  \"types\": [\n     // definition of the indexed types\n  ]\n}\n```\n\nIncreasing the interval has the advantage that updates on the same document will be applied only once, but has the downside that search results will not be up-to-date for a longer time. The optimal value depends on the application (number of updates, indexed properties, user expectations, etc.)\n\n### API\nThis section describes the REST API provided by mu-search.\n\nIn order to take access rights into account, each request requires the `MU_AUTH_ALLOWED_GROUPS` and `MU_AUTH_USED_GROUPS` headers to be present.\n\n#### GET `/:type/search`\nEndpoint to search the given `:type` index. The request format is JSON-API compliant and intended to match the request format of [mu-cl-resources](https://github.com/mu-semtech/mu-cl-resources). Search filters are passed using query params.\n\nA subset of the [Elasticsearch Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html) is supported, via the `filter`, `page`, and `sort` query parameters. More complex queries should be sent via `POST /:type/search` endpoint.\n\n##### Examples\nTo search for `document`s on all fields:\n\n```\nGET /documents/search?filter[_all]=fish\n```\n\nTo search for `document`s on the field `name`:\n\n```\nGET /documents/search?filter[name]=fish\n```\n\nTo search for `document`s on multiple fields, combined with 'OR':\n\n```\nGET /documents/search?filter[name,description]=fish\n```\n\nTo search for `document`s by their URI:\n\n```\nGET /documents/search?filter[:uri:]=http://data.semte.ch/documents/c020b82b-61f6-4264-93c5-aba0d09812d3\n```\n\n##### Searching in a file property\nTo search for a field indexing a file, a specific property of the resulting attachment object must be specified as filter key using the `.`-notation.\n\nCurrently the following properties are available on an attachment object:\n- `content` : text content of the file\n\nFor example, for a property `attachment` indexing a file, searching the content of the file is done using the following filter query:\n```\nGET /documents/search?filter\\[attachment.content\\]=Adobe\"\n```\n\n##### Supported search methods\nMore advanced search options, such as term, range and fuzzy searches, are supported via flags. Flags are expressed in the filter key between `:` before the field name(s). E.g. the `term` search flag looks as follows:\n\n```\nGET /documents/search?filter[:term:tag]=fish\n```\n\nThe following sections list the flags that are currently implemented:\n\n###### Identifier queries\n- `:id:` Filter documents by their uuid. Multiple values should be comma-seperated, such as `filter[:id:]=c9e0fe90-3785-4221-9c4b-bda70bd8d83b,e8cbc03a-97e0-4b97-931b-97caa720db14`\n- `:uri:` Filter documents by their URI. Multiple values should be comma-seperated.\n\n###### Term-level queries\n- `:term:` : [Term query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html)\n- `:terms:` : [Terms query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-query.html), terms should be comma-separated, such as: `filter[:terms:tag]=fish,seafood`\n- `:prefix:` : [Prefix query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-prefix-query.html)\n- `:wildcard:` : [Wildcard query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-wildcard-query.html)\n- `:regexp:` : [Regexp query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html)\n- `:fuzzy:` : [Fuzzy query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html) with [fuziness](https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#fuzziness) set to `\"AUTO\"` and allowing to match multiple fields.\n- `:gt:`,`lt:`, `:gte:`, `:lte:` : [Range query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html)\n- `:lt,gt:`, `:lte,gte:`, `:lt,gte:`, `:lte,gt:` : Combined [range query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html), range limits should be comma-separated such as: `GET /documents/search?filter[:lte,gte:importance]=3,7`\n- `:has:`: Filter on documents having any value for the supplied field. To enable the filter, it's value must be `t`. E.g. `filter[:has:translation]=t`.\n- `:has-no:`: Filter on documents not having a value for the supplied field. To enable the filter, it's value must be `t`. E.g. `filter[:has-no:translation]=t`.\n\n###### Full text queries\n- `:phrase:` : [Match phrase query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase.html)\n- `:phrase_prefix:` : [Match phrase prefix query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase-prefix.html)\n- `:query:` : [Query string query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html)\n- `:sqs:` : [Simple query string query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html)\n- `:common:` [Common terms query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-common-terms-query.html). The flag takes additional options `cutoff_frequency` and `minimum_should_match` appended with commas such as `:common,{cutoff_frequence},{minimum_should_match}:{field}`. The `cutoff_frequency` can also be set application-wide in [the configuration file](#configuration-options). The common terms query was deprecated and removed from elasticsearch. It is replaced by its recommended replacement, the [match query](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-match-query)\n- `:match` [Match query](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-match-query). The flag takes additional options `cutoff_frequency` and `minimum_should_match` appended with commas such as `:common,{cutoff_frequence},{minimum_should_match}:{field}`. The `cutoff_frequency` can also be set application-wide in [the configuration file](#configuration-options).\n\n###### Custom queries\n- `:fuzzy_phrase:` : A fuzzy phrase query based on [span_near](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-span-near-query.html) and [span_multi](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-span-multi-term-query.html). See also [this](https://stackoverflow.com/questions/38816955/elasticsearch-fuzzy-phrases) Stack Overflow issue or [the code](./framework/elastic_query_builder.rb).\n\nCurrently searching on multiple fields is only supported for the following flag:\n- `:phrase:`\n- `:phrase_prefix:`\n- `:fuzzy:`\n\nMultiple filter parameters are supported.\n\nExamples\n\n```\nGET /documents/search?filter[:common:description]=a+cat+named+Barney\n\nGET /documents/search?filter[:common,0.002:description]=a+cat+named+Barney\n\nGET /documents/search?filter[:common,0.002,2:description]=a+cat+named+Barney\n\nGET /documents/search?filter[:sqs:name]=Barney\u0026[:has:address]=t\n```\n\n##### Sorting\nSorting is specified using the `sort` query parameter, providing the field to sort on and the sort direction (`asc` or `desc`). Multiple sort query parameters may be provided.\n\n```\nGET /documents/search?filter[name]=fish\u0026sort[priority]=asc\u0026sort[budget]=desc\n```\n\nFlags can be used to specify [Elasticsearch sort modes](https://www.elastic.co/guide/en/elasticsearch/reference/current/sort-search-results.html#_sort_mode_option) to sort on multi-valued fields. The following sort mode flags are supported: `:min:`, `:max:`, `:sum:`, `:avg:`, `:median:`.\n\n```\nGET /documents/search?filter[name]=fish\u0026sort[:avg:score]=asc\n```\n\nNote that sorting cannot be done on text fields, unless fielddata is enabled (not recommended). Keyword and numerical data types (declared in the [type mapping](#elasticsearch-mappings)) are recommended for sorting.\n\n##### Pagination\n\nPagination is specified using the `page[number]` and `page[size]` query parameters:\n\n```\nGET /documents/search?filter[name]=fish\u0026page[number]=2\u0026page[size]=20\n```\n\nThe page number is zero-based.\n\nBy default the search endpoint doesn't return exact result counts if the result set contains more than 10K items. To enable exact counts pass `count=exact` as query param (at the cost of some performance).\n\n##### Highlighting\n\nHighlighting is specified using the `highlight[:fields:]` query parameter, where a comma separated list of fields you want highlighted should be provided.\nYou can use `*` as field name to highlight all fields.\n\nNo settings are currently supported.\n\nSee also \u003chttps://www.elastic.co/guide/en/elasticsearch/reference/current/highlighting.html\u003e.\n\n```\nGET /documents/search?filter[:sqs:]=fish\u0026highlight[:fields:]=name,description\nGET /documents/search?filter[:sqs:]=fish\u0026highlight[:fields:]=*\n```\n\n##### Removing duplicate results\nWhen querying multiple indexes (with [additive indexes](#additive-indexes)), identical documents may be returned multiple times. Unique results can be assured using [Elasticsearch's search result collapsing](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/collapse-search-results.html) on the `uuid` field. The search result collapsing can be toggled using the `collapse_uuids` query parameter:\n\n```\nGET /documents/search?filter[name]=fish\u0026collapse_uuids=t\n```\n\nHowever, note that `count` property in the response still designates total non-unique results.\n\n#### [Experimental] POST `/:type/search`\nAccepts a raw [Elasticsearch Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html) as request body to search the given `:type` index.\n\nThis endpoint is mainly intended for testing purposes and sending more complex queries than can be expressed with the `GET /:type/search` endpoint.\n\nFor security reasons, the endpoint is disabled by default. It can be enabled by setting the `enable_raw_dsl_endpoint` flag in the root of the configuration file:\n```javascript\n{\n  \"enable_raw_dsl_endpoint\": true,\n  \"types\": [\n     // definition of the indexed types\n  ]\n}\n```\n\n#### Admin endpoints\nThe admin endpoints can be used to manage the indexes. These endpoints should not be publicly exposed in your application, since they allow 'root' access when no authorization headers are specified on the request.\n\n##### POST `/:type/index`\nUpdates the index(es) for the given `:type`. If the request is sent with authorization headers, only the authorized indexes are updated. Otherwise, all indexes for the type are updated.\n\nType `_all` will update all indexes.\n\n##### POST `/:type/invalidate`\nInvalidates the index(es) for the given `:type`. If the request is sent with authorization headers, only the authorized indexes are invalidated. Otherwise, all indexes for the type are invalidated.\n\nType `_all` will invalidate all indexes.\n\nAn invalidated index will be updated before executing a new search query on it.\n\nNote that the search index is only marked as invalid in memory. I.e the index is not removed from Elasticsearch nor the triplestore. Hence, on restart of mu-search, the index will be considered valid again.\n\n##### DELETE `/:type`\nDeletes the index(es) for the given `:type` in Elasticsearch and the triplestore. If the request is sent with authorization headers, only the authorized indexes are deleted. Otherwise, all indexes for the type are deleted.\n\nType `_all` will delete all indexes.\n\nA deleted index will be recreated before executing a new search query on it.\n\n##### POST `/update`\nProcesses an update of the delta-notifier. See [delta integration](#delta-integration).\n\nCurrently only delta format v.0.0.1 is supported.\n\n### Configuration options\nThis section gives an overview of all configurable options in the search configuration file `config.json`. Most options are explained in more depth in other sections.\n\n- (*) **persist_indexes** : flag to enable the persistence of search indexes on startup. Defaults to `false`. See [persist indexes](#persist-indexes).\n- (*) **automatic_index_updates** : flag to apply automatic index updates instead of invalidating indexes on receiving deltas. Defaults to `false`. See [delta integration](#delta-integration).\n- **eager_indexing_groups** : list of user search profiles (list of authorization groups) to be indexed at startup. Defaults to `[]`. See [eager indexes](#eager-indexes).\n- (*) **batch_size** : number of documents loaded from the RDF store and indexed together in a single batch. Defaults to 100.\n- (*) **max_batches** : maximum number of batches to index. May result in an incomplete index and should therefore only be used during development. Defaults to 1.\n- (*) **number_of_threads** : number of threads to use during indexing. Defaults to 1.\n- (*) **connection_pool_size** : number of connections in the SPARQL/Elasticsearch/Tika connection pools. Defaults to 20. Typically increased up to 200 on systems with heavy load.\n- (*) **update_wait_interval_minutes** : number of minutes to wait before applying an update. Allows to prevent duplicate updates of the same documents. Defaults to 1.\n- (*) **common_terms_cutoff_frequency** : [REMOVED] default cutoff frequency for a [Common terms query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-common-terms-query.html). This parameter was removed by elastic search and is now ignored [supported search methods](#supported-search-methods).\n- (*) **enable_raw_dsl_endpoint** : flag to enable the [raw Elasticsearch DSL endpoint](#api). This endpoint is disabled by default for security reasons.\n- (*) **attachments_path_base** : path inside the Docker container where files for the attachment pipeline are mounted. Defaults to `/data`.\n\nAll options prefixed with (*) can also be configured using an UPPERCASED variant as Docker environment variables on the mu-search container. E.g. the `batch_size` option can be set via the environment variable `BATCH_SIZE`. Environment variables take precedence over settings configured in `config.json`.\n\nIn development mode (setting the environment variable `RACK_ENV` to `development`), the application will listen for changes in `config.json`. Any change will trigger a complete reload of the full application, including deleting existing indexes, and building any default indexes specified in eager indexing. This behaviour overrules the `persist_indexes` flag.\n\n### Logging\nLog messages are logged in a specific scope. A different log level can be configured per scope via environment variables like `LOG_SCOPE_{scopeName}\u003e`.\n\nE.g.\n```yml\nsearch:\n  environment:\n     LOG_SCOPE_TIKA: \"warn\"\n     LOG_SCOPE_DELTA: \"debug\"\n```\n\nThe following scopes are known:\n* **SETUP**: system setup and initialization (default: `info`)\n* **INDEX_MGMT**: creation, updates and deletion of indexes (default: `info`)\n* **INDEXING**: indexing of documents (default: `info`)\n* **SEARCH**: execution of search queries (default: `warn`)\n* **TIKA**: extraction and indexing of file content using Tika (default: `warn`)\n* **ELASTICSEARCH**: all communication with Elasticsearch (default: `error`)\n* **SPARQL**: all communication with the database (default: `warn`)\n* **AUTHORIZATION**: incoming access rights on requests (default: `warn`)\n* **DELTA**: handling of incoming delta's (default: `info`)\n* **UPDATE_HANDLER**: processing of the updates triggered by delta's (default: `info`)\n\nThe same log levels as the [mu-ruby-template](https://github.com/mu-semtech/mu-ruby-template) are available:\n* `debug`\n* `info`\n* `warn`\n* `error`\n* `fatal`\n\n### Environment variables\nThis section gives an overview of all options that are configurable via environment variables. The options that can be configured in the `config.json` file as well are not repeated here. This list contains options that can only be configured via environment variables.\n\n- **MAX_REQUEST_URI_LENGTH** : maximum length of an incoming request URL. Defaults to 10240.\n- **MAX_REQUEST_HEADER_LENGTH** : maximum length of the headers of an incoming request. Defaults to 1024000.\n- **MAXIMUM_FILE_SIZE** : maximum size in bytes of files to extract and index content from. Defaults to 209715200.\n- **ELASTIC_READ_TIMEOUT** : timeout in seconds of requests to Elasticsearch. Defaults to 180.\n- **MAX_YAML_SIZE**: Set the code point limit for the psych yaml parser (in bytes), affects the maximum queue size that can be stored (and read). defaults to 20_000_000 bytes.\n\n## Discussions\n### Why a custom Elasticsearch docker image?\nThe [mu-semtech/search-elastic-backend](https://github.com/mu-semtech/mu-search-elastic-backend) is a custom Docker image based on the official Elasticsearch image. Providing a custom image allows better control on the version of Elasticsearch, currently v7.2.0, used in combination with the mu-search service.\n\nThe custom image also makes sure the required Elasticsearch plugins, such as the ingest-attachments plugin, are already pre-installed making the integration of mu-search in your stack a lot easier.\n\n### Authorization groups vs indexes\nAccess rights are determined according to the contents of two headers, `MU_AUTH_ALLOWED_GROUPS` and `MU_AUTH_USED_GROUPS`.\n\nCurrently, a separate Elasticsearch index is created for each combination of document type and authorization group.\n\n[To be completed...]\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmu-semtech%2Fmu-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmu-semtech%2Fmu-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmu-semtech%2Fmu-search/lists"}