{"id":42212907,"url":"https://github.com/foliant-docs/foliantcontrib.reindexer","last_synced_at":"2026-01-27T01:06:54.469Z","repository":{"id":57431758,"uuid":"288388674","full_name":"foliant-docs/foliantcontrib.reindexer","owner":"foliant-docs","description":"Foliant extension for integration with Reindexer in-memory DBMS","archived":false,"fork":false,"pushed_at":"2022-06-06T16:18:12.000Z","size":22,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"develop","last_synced_at":"2025-09-28T00:22:58.637Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/foliant-docs.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-18T07:41:35.000Z","updated_at":"2022-06-06T16:18:15.000Z","dependencies_parsed_at":"2022-09-18T05:36:04.688Z","dependency_job_id":null,"html_url":"https://github.com/foliant-docs/foliantcontrib.reindexer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/foliant-docs/foliantcontrib.reindexer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foliant-docs%2Ffoliantcontrib.reindexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foliant-docs%2Ffoliantcontrib.reindexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foliant-docs%2Ffoliantcontrib.reindexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foliant-docs%2Ffoliantcontrib.reindexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/foliant-docs","download_url":"https://codeload.github.com/foliant-docs/foliantcontrib.reindexer/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/foliant-docs%2Ffoliantcontrib.reindexer/sbom","scorecard":{"id":406096,"data":{"date":"2025-08-11","repo":{"name":"github.com/foliant-docs/foliantcontrib.reindexer","commit":"9150fbf3963955f052fce8089a9c2d6c6f910e95"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/12 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'develop'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-18T21:17:52.687Z","repository_id":57431758,"created_at":"2025-08-18T21:17:52.687Z","updated_at":"2025-08-18T21:17:52.687Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28794642,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T21:49:50.245Z","status":"ssl_error","status_checked_at":"2026-01-26T21:48:29.455Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-27T01:06:53.769Z","updated_at":"2026-01-27T01:06:54.443Z","avatar_url":"https://github.com/foliant-docs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![](https://img.shields.io/pypi/v/foliantcontrib.reindexer.svg)](https://pypi.org/project/foliantcontrib.reindexer/) [![](https://img.shields.io/github/v/tag/foliant-docs/foliantcontrib.reindexer.svg?label=GitHub)](https://github.com/foliant-docs/foliantcontrib.reindexer)\n\n# Reindexer Integration Extension\n\nThis extension allows to integrate Foliant-managed documentation projects with the in-memory DBMS [Reindexer](https://github.com/Restream/reindexer/) to use it as a fulltext search engine.\n\nThe main part of this extension is a preprocessor that prepares data for a search index. In addition, the preprocessor performs basic manipulations with the database and the namespace in it.\n\nAlso this extension provides a simple working example of a client-side Web application that may be used to perform searching. By editing HTML, CSS and JS code you may customize it according to your needs.\n\n## Installation\n\nTo install the preprocessor, run the command:\n\n```bash\n$ pip install foliantcontrib.reindexer\n```\n\nTo use an example of a client-side Web application for searching, download [these HTML, CSS, and JS files](https://github.com/foliant-docs/foliantcontrib.reindexer/tree/master/webapp_example/) and open the file `index.html` in your Web browser.\n\n## Config\n\nTo enable the preprocessor, add `reindexer` to `preprocessors` section in the project config:\n\n```yaml\npreprocessors:\n    - reindexer\n```\n\nThe preprocessor has a number of options with the following default values:\n\n```yaml\npreprocessors:\n    - reindexer:\n        reindexer_url: http://127.0.0.1:9088/\n        insert_max_bytes: 0\n        database: ''\n        namespace: ''\n        namespace_renamed: ''\n        fulltext_config: {}\n        actions:\n            - drop_database\n            - create_database\n            - create_namespace\n            - insert_items\n        use_chapters: true\n        format: plaintext\n        escape_html: true\n        url_transform:\n            - '\\/?index\\.md$': '/'\n            - '\\.md$': '/'\n            - '^([^\\/]+)': '/\\g\u003c1\u003e'\n        require_env: false\n        targets: []\n```\n\n`reindexer_url`\n:   URL of your Reindexer instance. “Root” server URL should be used here, do not add any endpoints such as `/api/v1/db` to it.\n\n`insert_max_bytes`\n:   Reindexer itself or a proxy server may limit the available size of request body. Use this option, if it’s needed to split a large amount of content for indexing into several chunks, so each of them will be sent in a separate request. The value of this option represents maximum size of HTTP POST request body in bytes. Allowed values are positive integers starting from `1024`, and `0` (default) meaning no limits.\n\n`database`\n:   Name of the database that is used to store your search index.\n\n`namespace`\n:   Name of the namespace in the specified database. Namespace in Reindexer means the same as table in relational databases. To store the search index for one documentation project, single namespace is enough.\n\n`namespace_renamed`\n:   New namespace name to be applied if the `rename` option is used; see below.\n\n`fulltext_config`\n:   The value of the `config` field that refers to the description of the composite fulltext index over the `title` and `content` data fields. Used data structure is described below. [Fulltext indexes config options](https://github.com/Restream/reindexer/blob/master/cpp_src/server/contrib/server.md#fulltextconfig) are listed in the Reindexer’s official documentation.\n\n`actions`\n:   Sequence of actions that the preprocessor should to perform. Available item values are:\n\n* `drop_database` — fully remove the database that is specified as the value of the `database` option. Please be careful using this action when the single database is used to store multiple namespaces. Since this action is included to the default actions list, it’s recommended to use separate databases for each search index. The default list of actions assumes that in most cases it’s needed to remove and then fully rebuild the index, and wherein the database and the namespace may not exist;\n* `create_database` — create the new database with the name specified as the `database` option value;\n* `drop_namespace` — delete the namespace that is specified as the `namespace` option value. All `*_namespace` actions are applied to the existing database with the name from the `database` option;\n* `truncate_namespace` — remove all items from the namespace that is specified as the `namespace` option value, but keep the namespace itself;\n* `rename_namespace` — rename the existing namespace that has the name specified as the `namespace` option value, to the new name from the `renamed_namespace` option. This action may be useful when a common search index is created for multiple Foliant projects, and the index may remain incomplete for a long time during their building;\n* `create_namespace` — create the new namespace with the name from the `namespace` option;\n* `insert_items` — fill the namespace that is specified in the `namespace` option, with the content that should be indexed. Each data item added to the namespace corresponds a single Markdown file of the documentation project.\n\n`use_chapters`\n:   If set to `true` (by default), the preprocessor applies only to the files that are mentioned in the `chapters` section of the project config. Otherwise, the preprocessor applies to all Markdown files of the project.\n\n`format`\n:   Format that the source Markdown content should be converted to before adding to the index; available values are: `plaintext` (by default), `html`, `markdown` (for no conversion).\n\n`escape_html`\n:   If set to `true` (by default), HTML syntax constructions in the content converted to `plaintext` will be escaped by replacing `\u0026` with `\u0026amp;`, `\u003c` with `\u0026lt;`, `\u003e` with `\u0026gt;`, and `\"` with `\u0026quot;`.\n\n`url_transform`\n:   Sequence of rules to transform local paths of source Markdown files into URLs of target pages. Each rule should be a dictionary. Its data is passed to the [`re.sub()` method](https://docs.python.org/3/library/re.html#re.sub): key as the `pattern` argument, and value as the `repl` argument. The local path (possibly previously transformed) to the source Markdown file relative to the temporary working directory is passed as the `string` argument. The default value of the `url_transform` option is designed to be used to build static websites with MkDocs backend.\n\n`require_env`\n:   If set to `true`, the `FOLIANT_REINDEXER` environment variable must be set to allow the preprocessor to perform any operations with the database and the namespace managed by Reindexer. This flag may be useful in CI/CD jobs.\n\n`targets`\n:   Allowed targets for the preprocessor. If not specified (by default), the preprocessor applies to all targets.\n\n## Usage\n\nThe preprocessor reads each source Markdown file and prepares three fields for indexing:\n\n* `url`—target page URL. This field is used as the primary key, so it must be unique;\n* `title`—document title, it’s taken from the first heading of source Markdown content;\n* `content`—source Markdown content, optionally converted into plain text or HTML.\n\nWhen all the files are processed, the preprocessor calls Reindexer API to insert data items (each item corresponds a single Markdown file) into the specified namespace.\n\nAlso the preprocessor may call Reindexer API to manipulate the database or namespace, e.g. to delete previously created search index.\n\nYou may perform custom search requests to Reindexer API.\n\nThe [simple client-side Web application example](https://github.com/foliant-docs/foliantcontrib.reindexer/tree/master/webapp_example/) that is provided as a part of this extension, sends to Reindexer queries like this:\n\n```json\n{\n    \"namespace\": \"testing\",\n    \"filters\": [\n        {\n            \"field\": \"indexed_content\",\n            \"cond\": \"EQ\",\n            \"value\": \"@title^3,content^1 foliant\"\n        }\n    ],\n    \"select_functions\": [\n        \"content = snippet(\u003cem\u003e,\u003c/em\u003e,100,100,'\\n\\n')\"\n    ],\n    \"limit\": 50\n}\n\n```\n\nTo learn how to write efficient queries to Reindexer, you may need to refer to its official documentation on topics: [general use](https://github.com/Restream/reindexer/blob/master/readme.md), [fulltext search](https://github.com/Restream/reindexer/blob/master/fulltext.md), [HTTP REST API](https://github.com/Restream/reindexer/blob/master/cpp_src/server/contrib/server.md).\n\nIn the example above, the `indexed_content` field corresponds to the composite index over two fields: `title` and `content` (this index is generated when the namespace is created by the request from the preprocessor). [Text of the search query](https://github.com/Restream/reindexer/blob/master/fulltext.md#text-query-format) starts with `@title^3,content^1` that means that the `title` field of the composite index has triple priority (i.e. weighting factor of 3), and the `content` field has normal priority (i.e. weight coefficient equals to 1). Also the example uses the `snippet()` [select function](https://github.com/Restream/reindexer/blob/master/fulltext.md#using-select-fucntions) to highlight the text that matches the query and to cut off excess.\n\nIf you use self-hosted instance of Reindexer, you may need to configure a proxy to append [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) headers to HTTP API responses.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffoliant-docs%2Ffoliantcontrib.reindexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffoliant-docs%2Ffoliantcontrib.reindexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffoliant-docs%2Ffoliantcontrib.reindexer/lists"}