{"id":21611715,"url":"https://github.com/ktsstudio/es2loki","last_synced_at":"2025-04-11T05:34:56.763Z","repository":{"id":86391378,"uuid":"573760909","full_name":"ktsstudio/es2loki","owner":"ktsstudio","description":"Elasticsearch to Grafana Loki logs transfer library","archived":false,"fork":false,"pushed_at":"2022-12-12T11:49:56.000Z","size":129,"stargazers_count":40,"open_issues_count":2,"forks_count":1,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-04-25T07:42:00.013Z","etag":null,"topics":["asyncio","elasticsearch","grafana","logs","loki","migration-tool","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ktsstudio.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-03T10:54:03.000Z","updated_at":"2024-04-09T23:32:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"3174ce1f-bce2-4b6a-81a8-c517ea7f7eba","html_url":"https://github.com/ktsstudio/es2loki","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktsstudio%2Fes2loki","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktsstudio%2Fes2loki/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktsstudio%2Fes2loki/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktsstudio%2Fes2loki/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ktsstudio","download_url":"https://codeload.github.com/ktsstudio/es2loki/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248348393,"owners_count":21088861,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asyncio","elasticsearch","grafana","logs","loki","migration-tool","python"],"created_at":"2024-11-24T21:13:38.986Z","updated_at":"2025-04-11T05:34:56.745Z","avatar_url":"https://github.com/ktsstudio.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# es2loki\n\n[![Build](https://github.com/ktsstudio/es2loki/actions/workflows/package.yml/badge.svg?branch=main)](https://github.com/ktsstudio/es2loki/actions)\n[![Build](https://github.com/ktsstudio/es2loki/actions/workflows/docker.yml/badge.svg?branch=main)](https://github.com/ktsstudio/es2loki/actions)\n[![PyPI](https://img.shields.io/pypi/v/es2loki.svg)](https://pypi.python.org/pypi/es2loki)\n[![Docker Image](https://img.shields.io/docker/v/ktshub/es2loki?label=docker\u0026sort=semver)](https://hub.docker.com/repository/docker/ktshub/es2loki)\n\n`es2loki` is a migration library that helps to transfer logs from\nElasticsearch to Grafana Loki.\n\nTo use es2loki currently you need to define your own mapping of elasticsearch documents\nto labels for Grafana Loki.\n\n## Demo\nYou may find helpful a [demo](demo) folder which contains a fully-sufficient demo stand\nthat demonstrates transferring logs using `es2loki`.\n\n## Usage\nIn the simplest form you don't need to write any Python code at all,\nLoki will receive no meaningful labels, but nevertheless - let's see how it works.\n\n```bash\n$ pip install -U es2loki\n$ ELASTIC_HOSTS=http://localhost:9200 \\\n  ELASTIC_INDEX=\"filebeat-*\" \\\n  LOKI_URL=http://localhost:3100 \\\n  python -m es2loki\n```\n\nIn order to override default `es2loki` behaviour you need to subclass\na `es2loki.BaseTransfer` class.\n\nTo declare how documents map to Loki labels you have to override a\n`extract_doc_labels` method (see [demo/example.py](demo/example.py)):\n\n```python\n\nfrom es2loki import BaseTransfer\n\n\nclass TransferLogs(BaseTransfer):\n    def extract_doc_labels(self, source: dict) -\u003e Optional[MutableMapping[str, str]]:\n        return dict(\n            app=source.get(\"fields\", {}).get(\"service_name\"),\n            job=\"logs\",\n            level=source.get(\"level\"),\n            node_name=source.get(\"host\", {}).get(\"name\"),\n            logger_name=source.get(\"logger_name\"),\n        )\n```\n\nYou can run this using the following code:\n```python\nimport sys\nfrom es2loki import run_transfer\n\nif __name__ == \"__main__\":\n    sys.exit(run_transfer(TransferLogs()))\n```\n\nYou can find more examples in the [demo](demo) folder.\n\n### Sorting\n\nBy default `es2loki` assumes that in the documents returned from Elasticsearch\nthere are fields `@timestamp` (you can change the name - see below) and `log.offset`.\nUsing these 2 fields we can be sure that we will not reread the same lines multiple times.\nBut if you have your fields that could guarantee such a behaviour - please\noverride a `make_es_sort` and `make_es_search_after` methods.\n\n* `make_es_sort` defines by which fields the sorting will happen.\n* `make_es_search_after` defines an initial \"offset\". It is needed to resume es2loki after a shutdown. By default it\n  extracts information from the internal state, which can be saved persistently.\n\n### Persistence\n\n`es2loki` has a mechanism to store the Elasticsearch scrolling state\nin the database (highly recommended). In this mode `es2loki` saves\nthe scrolling state inside an SQL database (PostgreSQL, MySQL, SQLite, ...).\n\nYou can opt out of enabling persistence completely using `STATE_MODE=none` env variable, which is the default.\nBut we highly recommend to enable persistence with some SQL storage.\n\n### Deployment\n\nYou can deploy `es2loki` via our helm chart.\n\nAdd `kts` repo:\n```bash\nhelm repo add kts https://charts.kts.studio\nhelm repo update\n```\n\nInstall the chart:\n```bash\nhelm upgrade --install RELEASE_NAME kts/es2loki\n```\n\nMore information about helm chart deployment can be found [here](https://github.com/ktsstudio/helm-charts/tree/main/charts/es2loki).\n\n## Configuration\n\nYou can configure `es2loki` using the following environment variables:\n\n| name                    | default                            | description                                                                                        |\n|-------------------------|------------------------------------|----------------------------------------------------------------------------------------------------|\n| ELASTIC_HOSTS           | http://localhost:9200              | Elasticsearch hosts. Separate multiple hosts using `,`                                             |\n| ELASTIC_USER            | \"\"                                 | Elasticsearch username                                                                             |\n| ELASTIC_PASSWORD        | \"\"                                 | Elasticsearch password                                                                             |\n| ELASTIC_INDEX           | \"\"                                 | Elasticsearch index pattern to search documents in                                                 |\n| ELASTIC_BATCH_SIZE      | 3000                               | How much documents to extract from ES in one batch                                                 |\n| ELASTIC_TIMEOUT         | 120                                | Elasticsearch `search` query timeout                                                               |\n| ELASTIC_MAX_DATE        |                                    | Upper date limit (format is the same as @timestamp field)                                          |\n| ELASTIC_TIMESTAMP_FIELD | @timestamp                         | Name of timesteamp field in Elasticsearch                                                          |\n| LOKI_URL                | http://localhost:3100              | Loki instance URL                                                                                  |\n| LOKI_USERNAME           | \"\"                                 | Loki username                                                                                      |\n| LOKI_PASSWORD           | \"\"                                 | Loki password                                                                                      |\n| LOKI_TENANT_ID          | \"\"                                 | Loki Tenant ID (Org ID)                                                                            |\n| LOKI_BATCH_SIZE         | 1048576                            | Maximum batch size (in bytes)                                                                      |\n| LOKI_POOL_LOAD_FACTOR   | 10                                 | Maximum number of push non-waiting requests                                                        |\n| LOKI_PUSH_MODE          | pb                                 | `pb` - protobuf + snappy, `gzip` - json + gzip, `json` - just json                                 |\n| LOKI_WAIT_TIMEOUT       | 0                                  | How much time (in seconds) to wait after a Loki push request                                       |\n| STATE_MODE              | none                               | Configures es2loki persistence (`db` is recommended). Use `none` to disable persistence completely |\n| STATE_START_OVER        |                                    | Clean up persisted data and start over                                                             |\n| STATE_DB_URL            | postgres://127.0.0.1:5432/postgres | Database URL for `db` persistence                                                                  |\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fktsstudio%2Fes2loki","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fktsstudio%2Fes2loki","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fktsstudio%2Fes2loki/lists"}