{"id":14981048,"url":"https://github.com/abdullahkhawer/simple-elasticsearch-cleaner","last_synced_at":"2026-02-07T13:02:45.744Z","repository":{"id":252266269,"uuid":"839729803","full_name":"abdullahkhawer/simple-elasticsearch-cleaner","owner":"abdullahkhawer","description":"A simple Elasticsearch cleaner based on a Python script to clean up all indices and data streams from the Elasticsearch cluster which are older than the specified number of days.","archived":false,"fork":false,"pushed_at":"2024-12-13T20:13:03.000Z","size":51,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-29T04:37:23.385Z","etag":null,"topics":["cleaner","curation","curator","data-cleaner","data-cleaning","docker","docker-image","dockerfile","elasticsearch","elasticsearch-cleaner","elasticsearch-client","elasticsearch-cluster","elasticsearch-curator","helm","helm-chart","helm3","python","python3","terraform","terraform-module"],"latest_commit_sha":null,"homepage":"https://hub.docker.com/r/abdullahkhawer/simple-elasticsearch-cleaner","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abdullahkhawer.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-08-08T08:01:28.000Z","updated_at":"2024-12-13T20:04:13.000Z","dependencies_parsed_at":"2025-10-29T04:31:45.928Z","dependency_job_id":"a0b6d1e4-3770-4fd0-af6f-c74236be7d41","html_url":"https://github.com/abdullahkhawer/simple-elasticsearch-cleaner","commit_stats":{"total_commits":11,"total_committers":2,"mean_commits":5.5,"dds":0.09090909090909094,"last_synced_commit":"0721467c44e0842d29177cd1d0939a6f8d740d3d"},"previous_names":["abdullahkhawer/simple-elasticsearch-cleaner"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/abdullahkhawer/simple-elasticsearch-cleaner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdullahkhawer%2Fsimple-elasticsearch-cleaner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdullahkhawer%2Fsimple-elasticsearch-cleaner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdullahkhawer%2Fsimple-elasticsearch-cleaner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdullahkhawer%2Fsimple-elasticsearch-cleaner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abdullahkhawer","download_url":"https://codeload.github.com/abdullahkhawer/simple-elasticsearch-cleaner/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdullahkhawer%2Fsimple-elasticsearch-cleaner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29194426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-07T12:38:28.597Z","status":"ssl_error","status_checked_at":"2026-02-07T12:38:23.888Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cleaner","curation","curator","data-cleaner","data-cleaning","docker","docker-image","dockerfile","elasticsearch","elasticsearch-cleaner","elasticsearch-client","elasticsearch-cluster","elasticsearch-curator","helm","helm-chart","helm3","python","python3","terraform","terraform-module"],"created_at":"2024-09-24T14:02:47.649Z","updated_at":"2026-02-07T13:02:45.718Z","avatar_url":"https://github.com/abdullahkhawer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Simple Elasticsearch Cleaner\n\n- Founder: [Abdullah Khawer - LinkedIn](https://www.linkedin.com/in/abdullah-khawer)\n\n# Introduction\n\nA Simple Elasticsearch Cleaner based on a Python script to delete all indices and data streams from the Elasticsearch cluster which are older than the specified number of days. It is basically a curator but with the ability to delete only, removing both indices and their associated data streams.\n\nBelow you can find an example of the execution:\n```\nsimple-elasticsearch-cleaner % python simple_elasticsearch_cleaner.py 30\nScript Execution Started!\nCalculating the date that was 30 day(s) ago...\nConnecting to Elasticsearch...\nConnected to Elasticsearch successfully.\nFetching the list of all the data streams...\nFetching the list of all the indices...\nFinding and deleting all the indices older than 30 days '2024.07.14'...\nDeleted index '__REDACTED__' which was created on '2024.07.12'.\nDeleted index '__REDACTED__' which was created on '2024.07.13'.\nDeleted index '__REDACTED__' which was created on '2024.07.12'.\nDeleted index '__REDACTED__' which was created on '2024.07.13'.\nFinding and deleting all the data streams older than 30 days '2024.07.14'...\nSkipping data stream '__REDACTED__' as it is a managed data stream.\nSkipping data stream '__REDACTED__' as it is a managed data stream.\nDeleted data stream '__REDACTED__' which was older than '2024.07.14'.\nDeleted data stream '__REDACTED__' which was older than '2024.07.14'.\nSkipping data stream '__REDACTED__' as it is a managed data stream.\nClosing the connection with Elasticsearch...\nConnection with Elasticsearch closed successfully.\nScript Execution Completed!\n```\n\n*Note: In the actual execution, you will see the actual values instead of `__REDACTED__` values.*\n\n# Usage Notes\n\nFollowing are the 4 ways to use it:\n- Python script.\n- Docker container running that Python script.\n- Helm chart to create a Helm release to create a cron job running that Docker container in a pod on a Kubernetes cluster.\n- Terraform module to deploy that Helm chart as a Helm release on a Kubernetes cluster.\n\n## Terraform Module\n\nYou can use its [Terraform module](https://github.com/abdullahkhawer/simple-elasticsearch-cleaner/blob/master/terraform) to create a Helm release using its Helm chart to create a cron job running that Docker container in a pod on a Kubernetes cluster.\n\nFor more details, check out its [README](https://github.com/abdullahkhawer/simple-elasticsearch-cleaner/blob/master/terraform/README.md).\n\n## Helm Chart\n\nYou can use its [Helm chart](https://github.com/abdullahkhawer/simple-elasticsearch-cleaner/blob/master/helm-charts/simple-elasticsearch-cleaner) to create a Helm release to create a cron job running that Docker container in a pod on a Kubernetes cluster.\n\nIts repository on [Artifact Hub](https://artifacthub.io) can be found [here](https://artifacthub.io/packages/helm/simple-elasticsearch-cleaner/simple-elasticsearch-cleaner).\n\nFor more details, check out its [README](https://github.com/abdullahkhawer/simple-elasticsearch-cleaner/blob/master/helm-charts/simple-elasticsearch-cleaner/README.md).\n\n## Docker Image\n\nYou can use its [Docker image](https://hub.docker.com/r/abdullahkhawer/simple-elasticsearch-cleaner/) which is publicly available to run its Docker container to run its Python script.\n\nThe Dockerfile used to build the image can be found [here](https://github.com/abdullahkhawer/simple-elasticsearch-cleaner/blob/master/docker).\n\nFor more details, check out its [README](https://github.com/abdullahkhawer/simple-elasticsearch-cleaner/blob/master/docker/README.md).\n\n## Python Script\n\nYou can use its [Python script](https://github.com/abdullahkhawer/simple-elasticsearch-cleaner/blob/master/simple_elasticsearch_cleaner.py) to run it directly which is the main script to be executed in order to clean the old indices and data streams.\n\n### Prerequisites\n\nFollowing are the prerequisites to be met once before you begin:\n\n- Following packages should be installed on your system:\n   - `pip`\n   - `python`\n   - Using `pip`, install the following via `requirements.txt` file:\n      - `urllib3`\n      - `elasticsearch`\n\n### Execution Instructions\n\nOnce all the prerequisites are met, set the following environment variables:\n   - `ELASTICSEARCH_HOST`\n      - Description: Host of Elasticsearch cluster.\n      - Example: `https://localhost`\n      - Requirement: REQUIRED\n   - `ELASTICSEARCH_PORT`\n      - Description: Port of Elasticsearch cluster.\n      - Example: `9200`\n      - Requirement: REQUIRED\n   - `ELASTICSEARCH_USER`\n      - Description: Username of the user of the Elasticsearch cluster.\n      - Example: `admin`\n      - Requirement: REQUIRED\n   - `ELASTICSEARCH_PASSWORD`\n      - Description: Password of the user of the Elasticsearch cluster.\n      - Example: `123456789`\n      - Requirement: REQUIRED\n   - `NUMBER_OF_DAYS`\n      - Description: Number of days to delete the indices and data streams older than that.\n      - Example: `30`\n      - Requirement: REQUIRED\n\nAnd then simply run the following command:\n- `python simple_elasticsearch_cleaner.py`\n\n# License\n\nThis project is licensed under the Apache License - see the [LICENSE](https://github.com/abdullahkhawer/simple-elasticsearch-cleaner/blob/master/LICENSE) file for details. This LICENSE file applies to all the code in this repository including `docker`, `helm-charts` and `terraform` subdirectories.\n\n#### Any contributions, improvements and suggestions will be highly appreciated. 😊\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabdullahkhawer%2Fsimple-elasticsearch-cleaner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabdullahkhawer%2Fsimple-elasticsearch-cleaner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabdullahkhawer%2Fsimple-elasticsearch-cleaner/lists"}