{"id":19866458,"url":"https://github.com/devopsmakers/kibana-index-pattern-creator","last_synced_at":"2025-05-02T06:30:34.815Z","repository":{"id":95879464,"uuid":"323887042","full_name":"devopsmakers/kibana-index-pattern-creator","owner":"devopsmakers","description":"A python script and Docker image for creating index patterns and refreshing fields.","archived":false,"fork":false,"pushed_at":"2020-12-23T23:12:55.000Z","size":15,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-06T23:13:19.491Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devopsmakers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-23T11:42:26.000Z","updated_at":"2023-01-24T06:39:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"f3c6a53a-755f-4b8d-b3f3-39148f977e72","html_url":"https://github.com/devopsmakers/kibana-index-pattern-creator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devopsmakers%2Fkibana-index-pattern-creator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devopsmakers%2Fkibana-index-pattern-creator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devopsmakers%2Fkibana-index-pattern-creator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devopsmakers%2Fkibana-index-pattern-creator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devopsmakers","download_url":"https://codeload.github.com/devopsmakers/kibana-index-pattern-creator/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251998027,"owners_count":21677913,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T15:25:54.161Z","updated_at":"2025-05-02T06:30:34.809Z","avatar_url":"https://github.com/devopsmakers.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kibana-index-pattern-creator\nA python script and Docker image for creating index patterns and refreshing fields.\n\n## Background\nIf you're like me, you send your logs into elasticsearch, you realised that shipping logs into a big old `logstash-YYYY.MM.DD` bucket is probably not great and split your logs into different indexes.\n\nIndex names might be based on an applications name, Kubernetes namespace, log type, or something completely exotic. You probably found out that you should keep a common prefix like `logstash-` or `fluentd-` to make curator index management jobs easier by applying basic operations to all logs starting with the prefix.\n\nYou probably get a bit annoyed when a new namespace, app, or other index turns up and you have to go into Kibana and create a new index pattern for it, you also get annoyed at indexes field lists getting out of date too... That is the annoyance that this script will rid you of.\n\n## Tell me more...\nThe script: `src/main.py` gets a list of all indexes in Elasticsearch that start with a user specified prefix. It then creates a Kibana index pattern for each one (if it doesn't already exist). It can be run with an environment variable set to refresh the fields on **all** index patterns in Kibana.\n\n## Mmmkay, how do I do that?\nThe script is configured solely by environment variables. It's designed to be run as a `CronJob` in Kubernetes, but can be run anywhere.\n\n| Variable | Description | Default | Example |\n| -------- | ----------- | ------- | ------- |\n| `LOG_LEVEL` | Sets the log level for the script | `INFO` | `DEBUG` |\n| `KIBANA_URL` | The URL that the script can find Kibana at | `http://kibana:5601` | `http://notadmin:aPassword@kibana.dev.example.com` |\n| `ELASTICSEARCH_URL` | The URL that the script can find Elasticsearch at | `http://elasticsearch:9200` | `https://notadmin:aPassword@elasticsearch.logging.svc.cluster.local:9200` |\n| `INDEX_PREFIX` | The prefix that the script should filter Elasticsearch indexes on | `logstash-` | `fluentd-` |\n| `EXACT_MATCHES` | Set this to create exact (not wildcard) index patterns | `False` | `yes` |\n| `LAST_CHARACTER` | The last character that should come before the `*` | `-` | `_` |\n| `REFRESH_FIELDS` | Whether to refresh the field lists of all index patterns in Kibana | `False` | `yes` |\n| `DRY_RUN` | Set this to prevent creating or updating index patterns | `False` | `yes` |\n\n### Running the script locally\nIf your Elasticsearch and Kibana are in Kubernetes, you can run this script locally using the something like telepresence.io and the service URLs within the cluster.\n\n```bash\n# Setup a venv\npython3 -m venv venv\nsource venv/bin/activate\n\n# Install dependencies\npip install -r requirements.txt\n\n# Run the script\nLOG_LEVEL=DEBUG \\\nKIBANA_URL=http://localhost:5601 \\\nELASTICSEARCH_URL=http://localhost:9200 \\\n# Other settings\npython3 src/main.py\n\n# To refresh field lists on all index patterns in Kibana\nLOG_LEVEL=DEBUG \\\nKIBANA_URL=http://localhost:5601 \\\nELASTICSEARCH_URL=http://localhost:9200 \\\nREFRESH_FIELDS=yes \\\n# Other settings\npython3 src/main.py\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevopsmakers%2Fkibana-index-pattern-creator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevopsmakers%2Fkibana-index-pattern-creator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevopsmakers%2Fkibana-index-pattern-creator/lists"}