{"id":29370908,"url":"https://github.com/nuung/elk-psql-cdc-boilerplate","last_synced_at":"2025-07-09T14:11:34.358Z","repository":{"id":181573405,"uuid":"666976491","full_name":"Nuung/elk-psql-cdc-boilerplate","owner":"Nuung","description":"2023) [Boilerplate] Elasticsearch Logstash Kibana stack for query based CDC(change data capture) from postgresql  ","archived":false,"fork":false,"pushed_at":"2023-07-18T17:13:49.000Z","size":1268,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-04-19T08:36:46.511Z","etag":null,"topics":["cdc","elasticsearch","elk-stack","kibana","logstash","postgresql"],"latest_commit_sha":null,"homepage":"https://velog.io/@qlgks1/Elasticsearch-ELK-stack-Postgresql-Logstash-query-based-CDC-%EB%A7%8C%EB%93%A4%EA%B8%B0-by-docker-compose","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Nuung.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-07-16T08:11:59.000Z","updated_at":"2023-07-16T08:58:41.000Z","dependencies_parsed_at":"2023-07-16T10:12:25.033Z","dependency_job_id":null,"html_url":"https://github.com/Nuung/elk-psql-cdc-boilerplate","commit_stats":null,"previous_names":["nuung/elk-psql-cdc-boilerplate"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Nuung/elk-psql-cdc-boilerplate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nuung%2Felk-psql-cdc-boilerplate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nuung%2Felk-psql-cdc-boilerplate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nuung%2Felk-psql-cdc-boilerplate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nuung%2Felk-psql-cdc-boilerplate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Nuung","download_url":"https://codeload.github.com/Nuung/elk-psql-cdc-boilerplate/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nuung%2Felk-psql-cdc-boilerplate/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264473897,"owners_count":23613961,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc","elasticsearch","elk-stack","kibana","logstash","postgresql"],"created_at":"2025-07-09T14:11:32.625Z","updated_at":"2025-07-09T14:11:34.340Z","avatar_url":"https://github.com/Nuung.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FNuung%2Felk-psql-cdc-boilerplate\u0026count_bg=%23A3BCFF\u0026title_bg=%23555555\u0026icon=elastic.svg\u0026icon_color=%23E7E7E7\u0026title=hits\u0026edge_flat=false)](https://hits.seeyoufarm.com)\n\n# Elasticsearch Logstash Kibana stack, query based CDC\n\n\u003e 2023) [Docker Compose Boilerplate] ELK stack for CDC from postgresql\n\n## 1. Getting Started\n\n- (KR) Please check out this post: [Elasticsearch - ELK stack \u0026 Postgresql \u0026 Logstash, query based CDC 만들기 by docker compose\n](https://velog.io/@qlgks1/Elasticsearch-ELK-stack-Postgresql-Logstash-query-based-CDC-%EB%A7%8C%EB%93%A4%EA%B8%B0-by-docker-compose)\n\n1. `git clone` (or `git init` \u0026 `git remote add origin ...` \u0026 `git pull`)\n2. check out the `.env-example` and `cp ./.env-example ./.env`\n3. **Postgreasql** - create table \u0026 insert datas by `src/db_dummy.sql`\n4. **Logstash** - create a file named `inspector-index.dat` at `logstash` directory with the content `--- 0`\n5. run `start.sh` (`source start.sh`), It will take more than 5min.\n6. Wait until the `setup` service finishes generating SSL certificates\n7. When it's done, go to kibana by `http://localhost:5601`\n8. This project's images are optimized for Intel-based operating systems.\n\n![](./imgs/img1.png)\n\n![](./imgs/img2.png)\n\n## 2. Container ENV \n\n1. `node.name=es-node1`: Specifies the name of the Elasticsearch node.\n2. `discovery.seed_hosts=es1,es2`: Lists the seed hosts that the node will use for cluster discovery. This helps nodes find and join the cluster.\n3. `cluster.initial_master_nodes=es-node0,es-node1,es-node2`: Specifies the initial master nodes in the cluster. It helps bootstrap the cluster formation.\n4. `cluster.name=docker-cluster`: Sets the name of the Elasticsearch cluster.\n5. `bootstrap.memory_lock=true`: Enables memory locking to prevent Elasticsearch from swapping memory to disk.\n6. `\"ES_JAVA_OPTS=-Xms512m -Xmx512m\"`: Sets the JVM options for Elasticsearch, specifying the minimum and maximum heap sizes.\n\n## 3. Additional Info\n\n- To properly run the docker compose, it is crucial to verify the permissions of the executing user and the files and directories bound to the volumes! \n\n- If you want to check out the jdbc option at logstash input, go to https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#plugins-inputs-jdbc-jdbc_paging_enabled\n\n- If you want to check out the more options for elasticsearch, go to https://www.elastic.co/guide/en/elasticsearch/reference/8.8/settings.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnuung%2Felk-psql-cdc-boilerplate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnuung%2Felk-psql-cdc-boilerplate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnuung%2Felk-psql-cdc-boilerplate/lists"}