{"id":25905412,"url":"https://github.com/stevekm/pg-elk","last_synced_at":"2025-10-12T17:10:22.294Z","repository":{"id":94518053,"uuid":"323986027","full_name":"stevekm/pg-elk","owner":"stevekm","description":"Example Postgres database connection to Kibana via ElasticSearch and Logstash","archived":false,"fork":false,"pushed_at":"2024-04-16T15:06:13.000Z","size":1756,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-05-12T00:46:27.635Z","etag":null,"topics":["elasticsearch","kibana","logstash","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stevekm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-12-23T19:31:27.000Z","updated_at":"2024-04-16T15:05:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"a79cf0a8-0e4a-472b-a8cd-0f958abc48ec","html_url":"https://github.com/stevekm/pg-elk","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stevekm%2Fpg-elk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stevekm%2Fpg-elk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stevekm%2Fpg-elk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stevekm%2Fpg-elk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stevekm","download_url":"https://codeload.github.com/stevekm/pg-elk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241610990,"owners_count":19990508,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","kibana","logstash","postgresql"],"created_at":"2025-03-03T05:15:07.280Z","updated_at":"2025-10-12T17:10:17.273Z","avatar_url":"https://github.com/stevekm.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pg-elk\n\nMethods for connecting a PostgreSQL database to ElasticSearch and Kibana via Logstash\n\n**NOTE**: There is a newer version of this that uses MySQL and Docker Compose instead located here; https://github.com/stevekm/mysql-elk. If you are trying to connect a SQL db to ELK then you might want to review that version as well since its Docker usage is preferable to the conda + binary installation and management methods here. The methods for connecting the SQL db to Logstash are mostly the same in both cases.\n\n# Setup\n\n- *NOTE*: see the `Makefile` contents for the exact commands and enviornment configurations used in the recipes described here.\n\n## Install\n\nInstall dependencies:\n\n```\nmake install\n```\n\nThis will install a fresh `conda` to the local directory with PostgreSQL installed inside, and download and extract the software needed to run ElasticSearch, Logstash, and Kibana. The `Makefile` has been pre-configured to use this installed software for the demonstration.\n\n## Initialize Servers\n\nInitialize Postgres database (give it a password of 'admin'):\n\n```\nmake pg-init\n```\n\nStart ElasticSearch:\n\n```\nmake es-start\n```\n\nIn a separate terminal session, start Logstash:\n\n```\nmake ls-start\n```\n\n## Data Import\n\nImport some rows to Postgres database (run this a few times, using the password from before)\n\n```\nmake pg-import\n```\n\nYou can verify that results were imported to the database with \n\n```\nmake pg-show\n```\n\nThe results should look like this;\n\n```\n1|817|Sample7|Lung|Normal|2020-12-23 09:49:44.56633\n2|864|Sample9|Skin|Tumor|2020-12-23 09:49:44.56633\n3|575|Sample10|Lung|Tumor|2020-12-23 09:49:44.56633\n4|437|Sample3|Skin|Tumor|2020-12-23 09:49:44.56633\n```\n\nLogstash should be automatically importing new entries from PostgreSQL to ElasticSearch, and its console log should show entries that look like this;\n\n```\n[2020-12-23T13:01:00,255][INFO ][logstash.inputs.jdbc     ][main][1169d815293ec69820cf79b20b70d8d5341059d0eff6abd10a319d72e8f2c0f6] (0.001520s) SELECT * from data WHERE id \u003e 108\n{\n      \"coverage\" =\u003e 302,\n    \"@timestamp\" =\u003e 2020-12-23T18:01:00.285Z,\n      \"sampleid\" =\u003e \"Sample8\",\n       \"created\" =\u003e 2020-12-23T18:00:40.545Z,\n      \"@version\" =\u003e \"1\",\n        \"tissue\" =\u003e \"Heart\",\n            \"id\" =\u003e 116,\n          \"type\" =\u003e \"Tumor\"\n}\n```\n\nYou can verify that the entries have been imported to ElasticSearch with the command\n\n```\nmake es-show\n```\n\nThe outputs should look like this;\n\n```\ncurl  \"http://localhost:9200/pg_data/_search?pretty=true\"\n{\n  \"took\" : 39,\n  \"timed_out\" : false,\n  \"_shards\" : {\n    \"total\" : 1,\n    \"successful\" : 1,\n    \"skipped\" : 0,\n    \"failed\" : 0\n  },\n  \"hits\" : {\n    \"total\" : {\n      \"value\" : 135,\n      \"relation\" : \"eq\"\n    },\n    \"max_score\" : 1.0,\n    \"hits\" : [\n      {\n        \"_index\" : \"pg_data\",\n        \"_type\" : \"_doc\",\n        \"_id\" : \"38\",\n        \"_score\" : 1.0,\n        \"_source\" : {\n          \"@version\" : \"1\",\n          \"sampleid\" : \"Sample4\",\n          \"@timestamp\" : \"2020-12-23T14:59:00.136Z\",\n          \"coverage\" : 381,\n          \"tissue\" : \"Brain\",\n          \"type\" : \"Tumor\",\n          \"id\" : 38,\n          \"created\" : \"2020-12-23T14:58:15.224Z\"\n        }\n...\n```\n\n## Web Dashboard\n\nIn a separate terminal session, start Kibana\n\n```\nmake kib-start\n```\n\nOpen your web browser to http://localhost:5602\n\nGo to Management \u003e Stack Management \u003e Index Patterns \u003e Create Index Pattern \u003e add a pattern for \"pg_data\", the index created for the imported PostgreSQL entries.\n\n![screenshot](https://github.com/stevekm/pg-elk/raw/master/images/Screen%20Shot%202020-12-23%20at%202.56.48%20PM.png)\n\nThen you can go to Visualize \u003e Create New Visualization to start making visualizations on the data in the \"pg_data\" index. Examples:\n\n![screenshot](https://github.com/stevekm/pg-elk/raw/master/images/Screen%20Shot%202020-12-23%20at%2010.04.33%20AM.png)\n\n![screenshot](https://github.com/stevekm/pg-elk/raw/master/images/Screen%20Shot%202020-12-23%20at%2010.08.09%20AM.png)\n\nIf needed, you can refresh the pg_data index by going to Management \u003e Stack Management \u003e Index Management, select 'pg_data' check box, then Manage Index \u003e Refresh Index\n\n# Extras\n\nSee the `Makefile` for all included recipes. Some useful ones are listed here.\n\n- stop the PostgreSQL database server, and ElasticSearch server\n\n```\nmake pg-stop\nmake es-stop\n```\n\n- check the status of the Postgres and ElasticSearch servers\n\n```\nmake pg-check\nmake es-check\n```\n\n- run some example queries on ElasticSearch\n\n```\nmake es-query1\nmake es-query2\nmake es-query3\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstevekm%2Fpg-elk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstevekm%2Fpg-elk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstevekm%2Fpg-elk/lists"}