{"id":18367300,"url":"https://github.com/120dev/elasticsearch-postgres","last_synced_at":"2025-07-01T09:39:14.134Z","repository":{"id":181454249,"uuid":"132434870","full_name":"120dev/ElasticSearch-postgres","owner":"120dev","description":"PostGres To ElasticSearch in Go","archived":false,"fork":false,"pushed_at":"2024-03-30T10:04:20.000Z","size":2586,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-06T16:39:55.228Z","etag":null,"topics":["elasticsearch","go","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/120dev.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-07T09:03:04.000Z","updated_at":"2024-06-21T19:46:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"7f818518-ad01-4932-a083-8efd334f9b88","html_url":"https://github.com/120dev/ElasticSearch-postgres","commit_stats":null,"previous_names":["120dev/elasticsearch-postgres"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/120dev/ElasticSearch-postgres","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/120dev%2FElasticSearch-postgres","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/120dev%2FElasticSearch-postgres/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/120dev%2FElasticSearch-postgres/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/120dev%2FElasticSearch-postgres/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/120dev","download_url":"https://codeload.github.com/120dev/ElasticSearch-postgres/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/120dev%2FElasticSearch-postgres/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262937967,"owners_count":23387705,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","go","postgresql"],"created_at":"2024-11-05T23:21:02.190Z","updated_at":"2025-07-01T09:39:14.086Z","avatar_url":"https://github.com/120dev.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PostgreSQL To ElasticSearch in Go\n\n\n# Requirements\nProject in **Go + SQL**  to create an ElasticSearch **index** from PostgreSQL database.\n\nThis procedure is for a Linux operating system.\n\nThe following dependencies needs to be previously installed :\n* PostgreSQL (database)\n* ElasticSearch (index)\n* Docker (containerizer)\n* PG Admin (Sql manager)\n\nTwo ways to start listening:\n\n 1. Via PostGresToES.go: this will save the raw data from the database\n2. Via PostGresToES_ApiGateway.go: this will query the API in order to retrieve JSON which will be indexed in ES\n\nNothing changes in terms of triggers\n\n\n# Installation\n\n1. Docker :  https://docs.docker.com/install/\n2. PostgreSQL\n```\ndocker pull postgres\ndocker run --name abcd -e POSTGRES_PASSWORD=abcd -d postgres\n```\n\n3. PG Admin\n```\ndocker pull thajeztah/pgadmin4\ndocker run --restart=always -p 5050:5050 thajeztah/pgadmin4\n```\n4. ElasticSearch\n```\ndocker pull elasticsearch\ndocker run --restart=always -d -p 9200:9200 -p 9300:9300 -it -h elasticsearch --name elasticsearch elasticsearch\n```\n\n## Configure :\n1. Create index :\n```\nPUT http://localhost:9200/\u003cINDEX_NAME\u003e\n{\n    \"settings\" : {\n        \"index\" : {\n            \"number_of_shards\" : 5,\n            \"number_of_replicas\" : 2\n        }\n    }\n}\nOR :\ncurl -XPUT  'http://localhost:9200/\u003cINDEX_NAME\u003e' -d '{\"settings\" : {\"number_of_shards\" : 5, \"number_of_replicas\" : 2}}'\n```\n2. Create the PostgreSql function :\n```\nCREATE OR REPLACE FUNCTION public.notify_event()\n    RETURNS trigger\n    LANGUAGE 'plpgsql'\n    COST 100\n    VOLATILE NOT LEAKPROOF\nAS $BODY$\n\n    DECLARE\n        data json;\n        notification json;\n        id integer;\n    BEGIN\n\n        -- Convert the old or new row to JSON, based on the kind of action.\n        -- Action = DELETE?             -\u003e OLD row\n        -- Action = INSERT or UPDATE?   -\u003e NEW row\n        IF (TG_OP = 'DELETE') THEN\n            data = row_to_json(OLD);\n            id = OLD.id;\n        ELSE\n            data = row_to_json(NEW);\n            id = NEW.id;\n        END IF;\n\n        -- Contruct the notification as a JSON string.\n        notification = json_build_object(\n                          'table',TG_TABLE_NAME,\n                          'action', TG_OP,\n                          'id', id,\n                          'data', data);\n\n        -- Execute pg_notify(channel, notification)\n        PERFORM pg_notify('events',notification::text);\n\n        -- Result is ignored since this is an AFTER trigger\n        RETURN NULL;\n    END;\n$BODY$;\n```\n\n3. Create the PostgreSql trigger :\n```\nCREATE TRIGGER products_notify_event\n    AFTER INSERT OR DELETE OR UPDATE\n    ON public.users\n    FOR EACH ROW\n    EXECUTE PROCEDURE public.notify_event();\n```\n\n4. Install Go\n```https://go.dev/doc/install```\n5. Init go.mod\n```go mod init PostGresToES```\n6. Install dependencies\n```go mod tidy```\n7. Setup (auth, indexName) : https://github.com/120dev/ElasticSearch-postgres/blob/master/PostGresToES.go#L20\n8. if PostGresToES_ApiGateway.go : setup conf.yaml\n9. Build\n```go build PostGresToES.go \u0026\u0026 chmod +x ./PostGresToES``` or ```go build PostGresToES_ApiGateway.go \u0026\u0026 chmod +x ./PostGresToES_ApiGateway```\n10. Run Go script\n```\n./PostGresToES or ./PostGresToES_ApiGateway\n```\nAnd wait, all events are logged.\n\n# UML diagrams\n\nYou can render UML diagrams using [Mermaid](https://mermaidjs.github.io/). For example, this will produce a sequence diagram:\n\n```mermaid\nsequenceDiagram\nPostgreSQL -\u003e\u003e Trigger: Event crud in Json\nTrigger -\u003e\u003e Go: Read and set ES _id\nGo -\u003e\u003e ES: Post Json to ES\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F120dev%2Felasticsearch-postgres","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F120dev%2Felasticsearch-postgres","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F120dev%2Felasticsearch-postgres/lists"}