{"id":21476049,"url":"https://github.com/n25a/eavesdropper","last_synced_at":"2025-07-15T10:31:51.021Z","repository":{"id":211408826,"uuid":"568183486","full_name":"n25a/eavesdropper","owner":"n25a","description":"Eavesdropper is project that used for consuming in message queue and storing in database.","archived":false,"fork":false,"pushed_at":"2022-12-09T17:53:09.000Z","size":65,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-06-21T19:15:38.976Z","etag":null,"topics":["database","go","golang","message-queue","nats","timescaledb"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/n25a.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-11-19T17:58:04.000Z","updated_at":"2024-06-04T17:49:56.000Z","dependencies_parsed_at":"2023-12-08T10:47:22.796Z","dependency_job_id":null,"html_url":"https://github.com/n25a/eavesdropper","commit_stats":null,"previous_names":["n25a/eavesdropper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/n25a%2Feavesdropper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/n25a%2Feavesdropper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/n25a%2Feavesdropper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/n25a%2Feavesdropper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/n25a","download_url":"https://codeload.github.com/n25a/eavesdropper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226033272,"owners_count":17563125,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","go","golang","message-queue","nats","timescaledb"],"created_at":"2024-11-23T10:46:51.341Z","updated_at":"2024-11-23T10:46:51.923Z","avatar_url":"https://github.com/n25a.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![LOGO](https://user-images.githubusercontent.com/49960770/204108690-64eb8ae5-1fac-41f4-a846-c84a9f8c800f.png)\n\n# Eavesdropper\nEavesdropper is project that used for consuming in message queue and storing in database.\n\n\n## Features\n- Consume message from Nats\n- Store message in timescaleDB\n\n\n## How does eavesdropper work?\nEavesdropper configure message queue and database with input data from config file.\nAfter that, eavesdropper will parse schema file to understand the structure of message.\nThen, eavesdropper will consume message from message queue and store it in database.  \nEavesdropper will create table in database with migration files and create column in table with name of message queue's field.\n\n\n### What is schema file?\nSchema file is a file that used for understanding the structure of message. It's contain two parts:\n- Subject (required)\n- Storage (required)\n\n\n#### Subject\nSubject is a topic that message queue should consume on it.\n\n#### Storage\nStorage is a metadata of example message that received from message queue. It's contain list of two parts:\n- Table (required)\n- Field To DB Column (required)\n\n##### Table\nTable name is a name of table that message should store in it for selected subject.\n\n##### Field To DB Column\nField To DB Column is a list of field that message queue should store in database. It's a dictionary that key is \nname of field in message and value is name of column in database.\n\n##### NOTE: You can define multiple subject in one schema file! Also, You can store fields of message in multiple table. Check example below.\n````yaml\n- subject: \"test-subject-1\"\n  storage:\n    - table: \"test_one\"\n      field_to_db_column:\n        data: \"data\"\n        status: \"state\"\n\n- subject: \"test-subject-2\"\n  storage:\n    - table: \"test_two\"\n      field_to_db_column:\n        var1: \"var_one\"\n        var2: \"status\"\n    - table: \"test_three\"\n      field_to_db_column:\n        var1: \"var_one\"\n        var3: \"var_three\"\n````\nIn this example, Eavesdropper has three table in database with name of test_one and test_two. It's store data of message that received from test-subject-1 and test-subject-2.\nIn \"test-subject-2\" topic, Eavesdropper store data of message in two table. var1 and var2 store in `test_two` and\nvar1 and var3 store in `test_three`. \n\n\n## How to run locally?\nAfter cloning the project, you can run it locally by using the following commands:\n\n```bash\nmake mod\nmake build-eavesdropper\ndocker-compose -f docker-compose.dev.yaml up -d\neavesdropper --config config.yaml\n```\n\n\n## Usage\n1) Clone this repository\n2) Install dependencies\n3) Write your own configuration\n4) Write your own schema\n5) Run the project with docker-compose\n\n```mermaid\ngraph LR\nA[Clone this repository] --\u003e B[Install dependencies]\nB --\u003e C[Write your own configuration]\nC --\u003e D[Write your own schema]\nD --\u003e E[Run the project with docker-compose]\n```\n\n## License\n\n[MIT](https://choosealicense.com/licenses/mit/)\n\n\n## Authors\n\n- [@n25a](https://www.github.com/n25a)\n\n\n## todo\n* [X] GitHub actions\n* [X] app package\n* [X] DSN\n* [X] CLI\n* [X] license\n* [X] security\n* [X] issue templates\n* [X] feature templates\n* [X] pull request templates\n* [X] gitignore\n* [X] query builder\n* [X] logger\n* [X] config file\n* [X] dockerized\n* [X] makefile\n* [X] godoc\n* [X] readme\n* [ ] test\n* [ ] Nats config\n* [ ] helm\n* [ ] changelog\n* [ ] release\n* [ ] contributing\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fn25a%2Feavesdropper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fn25a%2Feavesdropper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fn25a%2Feavesdropper/lists"}