{"id":21125819,"url":"https://github.com/jrobles/data-ingestion-poc","last_synced_at":"2026-05-22T05:05:18.720Z","repository":{"id":77452598,"uuid":"58083595","full_name":"jrobles/data-ingestion-poc","owner":"jrobles","description":"POC for a data ingestion microservice using Go, Elasticsearch, and Rabbitmq","archived":false,"fork":false,"pushed_at":"2016-05-18T02:13:09.000Z","size":24,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-21T05:41:45.692Z","etag":null,"topics":["elasticsearch","golang","rabbitmq","workers"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jrobles.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-04T20:58:11.000Z","updated_at":"2017-10-20T14:25:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"b759ed5f-2a67-4f6a-9d9e-a51d491f93fd","html_url":"https://github.com/jrobles/data-ingestion-poc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrobles%2Fdata-ingestion-poc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrobles%2Fdata-ingestion-poc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrobles%2Fdata-ingestion-poc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrobles%2Fdata-ingestion-poc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jrobles","download_url":"https://codeload.github.com/jrobles/data-ingestion-poc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243573167,"owners_count":20312879,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","golang","rabbitmq","workers"],"created_at":"2024-11-20T04:37:03.448Z","updated_at":"2026-05-22T05:05:18.648Z","avatar_url":"https://github.com/jrobles.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# data-ingestion-poc\nPOC for a data ingestion microservice using Go, Elasticsearch, and Rabbitmq. The concept is: large feeds are imported via an API written in go which concurrently distributes the messages to N workers (Go) via rabbitmq. Each worker processes the record into elasticsearch.\n\nRun the app via docker-compose\n```\n./bin/rebuild\n```\n\nPost to API\n```\ncurl -X POST -H \"Content-Type: application/json\" -d '{\n    \"filename\": \"your_filename\",\n    \"path\": \"/file/path/on/s3\",\n    \"extension\": \"csv\",\n    \"bucket\": \"your-bucket-name\",\n    \"region\": \"your-region\",\n    \"key\": \"YOUR_S3_KEY\",\n    \"secret\": \"YOUR_S3_SECRET\"\n}' \"http://{YOUR_DOCKER_MACHINE_IP}:1337/api/v1/csv\"\n```\n\nAssuming the Item Number column had a value of 44870, the data can be obtained from Elasticsearch via:\n```\ncurl -X GET \"http://{YOUR_DOCKER_MACHINE_IP}:19200/{index}/{type}/_search?q=Item\\ Number:440870\"\n```\nyields:\n```json\n{\n  \"took\": 17,\n  \"timed_out\": false,\n  \"_shards\": {\n    \"total\": 5,\n    \"successful\": 5,\n    \"failed\": 0\n  },\n  \"hits\": {\n    \"total\": 1,\n    \"max_score\": 8.711549,\n    \"hits\": [\n      {\n        \"_index\": \"spacely_sprockets\",\n        \"_type\": \"products\",\n        \"_id\": \"399088\",\n        \"_score\": 8.711549,\n        \"_source\": {\n          \"Active Location\": \"3-A4-076-A-2\",\n          \"Active Lock Code\": \"\",\n          \"Active Units\": \"158\",\n          \"Carton Units\": \"2\",\n          \"Dept\": \"16\",\n          \"Item Description\": \"TEAPOT ELEPHANT WHITE THING STYFF\",\n          \"Item Number\": \"399088\",\n          \"Receipt Date\": \"20140304\",\n          \"Receipt ETA\": \"20140702\",\n          \"Reserve Units\": \"2220\",\n          \"SDC ECom Units\": \"0\"\n        }\n      }\n    ]\n  }\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjrobles%2Fdata-ingestion-poc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjrobles%2Fdata-ingestion-poc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjrobles%2Fdata-ingestion-poc/lists"}