{"id":13714061,"url":"https://github.com/joelmcdaniel/injest-delivery","last_synced_at":"2026-04-10T21:04:18.252Z","repository":{"id":121888312,"uuid":"191490476","full_name":"joelmcdaniel/injest-delivery","owner":"joelmcdaniel","description":"An coding exercise solution that led to subsequent interview rounds. The ingest agent was requested to be in php7 and the delivery agent in Go. I had never coded in php7 at all when I coded this and was still getting my feet wet around Go. I liked using Redis as a queue.","archived":false,"fork":false,"pushed_at":"2019-06-12T03:36:34.000Z","size":5,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-05-18T01:58:33.794Z","etag":null,"topics":["go","golang","php7","redis"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joelmcdaniel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-12T03:26:35.000Z","updated_at":"2024-05-18T01:58:34.260Z","dependencies_parsed_at":null,"dependency_job_id":"8f478b1a-4386-4e61-80fe-127604c9ee27","html_url":"https://github.com/joelmcdaniel/injest-delivery","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joelmcdaniel%2Finjest-delivery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joelmcdaniel%2Finjest-delivery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joelmcdaniel%2Finjest-delivery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joelmcdaniel%2Finjest-delivery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joelmcdaniel","download_url":"https://codeload.github.com/joelmcdaniel/injest-delivery/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247876942,"owners_count":21011147,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","php7","redis"],"created_at":"2024-08-02T23:01:50.906Z","updated_at":"2026-04-10T21:04:18.164Z","avatar_url":"https://github.com/joelmcdaniel.png","language":"Go","funding_links":[],"categories":["Repositories"],"sub_categories":[],"readme":"# Injest-Delivery\n\nInjest-Delivery is a service that functions as a small scale simulation of how to distribute data to third parties in real time. It consist of two small applications, Injestion Agent (PHP7) and Delivery Agent (Go) which communicate between the two through a Redis job queue.\n\n**Data flow​ :**\n1. Web request (see sample request) \u003e\n2. \"Ingestion Agent\" (php7) \u003e\n3. \"Delivery Queue\" (redis)\n4. \"Delivery Agent\" (go) \u003e\n5. Web response (see sample response)\n\n## **Injestion Agent (PHP7)**\n Injests http requests in json format and pushes a \"postback\" object to a Redis queue for each data object contained in the raw post data. Placeholders {} for query string values in endpoint url will be replaced with their corresponding data property values.\n\n### **Sample Request​ :**\n(POST) \n\nhttp://{server_ip}/ingest.php\n\n(RAW POST DATA)\n```json\n{        \n\t\"endpoint\": {\n\t\t\"method\":\"GET\",         \n\t\t\"url\":\"http://sample_domain_endpoint.com/data?title={mascot}\u0026image={location}\u0026foo={bar}\"\n\t},\n\t\"data\":[\n\t\t{\n\t\t\t\"mascot\":\"Gopher\",\n\t\t\t\"location\":\"https://blog.golang.org/gopher/gopher.png\"\n\t\t},\n\t\t{\n\t\t\t\"mascot\":\"Gopher\",\n\t\t\t\"location\":\"https://blog.golang.org/gopher/gopher.png\",\n\t\t\t\"bar\":\"bar\"\n\t\t}\n\t]\n}\n```\n\n## **Delivery Agent (Go)**\n Continuously pulls the \"postback\" objects from the Redis delivery queue delivering them to the http endpoint url specified in the \"postback\" object. Logs the delivery time, response code, response time and response body of each \"postback\" object sent. \n\n### **Sample Response ​ (Postback):**\nGET\n\n`http://sample_domain_endpoint.com/data?title=Gopher\u0026image=https%3A%2F%2Fblog.golang.org%2Fgopher%2Fgopher.png\u0026foo=`\n\n## **Stack and Installation Steps**\n1. [Ubuntu](https://www.ubuntu.com/download) (preferably 18.04) or some variation of (I'm running [Peppermint 9](https://peppermintos.com/2019/01/peppermint-9-respin-2-released/))\n2. [Redis](https://redis.io/download)\n3. [Apache2](https://httpd.apache.org/download.cgi#apache24)\n4. [PHP 7](https://www.php.net/downloads.php)\n5. [PhpRedis](https://github.com/phpredis/phpredis)\n6. [go-redis](https://github.com/go-redis/redis)\n7. [Go](https://golang.org/dl/)\n\n### 1. Install Redis\n```bash\nsudo apt update\nsudo apt install redis\n```\n\nCheck installation:\n```bash\nredis-cli --version\n```\n### 2. Install Apache Web Server\n```bash\nsudo apt update\nsudo apt install apache2\n```\n\nVerify installation:\n```bash\napache2 -version\n```\n\n### 3. Install PHP 7.2\n```bash\nsudo apt update\nsudo apt-get install php libapache2-mod-php\n```\nVerify installation:\n```bash\nphp -v\n```\n\nRestart the Apache service to apply the changes:\n```bash\nsudo systemctl restart apache2\n```\n\n### 4. Install Redis command line client and PhpRedis extension\n```bash\n sudo apt-get update\n sudo apt-get install redis-tools php-redis\n```\n### 5. Install Go\nFollow the instructions here: https://golang.org/doc/install\n\n### 6. Install go-redis redis client for Go\ngo get -u `github.com/go-redis/redis`\n\n### 7. Install gotenv\ngo get `github.com/subosito/gotenv`\n\n## **Steps to Run**\n\n1. Copy config.php, injest.php and Postback.php from ingestion-agent directory to /var/www/html to run at http://{server_ip}/ingest.php. Or, alternatively copy whole injestion-agent directory to /var/www/html to run at http://{server_ip}/injestion-agent/ingest.php.\n2. Copy delivery-agent directory to /go/src\n3. From /go/src/delivery-agent run go build\n4. From /go/src/delivery-agent run go run delivery-agent\n5. Using Postman (or some similar tool) copy web request test json data from ingestion-agent/tests-json to begin posting web requests to http://{server_ip}/ingest.php or alternatively http://{server_ip}/injestion-agent/ingest.php\n6. Open delivery-log under /go/src/delivery-agent to view the logged delivery response info. \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoelmcdaniel%2Finjest-delivery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoelmcdaniel%2Finjest-delivery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoelmcdaniel%2Finjest-delivery/lists"}