{"id":16370768,"url":"https://github.com/integralist/go-requester","last_synced_at":"2025-03-23T02:34:08.588Z","repository":{"id":36406114,"uuid":"40711046","full_name":"Integralist/Go-Requester","owner":"Integralist","description":"HTTP service that accepts a collection of \"components\", fans-out requests and returns aggregated content","archived":false,"fork":false,"pushed_at":"2015-12-21T13:55:15.000Z","size":6572,"stargazers_count":32,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-18T17:04:12.792Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Integralist.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-14T11:05:48.000Z","updated_at":"2022-12-19T16:58:40.000Z","dependencies_parsed_at":"2022-09-07T08:10:48.521Z","dependency_job_id":null,"html_url":"https://github.com/Integralist/Go-Requester","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Integralist%2FGo-Requester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Integralist%2FGo-Requester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Integralist%2FGo-Requester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Integralist%2FGo-Requester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Integralist","download_url":"https://codeload.github.com/Integralist/Go-Requester/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245048012,"owners_count":20552431,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T03:05:55.961Z","updated_at":"2025-03-23T02:34:08.286Z","avatar_url":"https://github.com/Integralist.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eGo-Requester\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Completed-90%25-green.svg?style=flat-square\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eHTTP service\u003c/b\u003e that accepts a collection of \"components\"\u003cbr\u003ethen fans-out requests and returns \u003cb\u003eaggregated content\u003c/b\u003e\n\u003c/p\u003e\n\n## Summary\n\n- Components should be defined in a YAML page configuration file \n- Components are requested concurrently via goroutines\n- Components can be marked as \"mandatory\" (if they fail, the request summary is set to \"failure\")\n\n## Example Page Config\n\n```yaml\ncomponents:\n  - id: google\n    url: http://google.com\n  - id: integralist\n    url: http://integralist.co.uk\n    mandatory: true\n  - id: not-found\n    url: http://httpstat.us/404\n  - id: timeout\n    url: http://httpstat.us/408\n  - id: server-error\n    url: http://httpstat.us/500\n  - id: service-unavailable\n    url: http://httpstat.us/503\n```\n\n\u003e Note:  \n\u003e example config provided as part of this repo  \n\u003e `./config/page.yaml`\n\n## Example JSON Output\n\n```json\n{\n  \"summary\": \"success\",\n  \"components\": [\n    {\n      \"id\": \"google\",\n      \"status\": 200,\n      \"body\": \"\u003c!doctype html\u003e ... \u003c/html\u003e\",\n      \"mandatory\": false\n    },\n    {\n      \"id\": \"integralist\",\n      \"status\": 200,\n      \"body\": \"\u003c!doctype html\u003e ... \u003c/html\u003e\",\n      \"mandatory\": true\n    },\n    {\n      \"id\": \"slooow\",\n      \"status\": 408,\n      \"body\": \"Get http://localhost:3000/pugs: net/http: request canceled (Client.Timeout exceeded while awaiting headers)\",\n      \"mandatory\": false\n    }\n    {\n      \"id\": \"not-found\",\n      \"status\": 404,\n      \"body\": \"404 Not Found\",\n      \"mandatory\": false\n    }\n  ]\n}\n```\n\n\u003e Note: the toplevel `summary` key's value will be `failure` if any mandatory components fail\n\n## Installing Vendored Dependencies\n\nYou'll need [Glide](https://github.com/Masterminds/glide) installed:\n\n```bash\ngo get github.com/Masterminds/glide\n```\n\nIf using Go `1.6` you're OK, otherwise you'll need to set the vendor experiment:\n\n```bash\nexport GO15VENDOREXPERIMENT=1\n```\n\nTo install the dependencies found in the `Glide.lock` file:\n\n```bash\nglide install\n```\n\n## Run Tests\n\n```bash\ngo test $(glide novendor)\n```\n\n## Build Application\n\n```bash\ngo build\n```\n\nOnce the application is built and installed into your binary path, you can run it:\n\n```bash\ngo-requester \u003cpath/to/your/page/config\u003e\n```\n\n## Running Locally\n\nThere are two ways to run this application locally:\n\n1. Host Machine\n2. Docker\n\n### Host Machine\n\n```bash\ngo run main.go ./config/page.yaml\n```\n\n\u003e Alternative: `godo server --watch`\n\nOnce the application is running, view: `http://localhost:8080/`\n\nIf you want to see how latency/slow responses effect the application, then also try running: https://github.com/Integralist/go-slow-http-server which was specifically designed to be used alongside go-requester for testing purposes.\n\n### Docker\n\n```bash\ndocker build -t gorequester .\ndocker run --rm --name gr -v \"$PWD\":/go/src/app -p 5000:8080 gorequester\n```\n\nOnce the application is running, view: `http://\u003cdocker-machine-ip\u003e:5000/`\n\nTo debug the container:\n\n```bash\ndocker exec -it \u003ccontainer_id\u003e /bin/bash\ndocker run -it --entrypoint /bin/bash -v \"$PWD\":/go/src/app -p 5000:8080 gorequester\n```\n\nIf you specify a private repo as a dependency, then you'll need to pass in your SSH credentials:\n\n```bash\ndocker run \\\n  -it \\\n  -v \"$PWD\":/go/src/app \\\n  -v \"$HOME/.ssh/github_bbc_rsa\":/.ssh/github_rsa \\\n  -p 5000:8080 \\\n  gorequester ssh_setup\n```\n\nNotice we've passed `ssh_setup`. Our bootstrap script (which runs inside the Docker container) will identify when this argument is provided and subsequently load up the SSH agent within the container. \n\nBecause we've added the `-it` flag we're able to manually type in our SSH private key's passphrase (you *do* use a passphrase don't you?).\n\n#### Expect\n\nHaving to type in your SSH passphrase can be annoying an restrict your ability to automate. Although I personally wouldn't advise it, you *can* make a change to the bootstrap script and have it use the `ssh.exp` script included in this repository.\n\nUsing this Expect script will mean you don't have to manually enter the passphrase for your SSH private key. The MASSIVE downside to this process is that you'll need to edit the `ssh.exp` script to include your passphrase. This means your secret passphrase has now been codified into the file.\n\nIf your laptop is compromised or you commit the change accidentally, then you're in serious trouble. So really... don't use it unless you're really sure you know what you're doing.\n\nYou'll also need to comment out the line `ssh-add /.ssh/github_rsa` from the bootstrap script and replace it with `/ssh.exp`.\n\n## Curl Timings\n\nWe can verify the performance of this application using curl timings, like so:\n\n```bash\ncurl -w \"@curl-format.txt\" -o /dev/null -s http://localhost:8080/\n```\n\n\u003e Note: I've included a `curl-format.txt` file within the repo\n\n## TODO\n\n- Refactor code so that certain aspects are loaded from other packages\n- Add logic for loading page config remotely\n- Dynamically change port number when run as binary\n- Tests!\n\n## Licence\n\n[The MIT License (MIT)](http://opensource.org/licenses/MIT)\n\nCopyright (c) 2015 [Mark McDonnell](http://twitter.com/integralist)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintegralist%2Fgo-requester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintegralist%2Fgo-requester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintegralist%2Fgo-requester/lists"}