{"id":25080183,"url":"https://github.com/sitetester/worker-handler","last_synced_at":"2025-04-01T04:39:43.067Z","repository":{"id":130879349,"uuid":"290529712","full_name":"sitetester/worker-handler","owner":"sitetester","description":null,"archived":false,"fork":false,"pushed_at":"2020-08-26T15:53:46.000Z","size":10870,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-09T06:12:17.133Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sitetester.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-26T15:09:38.000Z","updated_at":"2024-06-19T09:10:30.653Z","dependencies_parsed_at":"2023-04-26T04:41:12.374Z","dependency_job_id":null,"html_url":"https://github.com/sitetester/worker-handler","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sitetester%2Fworker-handler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sitetester%2Fworker-handler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sitetester%2Fworker-handler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sitetester%2Fworker-handler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sitetester","download_url":"https://codeload.github.com/sitetester/worker-handler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246586099,"owners_count":20801026,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-07T03:18:29.257Z","updated_at":"2025-04-01T04:39:43.046Z","avatar_url":"https://github.com/sitetester.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"This is about managing and scaling a 3rd party worker application in order to compute a sequence of random numbers within a defined time constraint.   \nIt uses the enclosed worker binary to compute 150 data samples. This program spawn and manage multiple worker (child) processes.  \nThe worker provides a http streaming interface to provide the computed numbers with a throughput of 1 rnd/sec.   \nThere is only one connection per worker. A worker might crash after some time.\n\n### Key Points\n* Write a component to manage the worker processes\n* handle the http connections and consume the generated numbers\n* collect a total amount of 150 data samples\n* compute the total count of numbers you processed\n* compute the total time spent\n* [optional] scale the system to compute 150 numbers within 10sec\n* [optional] scale the system with a max of 16 workers\n\n\n### The Worker\nThe worker can be find enclosed (`bin/worker`), it's a compiled go application.\n\n### Start\n`./worker -workerId 1 -port 3001`\n\n### API\n`http://localhost:3001/rnd?n=100`\n\n`n` defines the number of random numbers the worker will generate.\n\nThe endpoint has a throughput of 1 rnd/sec. It's a chunked http stream.\n\nExample Response\n```bash\nHTTP/1.1 200 OK\nCache-Control: no-cache\nConnection: keep-alive\nContent-Type: application/text\nX-Worker-Id: 1\nDate: Wed, 28 Aug 2019 21:30:01 GMT\nTransfer-Encoding: chunked\n\nrnd=97\nrnd=18\nrnd=21\nrnd=95\nrnd=33\n```\n\n### How It Works  \nConfig params are provided in `WorkerHandler` struct. It launches 16 workers, starting from port # 3001 \u0026 so on in a separate goroutine.\nSince OS may take some time to launch given worker binary(e.g worker.mac), handler waits till `PauseInterval` and then makes an HTTP request to parse data. \nBehind the scenes, it opens TCP socket connection to each worker binary on separate port. In some cases, process might take longer to start (e.g. OS busy), \nso it will retry HTTP request `MaxRetry` times with a configured `PauseInterval`. If TCP connection is successful, it will read each line and parse number. \nEach number is then passed to a data `channel`, where it will append parsed number to `dataSamples` array. It `break`s from `for loop` when it has \npopulated 150 data samples, eventually It returns `Result` with `DataSamples` \u0026 `TotalTimeSpent`. \n\nThis whole process of parsing 150 numbers along with launching 16 workers happens within 10 seconds. This could be seen on terminal.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsitetester%2Fworker-handler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsitetester%2Fworker-handler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsitetester%2Fworker-handler/lists"}