{"id":25526256,"url":"https://github.com/exbotanical/surveil","last_synced_at":"2026-04-17T06:32:30.834Z","repository":{"id":106027734,"uuid":"349647209","full_name":"exbotanical/surveil","owner":"exbotanical","description":"A fault-tolerant task-runner for continuous integration","archived":false,"fork":false,"pushed_at":"2021-04-07T04:54:46.000Z","size":123,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-21T09:04:26.339Z","etag":null,"topics":["automated-testing","continuous-integration","fault-tolerance","socket-io","task-runner"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/exbotanical.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-03-20T06:32:50.000Z","updated_at":"2021-04-07T04:54:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"e1a4b727-ecc3-4dd3-b173-ae9c5b79db1d","html_url":"https://github.com/exbotanical/surveil","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/exbotanical/surveil","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exbotanical%2Fsurveil","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exbotanical%2Fsurveil/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exbotanical%2Fsurveil/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exbotanical%2Fsurveil/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/exbotanical","download_url":"https://codeload.github.com/exbotanical/surveil/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exbotanical%2Fsurveil/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31918479,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T18:22:33.417Z","status":"online","status_checked_at":"2026-04-17T02:00:06.879Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automated-testing","continuous-integration","fault-tolerance","socket-io","task-runner"],"created_at":"2025-02-19T21:17:05.614Z","updated_at":"2026-04-17T06:32:30.806Z","avatar_url":"https://github.com/exbotanical.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Surveil: A multi-threaded, fault-tolerant task-runner for continuous integration\n\n## Usage\n\nLaunch the observatory: `python3 -m surveil --dispatch=hostname:port /abs/path/to/git/repo`\n\nOn another machine (or in another shell), launch the dispatch service: `python3 -m surveil dispatcher --host=hostname --port=portnum`\n\nFinally, initialize the Threading service: `python3 -m surveil runner --host=hostname --port=portnum --dispatch=hostname:port /abs/path/to/git/repo`\n\n## Architecture\n\nSurveil observes a target git repository for commit SHA changes. When the main branch's HEAD changes, a message is broadcast to all deployments (task-runners), which then execute a series of pre-defined tasks. The runners then report the results of these tasks back to the dispatcher.\n\n![Architecture Control Flow](https://github.com/MatthewZito/surveil/blob/master/docs/surveil.png)\n\n### Observer\n\nThe observer component polls the target repository, notifying a dispatch service of new commit SHAs.\n\n### Dispatch Srv\n\nThe dispatch service delegates tasks. It listens for requests for task-runners from the observer. This service is also responsible for monitoring the health of all threads - if a task-runner process dies, or prematurely exits, the remaining task pool is redistributed to the available threads. If no threads are available, the dispatch service will spawn them.\n\n### Task-runners \u0026 Jobs\n\nThe redistribution and task-runner monitor each run on their own dedicated threads, for which the dispatcher provides failover redundancy.\n\nThe task-runner monitor pings each registered task runner to ensure liveness. If a runner has become unresponsive - and what constitutes as 'responsive' here is a configurable timeout period - the runner is removed from the pool and its task ID is reassigned to the next available thread.\n\nThe redistributor thread polls the tasks pool for new commit SHAs and assigns new work as needed.\n\n## Threaded Socket Server\n\nPython's SocketServer TCPServer out-of-the-box can only handle a single request at any given time. If the dispatch service needs to talk to more than one task-runner, for example, using this default would mean synchronous, blocking I/O.\n\nIn its stead, Surveil uses a modified socket server that handles threading; a new process is spawned for each inbound connection.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexbotanical%2Fsurveil","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fexbotanical%2Fsurveil","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexbotanical%2Fsurveil/lists"}