{"id":20315272,"url":"https://github.com/sysread/denim","last_synced_at":"2025-06-15T23:09:13.511Z","repository":{"id":146316010,"uuid":"13835595","full_name":"sysread/denim","owner":"sysread","description":null,"archived":false,"fork":false,"pushed_at":"2013-11-03T20:39:59.000Z","size":176,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-11T12:59:37.915Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sysread.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-10-24T15:10:57.000Z","updated_at":"2013-11-03T20:40:03.000Z","dependencies_parsed_at":"2023-03-23T23:15:44.408Z","dependency_job_id":null,"html_url":"https://github.com/sysread/denim","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sysread/denim","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysread%2Fdenim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysread%2Fdenim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysread%2Fdenim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysread%2Fdenim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sysread","download_url":"https://codeload.github.com/sysread/denim/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysread%2Fdenim/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260063762,"owners_count":22953592,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T18:18:33.601Z","updated_at":"2025-06-15T23:09:13.491Z","avatar_url":"https://github.com/sysread.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# denim\n\ndenim is a platform for distributed computing in Python. It is designed as a\nrobust, efficient alternative to solutions such as RabbitMQ and celery.\n\n## Design philosophy\n\n    * Ease of implementation of distributed systems\n    * Robust services and bounded queues\n    * Predictable scalability and degradation\n    * Ease of integration with existing systems\n\n## Ease of implementation\n\ndenim requires a minimum of configuration to set up. A single manager can\nmanage any number of worker nodes. Workers automatically notify their manager\nof their availability and begin servicing tasks immediately. The manager\nensures that work is distributed in a sensible fashion across workers in\norder to maximize throughput.\n\n## Robust services and bounded queues\n\ndenim is designed to ensure that services can function independently of each\nother. A service crashing or network lag will never cause other hosts to crash.\n\nWorkers that are restarted are gracefully handled by the manager, picking up\nslack as soon as they come back on line. Likewise, a restarted manager will\npick back up where it left off, with its workers immediately reestablishing\ntheir connections as soon as it becomes available again.\n\nAdditionally, task queues in a denim system are bounded to protect the system\nfrom DOS-style attacks or from simply being overwhelmed by legitimate work.\nThis prevents queues from growing overlarge due to spikes in traffic, resulting\nin a secondary delay in restoring responsiveness while the backlog of queued\ntasks is processed. This ensures that the server and workers remain at peak\nresponsiveness and never thrash their host systems, which can result in\nslowness due to paging and backlogged disk activity.\n\nTasks that cannot be queued due to a lack of resources will be rejected and\nautomatically retried by the client.\n\n## Predictable scalability and degradation\n\nBecause queues are strictly controlled, an administrator can easily estimate\nperformance requirements based on the current performance of a system. Adding\nanother worker node to the system will provide a linear increase in performance.\n\nFor example, in a system with a single manager and two worker nodes, the\naddition of another worker node will result in a 50% increase in performance.\n\nLikewise, as a system begins to reach maximum processing capacity, the\nperformance of workers and the time taken to process individual tasks will\nnot change.\n\nInstead, the client producing the tasks for the system will experience delays\nas tasks are rejected. The client automatically retries the tasks until they\nare accepted. This process is seamless and transparent to code using the client\nclass. Integrating code never needs to worry about handling errors caused by\nbusy systems.\n\n## Ease of integration\n\ndenim is written in Python with a single point of integration - the client\nclass. Integrating code connects to the service and begins sending tasks. No\nextra work is required outside of ensuring that the necessary application\nlibraries are available on worker systems.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysread%2Fdenim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsysread%2Fdenim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysread%2Fdenim/lists"}