{"id":13437139,"url":"https://github.com/mmstick/concurr","last_synced_at":"2025-03-17T00:33:48.715Z","repository":{"id":137272971,"uuid":"105162721","full_name":"mmstick/concurr","owner":"mmstick","description":"Performs distributed command execution, written in Rust w/ Tokio","archived":false,"fork":false,"pushed_at":"2018-06-29T17:04:48.000Z","size":72,"stargazers_count":110,"open_issues_count":1,"forks_count":7,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-09T01:29:50.006Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mmstick.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-09-28T15:02:20.000Z","updated_at":"2025-02-22T05:01:12.000Z","dependencies_parsed_at":"2024-01-07T21:53:57.804Z","dependency_job_id":null,"html_url":"https://github.com/mmstick/concurr","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mmstick%2Fconcurr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mmstick%2Fconcurr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mmstick%2Fconcurr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mmstick%2Fconcurr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mmstick","download_url":"https://codeload.github.com/mmstick/concurr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243835942,"owners_count":20355611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:00:54.594Z","updated_at":"2025-03-17T00:33:47.796Z","avatar_url":"https://github.com/mmstick.png","language":"Rust","funding_links":[],"categories":["Applications","Rust","应用程序 Applications","应用","others","应用 Applications"],"sub_categories":["System tools","系统工具 System tools","系统工具"],"readme":"# Concurr: Distributed and Concurrent Command Execution, in Rust\n\n\u003e This project is dual licensed under MIT and Apache 2.0.\n\nOriginally inspired by the GNU Parallel project, and my previous re-implementation of the project\nin Rust, Concurr is a distributed and concurrent command-line job server \u0026 client architecture for\ngenerating and executing commands in parallel on any number of systems. At it's core, Concurr\nuses Tokio to perform asynchronous network I/O. The client works similarly to GNU Parallel, but\nthere are key differences for the sake of simplicity in operation.\n\n## The Client\n\nThe client is responsible for parsing arguments, connecting to nodes and obtaining slot counts,\nsubmitting a command to each node, distributing inputs to each slot on each connected node,\ncollecting responses from those slots when they complete, and requesting new inputs from a shared\nbuffer. Simple stuff. Syntax is to be very similar to GNU Parallel, but there are some differences.\n\n### Example\n\n```sh\nconcurr 'echo job {#} on slot {%}: {}' : arg1 arg2 arg3 arg4\nconcurr 'echo job {#} on slot {%}: {}' :: file1 file2 file3\nconcurr 'echo {}' \u003c input_file\ncat file | concurr 'echo {}'\n```\n\n### How The Client Works\n\n## The Server\n\n### How The Server Works\n\nThe service works by listening for a number of possible instructions that can be supplied. A\ncommand instruction will tell the server to create a new command with a pool of threads that will\nlisten for inputs, henceforth named as slots. A delete instruction can be used to delete commands\nfrom the server, thus causing all threads to exit after completing their tasks. Input instructions\nwill supply inputs to commands, which are designated by an integer ID, and also specify the ID of\nthe job, which may be useful to the client knowing which input received what results.\n\nEffectively, a client can send a command to a server, which will spawn as many threads as there are\ncores in the system; initializing an instance of the Ion shell on each slot. These slots have\nshared access to an input buffer, output buffer, kill switch, and corresponding counter\nto count the number of threads that have exited. These are all wrapped up together in a unit.\n\nWhen an input is received from a client, that input is matched to a unit and then placed onto an\ninput buffer that is collectively owned by the threads that are attached to that command. When a\nslot grabs that input, it will perform a fork, capture the standard output and error of the fork,\nexecute the command within an embedded Ion instance attached to that slot on the child, and then\nwait for the child to exit before placing the exit status, job ID, and file descriptors containing\nthe standard output and error onto an output buffer.\n\nThe connection that submitted the input will have been waiting for a result that matches the ID of\nthe job that was submitted, and upon seeing that job, will immediately encode a response with the\nresults.\n\n### Example\n\n1. The following command, assigned ID 0, is sent to the server: `echo {#}: {}`\n2. The server has four cores, and creates four slots for that command.\n3. Some inputs are submitted to the command with ID 0:\n  - inp 0 1 one\n  - inp 0 2 two\n  - inp 0 3 three\n  - inp 0 4 four\n4. Slots concurrently grab inputs from the queue, process them, and return their outputs:\n  - 1: one\n  - 2: two\n  - 3: three\n  - 4: four\n5. The exit status, job ID, standard out, and standard error are returned to the client.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmmstick%2Fconcurr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmmstick%2Fconcurr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmmstick%2Fconcurr/lists"}