{"id":18524390,"url":"https://github.com/binpash/dish","last_synced_at":"2025-10-30T11:54:42.213Z","repository":{"id":104161353,"uuid":"608394094","full_name":"binpash/dish","owner":"binpash","description":"DiSh: Dynamic Shell-Script Distribution","archived":false,"fork":false,"pushed_at":"2024-12-09T02:15:38.000Z","size":4388,"stargazers_count":25,"open_issues_count":0,"forks_count":3,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-02-08T23:27:48.550Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/binpash.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-03-01T23:29:04.000Z","updated_at":"2024-11-29T22:45:06.000Z","dependencies_parsed_at":"2023-11-21T05:34:36.183Z","dependency_job_id":"0a501b51-c556-4954-9fb3-ce52983ddaa0","html_url":"https://github.com/binpash/dish","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/binpash%2Fdish","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/binpash%2Fdish/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/binpash%2Fdish/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/binpash%2Fdish/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/binpash","download_url":"https://codeload.github.com/binpash/dish/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248031642,"owners_count":21036446,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T17:41:15.038Z","updated_at":"2025-10-30T11:54:37.178Z","avatar_url":"https://github.com/binpash.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DiSh: Dynamic Shell-Script Distribution\n\n\u003e _A system for scaling out POSIX shell scripts on distributed file systems._\n\u003e _DiSh is part of the PaSh project, which is hosted by the [Linux Foundation](https://linuxfoundation.org/press-release/linux-foundation-to-host-the-pash-project-accelerating-shell-scripting-with-automated-parallelization-for-industrial-use-cases/)._\n\nDiSh builds heavily on and extends [PaSh](https://github.com/binpash/pash) (command annotations, compiler infrastructure, and JIT orchestration).\n\nQuick Jump: [Installation](#installation) | [Running DiSh](#running-dish) | [Repo Structure](#repo-structure) | [Evaluation](#evaluation) | [Community \u0026 More](#community--more) | [Citing](#citing)\n\n## Installation\n\nThe easiest way to play with DiSh is using docker.\n\nThe following steps commands will create a virtual cluster on one machine allow you to experiment with DiSh. If you have multiple machines, you can setup [docker-swarm](https://docs.docker.com/engine/swarm/swarm-tutorial/) and use the swarm instruction in [docker-hadoop](./docker-hadoop).\n\n```sh\n## Clone the repo\ngit clone --recurse-submodules https://github.com/binpash/dish.git\n\n## Install docker using our script (tested on Ubuntu)\n## Alternatively see https://docs.docker.com/engine/install/ to install docker.\n(cd dish; ./scripts/setup-docker.sh)\n\n\n## Create the virtual cluster on the host machine\n(cd docker-hadoop; ./setup-compose.sh) # currently takes several minutes due to rebuilding the images\n## The cluster can be torn down using `docker compose down`\n\n## Create a shell on the client\ndocker exec -it nodemanager1 bash\n```\n\n## Running DiSh\n\nLet's run a very simple example using DiSh:\n\n```sh\ncd $DISH_TOP\nhdfs dfs -put README.md /README.md # Copies the readme to hdfs\n```\n\nNow, you can run [this sample script](./scripts/sample.sh) (or create a script of your own). Run both DiSh and Bash and compare the results!\n\n```sh\n./di.sh ./scripts/sample.sh\nbash ./scripts/sample.sh\n```\n\n\u003c!-- We first want to download some input data and populate hdfs.\n\n```sh\ncd $DISH_TOP\n./setup.sh # Takes several minutes\n``` --\u003e\n\n\n## Repo Structure\n\nThis repo hosts most of the components of the `dish` development. Some of them are incorporated in [PaSh](https://github.com/binpash/pash). The structure is as follows:\n\n* [pash](./pash): Contains the complete PaSh repo as a submodule. DiSh uses and extends its annotations, compiler, and JIT orchestration infrastructure.\n* [evaluation](./evaluation): Shell scripts used for evaluation.\n* [runtime](./runtime): Runtime component — e.g., remote fifo channels.\n* [scripts](./scripts): Scripts related to installation, deployment, and continuous integration.\n\n\u003c!-- ## Evaluation --\u003e\n\n\u003c!-- __TODO:__ Describe how to run DiSh's evaluation (also setting up a cluster etc). --\u003e\n\n## Community \u0026 More\n\nChat:\n* [Discord Server](ttps://discord.com/channels/947328962739187753/) ([Invite](http://join.binpa.sh/))\n\n## Citing\n\nIf you used DiSh, consider citing the following paper:\n```\n@inproceedings{dish2023nsdi,\nauthor = {Mustafa, Tammam and Kallas, Konstantinos and Das, Pratyush and Vasilakis, Nikos},\ntitle = {{DiSh}: Dynamic {Shell-Script} Distribution},\nbooktitle = {20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)},\nyear = {2023},\nisbn = {978-1-939133-33-5},\naddress = {Boston, MA},\npages = {341--356},\nurl = {https://www.usenix.org/conference/nsdi23/presentation/mustafa},\npublisher = {USENIX Association},\nmonth = apr,\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbinpash%2Fdish","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbinpash%2Fdish","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbinpash%2Fdish/lists"}