{"id":15470449,"url":"https://github.com/akash-akya/ex_cmd","last_synced_at":"2025-12-16T08:17:41.417Z","repository":{"id":37897069,"uuid":"248574733","full_name":"akash-akya/ex_cmd","owner":"akash-akya","description":"ExCmd is an Elixir library to run external programs and to communicate with back pressure","archived":false,"fork":false,"pushed_at":"2023-11-24T18:06:27.000Z","size":14660,"stargazers_count":52,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-04-26T21:20:41.739Z","etag":null,"topics":["elixir","erlang","port","stream"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akash-akya.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-19T18:19:56.000Z","updated_at":"2024-06-22T03:35:46.664Z","dependencies_parsed_at":"2024-06-22T03:35:45.337Z","dependency_job_id":"57b0da51-87c7-4615-b347-e923b8f01ca5","html_url":"https://github.com/akash-akya/ex_cmd","commit_stats":{"total_commits":76,"total_committers":2,"mean_commits":38.0,"dds":"0.013157894736842146","last_synced_commit":"d351096ad84aeeef8fbf5fd1e4114a3991d4f6dd"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akash-akya%2Fex_cmd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akash-akya%2Fex_cmd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akash-akya%2Fex_cmd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akash-akya%2Fex_cmd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akash-akya","download_url":"https://codeload.github.com/akash-akya/ex_cmd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230408170,"owners_count":18220974,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","erlang","port","stream"],"created_at":"2024-10-02T02:04:44.377Z","updated_at":"2025-12-16T08:17:36.343Z","avatar_url":"https://github.com/akash-akya.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ExCmd\n\n[![CI](https://github.com/akash-akya/ex_cmd/actions/workflows/elixir.yml/badge.svg)](https://github.com/akash-akya/ex_cmd/actions/workflows/elixir.yml)\n[![Hex.pm](https://img.shields.io/hexpm/v/ex_cmd.svg)](https://hex.pm/packages/ex_cmd)\n[![docs](https://img.shields.io/badge/docs-hexpm-blue.svg)](https://hexdocs.pm/ex_cmd/)\n\nExCmd is an Elixir library for running and communicating with external programs using a back-pressure mechanism. It provides a robust alternative to Elixir's built-in [Port](https://hexdocs.pm/elixir/Port.html) with improved memory management through demand-driven I/O.\n\n## The Port I/O Problem\n\nWhen using Elixir's built-in [Port](https://hexdocs.pm/elixir/Port.html), running external programs that generate large amounts of output (like streaming video using `ffmpeg`) can quickly lead to memory issues. This happens because Port I/O is not demand-driven - it consumes output from stdout as soon as it's available and sends it to the process mailbox. Since BEAM process mailboxes are unbounded, the output accumulates there waiting to be received.\n\n### Memory Usage Comparison\n\nLet's look at how ExCmd handles memory compared to Port when processing large streams:\n\nUsing Port (memory grows unbounded):\n```elixir\nPort.open({:spawn_executable, \"/bin/cat\"}, [{:args, [\"/dev/random\"]}, {:line, 10}, :binary, :use_stdio])\n```\n\n![Port memory consumption](./images/port.png)\n\nUsing ExCmd (memory remains stable):\n```elixir\nExCmd.stream!(~w(cat /dev/random))\n|\u003e Enum.each(fn data -\u003e\n  IO.puts(IO.iodata_length(data))\nend)\n```\n\n![ExCmd memory consumption](./images/ex_cmd.png)\n\nExCmd solves this by implementing:\n- Demand-driven I/O with proper back-pressure\n- Efficient use of OS-backed stdio buffers\n- Stream-based API that integrates with Elixir's ecosystem\n\n## Key Features\n\n- **Back-pressure Support**: Controls data flow between your application and external programs\n- **Stream Abstraction**: Seamless integration with Elixir's Stream API\n- **Memory Efficient**: Demand-driven I/O prevents memory issues with large data transfers\n- **Cross-platform**: Pre-built binaries for MacOS, Windows, and Linux\n- **Process Management**: Proper program termination with no zombie processes\n- **Selective I/O Control**: Ability to close stdin while keeping stdout open\n- **No Dependencies**: No separate middleware or shim installation required\n\n## Installation\n\nAdd `ex_cmd` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:ex_cmd, \"~\u003e x.x.x\"}\n  ]\nend\n```\n\n## Quick Start Examples\n\n### Basic Command Execution\n\n```elixir\n# Simple command execution\nExCmd.stream!(~w(echo Hello))\n|\u003e Enum.into(\"\")\n# =\u003e \"Hello\\n\"\n\n# Get your IP address\nExCmd.stream!(~w(curl ifconfig.co))\n|\u003e Enum.into(\"\")\n```\n\n### Working with Input Streams\n\n```elixir\n# String input\nExCmd.stream!(~w(cat), input: \"Hello World\")\n|\u003e Enum.into(\"\")\n# =\u003e \"Hello World\"\n\n# List of strings\nExCmd.stream!(~w(cat), input: [\"Hello\", \" \", \"World\"])\n|\u003e Enum.into(\"\")\n# =\u003e \"Hello World\"\n\n# Binary data\nExCmd.stream!(~w(base64), input: \u003c\u003c1, 2, 3, 4, 5\u003e\u003e)\n|\u003e Enum.into(\"\")\n# =\u003e \"AQIDBAU=\\n\"\n\n# IOData\nExCmd.stream!(~w(base64), input: [\u003c\u003c1, 2\u003e\u003e, [3], [\u003c\u003c4, 5\u003e\u003e]])\n|\u003e Enum.into(\"\")\n# =\u003e \"AQIDBAU=\\n\"\n```\n\n### Media Processing Examples\n\n```elixir\n# Extract audio from video with controlled memory usage\nExCmd.stream!(~w(ffmpeg -i pipe:0 -f mp3 pipe:1),\n  input: File.stream!(\"music_video.mkv\", [], 65536))\n|\u003e Stream.into(File.stream!(\"music.mp3\"))\n|\u003e Stream.run()\n\n# Process video streams efficiently\nExCmd.stream!(~w(ffmpeg -i pipe:0 -c:v libx264 -f mp4 pipe:1),\n  input: File.stream!(\"input.mp4\", [], 65536),\n  max_chunk_size: 65536)\n|\u003e Stream.into(File.stream!(\"output.mp4\"))\n|\u003e Stream.run()\n```\n\n### Error Handling\n\n```elixir\n# stream!/2 raises on non-zero exit status\nExCmd.stream!([\"sh\", \"-c\", \"exit 10\"])\n|\u003e Enum.to_list()\n# =\u003e ** (ExCmd.Stream.AbnormalExit) program exited with exit status: 10\n\n# stream/2 returns exit status as last element\nExCmd.stream([\"sh\", \"-c\", \"echo 'foo' \u0026\u0026 exit 10\"])\n|\u003e Enum.to_list()\n# =\u003e [\"foo\\n\", {:exit, {:status, 10}}]\n```\n\n### Advanced Features\n\n```elixir\n# Redirect stderr to stdout\nExCmd.stream!([\"sh\", \"-c\", \"echo foo; echo bar \u003e\u00262\"],\n  stderr: :redirect_to_stdout)\n|\u003e Enum.into(\"\")\n# =\u003e \"foo\\nbar\\n\"\n```\n\n## Alternatives\n\n- For NIF-based solutions without middleware overhead, consider [Exile](https://github.com/akash-akya/exile)\n- For simple command execution without streaming, Elixir's built-in Port might be sufficient\n\n## Documentation\n\nDetailed documentation is available at [HexDocs](https://hexdocs.pm/ex_cmd/).\n\n## License\n\nSee [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakash-akya%2Fex_cmd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakash-akya%2Fex_cmd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakash-akya%2Fex_cmd/lists"}