{"id":32167602,"url":"https://github.com/rustysnek/venomous","last_synced_at":"2025-10-21T15:33:46.454Z","repository":{"id":243314415,"uuid":"812089417","full_name":"RustySnek/Venomous","owner":"RustySnek","description":"An Erlport wrapper for concurrent use of Python processes.","archived":false,"fork":false,"pushed_at":"2025-03-03T08:08:58.000Z","size":1742,"stargazers_count":36,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-07T01:21:57.677Z","etag":null,"topics":["elixir","erlport","python"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RustySnek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-07T23:56:34.000Z","updated_at":"2025-09-23T07:48:32.000Z","dependencies_parsed_at":"2024-10-28T16:23:19.136Z","dependency_job_id":"eee66994-5da4-42ec-b29a-410e3053bac2","html_url":"https://github.com/RustySnek/Venomous","commit_stats":null,"previous_names":["rustysnek/venomous"],"tags_count":26,"template":false,"template_full_name":null,"purl":"pkg:github/RustySnek/Venomous","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RustySnek%2FVenomous","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RustySnek%2FVenomous/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RustySnek%2FVenomous/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RustySnek%2FVenomous/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RustySnek","download_url":"https://codeload.github.com/RustySnek/Venomous/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RustySnek%2FVenomous/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280287117,"owners_count":26304874,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-21T02:00:06.614Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","erlport","python"],"created_at":"2025-10-21T15:33:44.313Z","updated_at":"2025-10-21T15:33:46.444Z","avatar_url":"https://github.com/RustySnek.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Venomous](https://github.com/RustySnek/Venomous/blob/master/assets/venomous_logo.png)\n\n\u003e A wrapper for managing concurrent [Erlport](http://erlport.org/) Python processes with ease.\n\n[![CI](https://github.com/rustysnek/venomous/actions/workflows/elixir.yml/badge.svg)](https://github.com/rustysnek/venomous/actions/workflows/elixir.yml)\n[![Hex.pm](https://img.shields.io/hexpm/v/venomous)](https://hex.pm/packages/venomous)\n[![Hex.pm](http://img.shields.io/hexpm/dt/venomous.svg)](https://hex.pm/packages/venomous)\n\nVenomous is a wrapper around erlport python Ports, designed to simplify concurrent use. It focuses on dynamic extensibility, like spawning, reusing and killing processes on demand. Furthermore, unused processes get automatically killed by scheduled process which can be configured inside config.exs. Venomous core functions capture and handle :EXIT calls ensuring that all python process die with it and do not continue their execution.\n\n## Installation\nAdd `:venomous` to your list of dependencies in `mix.exs`:\n```elixir\ndef deps do\n  [\n    {:venomous, \"~\u003e 1.0.0\"}\n  ]\nend\n```\n## Getting Started  \n  Check the [documentation](https://hexdocs.pm/venomous) for more in-depth information.\n  \n  For custom type conversion see the [Handling Erlport API](https://github.com/RustySnek/Venomous/blob/master/PYTHON.md)\n\n  \u003e By default the python modules to load are kept inside PYTHONPATH envvar.\n  \u003e but I highly recommend setting them inside python_opts[:module_paths] for hot reloading comp.\n\n  You can checkout examples [here](https://github.com/RustySnek/venomous-examples)\n\n### Configure the SnakeManager options\n  ```elixir\n  config :venomous, :snake_manager, %{\n    # Optional :erlport encoder/decoder for type conversion between elixir/python applied to all workers. The function may also include any :erlport callbacks from python api\n    erlport_encoder: %{\n      module: :my_encoder_module,\n      func: :encode_my_snakes_please,\n      args: []\n    },\n    # TTL whenever python process is inactive. Default: 15\n    snake_ttl_minutes: 10,\n    # Number of python workers that don't get cleared by SnakeManager when their TTL while inactive ends. Default: 10\n    perpetual_workers: 1,\n    # Interval for killing python processes past their ttl while inactive. Default: 60_000ms (1 min)\n    cleaner_interval: 5_000,\n    # reload module for hot reloading.\n    # default is already provided inside venomous python/ directory\n    reload_module: :reload,\n\n    # Erlport python options\n    python_opts: [\n    module_paths: [\"/path/to/my/modules\", \"/path/to/other/modules\"], # List of paths to your python modules.\n    cd: \"/\", # Change python's directory on spawn. Default is $PWD\n    compressed: 0, # Can be set from 0-9. May affect performance. Read more on [Erlport documentation](http://erlport.org/docs/python.html#erlang-api)\n    envvars: [SNAKE_VAR_ONE: \"I'm a snake\", SNAKE_VAR_TWO: \"No, you are not\"], # additional python process envvars\n    packet_bytes: 4, # Size of erlport python packet. Default: 4 = max 4GB of data. Can also be set to 1 = 256 bytes or 2 = ? bytes if you are sure you won't be transfering a lot of data.\n    python_executable: \"/bin/python\" # Change the path to python executable to use.\n    ]\n  }\n  ```\n### Enable the Hot reloading\n  Requires watchdog python module, which can be installed with `mix venomous.watchdog install`.\n  Currently only supports the SnakeManager's processes. Watches only directories specified in `module_paths`\n  ```elixir\n      config :venomous, :serpent_watcher, [\n        enable: true, # Defaults to false\n        logging: true, # log every hot reload. Default: true\n        module: :serpent_watcher, # Provided by default\n        func: :watch_directories, # Provided by default\n        manager_pid: Venomous.SnakeManager, # Provided by default\n        ]\n  ```\n### Configure the SnakeSupervisor and PetSnakeSupervisor (if needed) to start on application boot.\n  ```elixir\n  defmodule YourApp.Application do\n    @moduledoc false\n\n    use Application\n\n    @doc false\n    def start(_type, _args) do\n      children = [\n        {Venomous.SnakeSupervisor, [strategy: :one_for_one, max_restarts: 0, max_children: 50]},\n        {Venomous.PetSnakeSupervisor, [strategy: :one_for_one, max_children: 10]} # not necessary\n      ]\n      opts = [strategy: :one_for_one, name: YourApp.Supervisor]\n      Supervisor.start_link(children, opts)\n    end\n  end\n  ```\n\n## Quickstart\n### Basic way to call python process\n```elixir\nalias Venomous.SnakeArgs\nimport Venomous\n\ntimeout = 1_000\nargs = SnakeArgs.from_params(:builtins, :sum, [[0,1,2,3,4,5]])\n\ncase python(args, python_timeout: timeout) do\n    {:retrieve_error, msg} -\u003e \"No Snakes? #{inspect(msg)}\"\n    %{error: :timeout} -\u003e \"We timed out...\"\n    sum -\u003e assert sum == 15\nend\n\n# or just use python!/3 which waits for the available snake.\ntimeout = :infinity\nassert python!(args, python_timeout: timeout) == 15\n```\n### Concurrency and :EXIT signals\nVenomous is designed with concurrency, as well as proper exits in mind.\n```elixir\nalias Venomous.SnakeArgs\nimport Venomous\n\n# Venomous can handle as much concurrent python as you've setup\n# in your snake_manager configuration. However the python! will\n# wait for any process to free up in case none are available.\nargs = SnakeArgs.from_params(:time, :sleep, [0.5])\nEnum.map(1..100, fn _ -\u003e \n    Task.async(fn -\u003e\n        python!(args)\n    end)\nend) |\u003e Task.await_many(5_000)\n\n# You can view the spawned and ready snakes using the list_alive_snakes() \nlist_alive_snakes() |\u003e dbg\n```\n```elixir\nalias Venomous.SnakeArgs\nimport Venomous\n\n# Venomous kills the OS pid of the python process on :EXIT\n# ensuring the process will not proceed with the execution further\nEnum.map(1..200, fn _ -\u003e\n  {:ok, pid} =\n    Task.start(fn -\u003e\n      SnakeArgs.from_params(:time, :sleep, [1000]) |\u003e python!()\n    end)\n\n  pid\nend)\n|\u003e Enum.each(fn pid -\u003e\n  Process.send_after(pid, {:EXIT, :snake_slithered_away}, 100)\nend)\n\n# We'll sleep to make sure all exits got sent.\nProcess.sleep(1_000)\nassert list_alive_snakes() == []\n```\n\n## Struct/Class comp\nVenomous provides an easy way to convert structs into classes and back with VenomousTrait class and `mix venomous.structs ...` task.\n```\n$ mix venomous.structs\nSimple utility to create python elixir compatible classes.\n\n        VenomousTrait class provides 2 functions: \n          - def from_dict(cls, erl_map: Map | Dict, structs: Dict = {}) -\u003e cls\n            # converts Erlport Map or a Dict into the object class\n          - def into_erl(self, encoding_function \\\\ encode_basic_type_strings, *args \\\\ passed into the encoding function) -\u003e Map\n            # returns erlang compatible struct from self\n\n           \n        To create basic python classes and encode/decode functions based on structs: \n            - mix venomous.structs MyModule.MyStruct MyModule.MoreStructs ...\n\n        To create extended classes depending on existing python class: \n            - mix venomous.structs MyModule.MyStruct:PythonClassName ...\n\n        To create for all available structs inside an application\n            - mix venomous.structs all my_application\n```\n\nYou can see this used in the [struct_test.exs](https://github.com/RustySnek/Venomous/blob/master/test/struct_test.exs) and [test_venomous.py](https://github.com/RustySnek/Venomous/blob/master/python/test_venomous.py)\n\n## Dev/Test REPL\nVenomous provides dev/test only REPL\n```elixir\n$ iex -S mix test\nErlang/OTP 25 [erts-13.2.2.7] [source] [64-bit] [smp:16:2] [ds:16:2:10] [async-threads:1] [jit:ns]\n\nCompiling 1 file (.ex)\n\n15:45:10.953 [info] Started Snake Manager\n \n15:45:10.954 [info] Started Pet Snake Manager\n............\nFinished in 12.9 seconds (0.00s async, 12.9s sync)\n12 tests, 0 failures\n\nRandomized with seed 961929\nInteractive Elixir (1.16.2) - press Ctrl+C to exit (type h() ENTER for help)\niex(1)\u003e test_struct = %VenomousTest.TestStruct{test: \"123\", snake: [\"s\",\"s\",\"s\"]}\n%VenomousTest.TestStruct{test: \"123\", snake: [\"s\", \"s\", \"s\"]}\niex(2)\u003e VenomousREPL.repl(inputs: [test_struct: test_struct])\nPython REPL (module/outputs/pop/r (repeat)/exit): test_venomous\nPython REPL (function): \nAvailable functions:\n\nTest()\n        name: self\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: undefined\n\n        name: test\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: \u003cclass 'str'\u003e\n\n        name: snake\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: \u003cclass 'list'\u003e\nTestStruct()\n        name: self\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: undefined\n\n        name: test\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: \u003cclass 'str'\u003e\n\n        name: snake\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: \u003cclass 'list'\u003e\n\n        name: __struct__\n        kind: POSITIONAL_OR_KEYWORD\n        default: b'Elixir.VenomousTest.TestStruct'\n        annotation: \u003cclass 'erlport.erlterms.Atom'\u003e\nVenom()\n        name: self\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: undefined\n\n        name: test_struct\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: \u003cclass 'test_venomous.Test'\u003e\nVenomStruct()\n        name: self\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: undefined\n\n        name: test_struct\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: \u003cclass 'test_venomous.Test'\u003e\n\n        name: __struct__\n        kind: POSITIONAL_OR_KEYWORD\n        default: b'Elixir.VenomousTest.Venom'\n        annotation: \u003cclass 'erlport.erlterms.Atom'\u003e\ndecoder()\n        name: value\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: typing.Any\nencoder()\n        name: value\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: typing.Any\nerl_encode()\n        \ntest_venomous_trait()\n        name: test\n        kind: POSITIONAL_OR_KEYWORD\n        default: undefined\n        annotation: undefined\nPython REPL (function): test_venomous_trait\nPython REPL (arg 1): [{%{\"x\" =\u003e test_struct}}, \"abc\"]\nPython REPL (arg 2): \n[lib/repl.ex:121: VenomousREPL.repl/1]\nVenomous.python!(params) #=\u003e [\n  %VenomousTest.Venom{\n    test_struct: %{\n      \"__struct__\" =\u003e VenomousTest.TestStruct,\n      \"snake\" =\u003e [\"s\", \"s\", \"s\"],\n      \"test\" =\u003e \"123\"\n    }\n  },\n  \"abc\"\n]\n\nPython REPL (module/outputs/pop/r (repeat)/exit): r\n[lib/repl.ex:109: VenomousREPL.repl/1]\nVenomous.python!(previous_args) #=\u003e [\n  %VenomousTest.Venom{\n    test_struct: %{\n      \"__struct__\" =\u003e VenomousTest.TestStruct,\n      \"snake\" =\u003e [\"s\", \"s\", \"s\"],\n      \"test\" =\u003e \"123\"\n    }\n  },\n  \"abc\"\n]\n\nPython REPL (module/outputs/pop/r (repeat)/exit): outputs\n[lib/repl.ex:96: VenomousREPL.repl/1]\noutputs #=\u003e [\n  [\n    %VenomousTest.Venom{\n      test_struct: %{\n        \"__struct__\" =\u003e VenomousTest.TestStruct,\n        \"snake\" =\u003e [\"s\", \"s\", \"s\"],\n        \"test\" =\u003e \"123\"\n      }\n    },\n    \"abc\"\n  ]\n]\n\nPython REPL (module/outputs/pop/r (repeat)/exit):\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frustysnek%2Fvenomous","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frustysnek%2Fvenomous","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frustysnek%2Fvenomous/lists"}