{"id":50457994,"url":"https://github.com/jacotay7/shmpipeline","last_synced_at":"2026-06-01T03:08:13.547Z","repository":{"id":352754293,"uuid":"1190984560","full_name":"jacotay7/shmpipeline","owner":"jacotay7","description":"A tool for building and managing high performance CPU/GPU shared-memory compute pipelines. Ideal for real-time control and image processing applications ","archived":false,"fork":false,"pushed_at":"2026-05-05T23:55:49.000Z","size":1247,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-09T05:07:53.733Z","etag":null,"topics":["adaptive-optics","image-processing","pipeline","real-time-control","shared-memory"],"latest_commit_sha":null,"homepage":"https://shmpipeline.readthedocs.io/en/latest/index.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jacotay7.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-24T20:11:01.000Z","updated_at":"2026-05-05T23:55:52.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jacotay7/shmpipeline","commit_stats":null,"previous_names":["jacotay7/shmpipeline"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/jacotay7/shmpipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacotay7%2Fshmpipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacotay7%2Fshmpipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacotay7%2Fshmpipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacotay7%2Fshmpipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jacotay7","download_url":"https://codeload.github.com/jacotay7/shmpipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacotay7%2Fshmpipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33757796,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-01T02:00:06.963Z","response_time":115,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adaptive-optics","image-processing","pipeline","real-time-control","shared-memory"],"created_at":"2026-06-01T03:08:09.428Z","updated_at":"2026-06-01T03:08:13.538Z","avatar_url":"https://github.com/jacotay7.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# shmpipeline\n\n## Links\n\n- [PyPI](https://pypi.org/project/shmpipeline/)\n- [Read the Docs](https://shmpipeline.readthedocs.io/en/latest/)\n- [Issues](https://github.com/jacotay7/shmpipeline/issues)\n\n## What It Is\n\n`shmpipeline` builds local, process-based compute pipelines on top of named\nshared-memory streams from `pyshmem`.\n\nIt is a good fit when you want explicit dataflow, process isolation, and\nshared-memory throughput without giving up a small Python API, YAML-driven\nconfiguration, or interactive tooling.\n\nTypical uses include:\n\n- adaptive optics and other real-time sensor / control pipelines\n- CPU or GPU processing graphs around shared-memory streams\n- validating, inspecting, and running pipelines from Python or the CLI\n- editing and supervising pipelines from a desktop GUI\n- extending the runtime with custom kernels, sources, and sinks\n\nGood starting points:\n\n- [Installation guide](https://shmpipeline.readthedocs.io/en/latest/getting-started/installation.html)\n- [Quickstart](https://shmpipeline.readthedocs.io/en/latest/getting-started/quickstart.html)\n- [Worked examples](https://shmpipeline.readthedocs.io/en/latest/examples/index.html)\n\n## Installation\n\nFull docs: [Installation guide](https://shmpipeline.readthedocs.io/en/latest/getting-started/installation.html)\n\nChoose the smallest install that matches your workflow:\n\n- Base runtime and CLI: `pip install shmpipeline`\n- GPU support: `pip install \"shmpipeline[gpu]\"`\n- Desktop GUI: `pip install \"shmpipeline[gui]\"`\n- Remote control service: `pip install \"shmpipeline[control]\"`\n- Editable source install: `pip install -e .`\n- Full local development environment: `pip install -e \".[control,gpu,gui,test,docs]\"`\n\nQuick smoke test against a checked-in example:\n\n```bash\nshmpipeline validate examples/affine_transformation/pipeline.yaml\nshmpipeline describe examples/affine_transformation/pipeline.yaml --json\n```\n\n## Open The GUI For The Observatory AO Example\n\nRelevant docs:\n\n- [GUI guide](https://shmpipeline.readthedocs.io/en/latest/guides/gui.html)\n- [Observatory AO system example](https://shmpipeline.readthedocs.io/en/latest/examples/observatory-ao-system.html)\n\nFrom the repository root:\n\n```bash\npip install \"shmpipeline[gui]\"\nshmpipeline-gui examples/observatory_ao_system/pipeline.yaml\n```\n\nThe full GUI can auto-launch a local loopback control server when you press\n`Build` or `Start`.\n\nIf you want to launch the server yourself first:\n\n```bash\nshmpipeline serve examples/observatory_ao_system/pipeline.yaml --host 127.0.0.1 --port 8765\nshmpipeline-gui examples/observatory_ao_system/pipeline.yaml\n```\n\n## Quickstart For The Python API\n\nFull docs:\n\n- [Quickstart](https://shmpipeline.readthedocs.io/en/latest/getting-started/quickstart.html)\n- [Configuration guide](https://shmpipeline.readthedocs.io/en/latest/getting-started/configuration.html)\n\nMinimal pipeline config:\n\n```yaml\nshared_memory:\n  - name: input_frame\n    shape: [4]\n    dtype: float32\n    storage: cpu\n\n  - name: scaled_frame\n    shape: [4]\n    dtype: float32\n    storage: cpu\n\nkernels:\n  - name: scale_stage\n    kind: cpu.scale\n    input: input_frame\n    output: scaled_frame\n    parameters:\n      factor: 2.0\n```\n\nRun it from Python:\n\n```python\nimport numpy as np\n\nfrom shmpipeline import PipelineConfig, PipelineManager\n\nconfig = PipelineConfig.from_yaml(\"pipeline.yaml\")\nmanager = PipelineManager(config)\nmanager.build()\nmanager.start()\n\nmanager.get_stream(\"input_frame\").write(np.array([1, 2, 3, 4], dtype=np.float32))\nresult = manager.get_stream(\"scaled_frame\").read_new(timeout=2.0)\nprint(result)\n\nmanager.stop()\nmanager.shutdown()\n```\n\nIf you prefer the CLI for the same pipeline:\n\n```bash\nshmpipeline validate pipeline.yaml\nshmpipeline describe pipeline.yaml\nshmpipeline run pipeline.yaml --duration 5.0\n```\n\n## Popular Example Pages\n\n- [Affine transformation](https://shmpipeline.readthedocs.io/en/latest/examples/affine-transformation.html)\n- [Basic AO system](https://shmpipeline.readthedocs.io/en/latest/examples/basic-ao-system.html)\n- [Observatory AO system](https://shmpipeline.readthedocs.io/en/latest/examples/observatory-ao-system.html)\n- [Source and sink plugins](https://shmpipeline.readthedocs.io/en/latest/examples/source-sink-plugins.html)\n\n## More Documentation\n\nStart with these in roughly the order most users need them:\n\n1. [Configuration guide](https://shmpipeline.readthedocs.io/en/latest/getting-started/configuration.html) for the full YAML model, parameters, and shared-memory definitions.\n2. [Worked examples](https://shmpipeline.readthedocs.io/en/latest/examples/index.html) for complete CPU, GPU, custom-operation, AO, and plugin-backed pipelines.\n3. [CLI guide](https://shmpipeline.readthedocs.io/en/latest/guides/cli.html) for `validate`, `describe`, `run`, and `serve`.\n4. [GUI guide](https://shmpipeline.readthedocs.io/en/latest/guides/gui.html) for editing configs, validating locally, inspecting graphs, and launching viewers.\n5. [Runtime guide](https://shmpipeline.readthedocs.io/en/latest/guides/runtime.html) for lifecycle, metrics, worker health, and synthetic inputs.\n6. [Extensions guide](https://shmpipeline.readthedocs.io/en/latest/guides/extensions.html) for custom kernels, sources, and sinks.\n7. [Control plane guide](https://shmpipeline.readthedocs.io/en/latest/guides/control-plane.html) for remote management, SSE events, and the Python client.\n8. [API reference](https://shmpipeline.readthedocs.io/en/latest/reference/index.html), [core API](https://shmpipeline.readthedocs.io/en/latest/reference/core.html), and [kernel catalog](https://shmpipeline.readthedocs.io/en/latest/reference/kernels.html) for the detailed reference surface.\n9. [Troubleshooting](https://shmpipeline.readthedocs.io/en/latest/guides/troubleshooting.html) for common setup and runtime issues.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjacotay7%2Fshmpipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjacotay7%2Fshmpipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjacotay7%2Fshmpipeline/lists"}