{"id":22515365,"url":"https://github.com/scijava/ipc-shm-demo","last_synced_at":"2025-10-28T22:10:59.854Z","repository":{"id":139654894,"uuid":"599582053","full_name":"scijava/ipc-shm-demo","owner":"scijava","description":"IPC playground","archived":false,"fork":false,"pushed_at":"2023-02-15T21:16:46.000Z","size":16,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-02T03:41:23.504Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scijava.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-09T12:56:28.000Z","updated_at":"2023-02-10T12:33:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"f91ecbad-8d36-44a6-8e4f-cec5593cd219","html_url":"https://github.com/scijava/ipc-shm-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scijava%2Fipc-shm-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scijava%2Fipc-shm-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scijava%2Fipc-shm-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scijava%2Fipc-shm-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scijava","download_url":"https://codeload.github.com/scijava/ipc-shm-demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245955328,"owners_count":20699891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-07T03:30:10.755Z","updated_at":"2025-10-28T22:10:54.820Z","avatar_url":"https://github.com/scijava.png","language":"Java","readme":"# Interprocess shared memory proof of concept\n\n## Demo\n\nThis repository contains a working Python-first demo of interprocess shared\nmemory with no backing file on disk. Here are the steps to try it:\n\n1. Create and activate a conda environment with pyimagej installed.\n\n2. Run this command:\n   ```shell\n   python -i src/main/python/open-with-pyimagej-into-shm.py /path/to/data/image.ome.tif\n   ```\n   Where `/path/to/data/image.ome.tif` is the path an image file, which will\n   be opened using [SCIFIO](https://scif.io/) into an ImageJ2 `Dataset`,\n   then copied into a NumPy ndarray backed by a new shared memory buffer.\n\n   The `-i` is important to keep the process alive after the work finished,\n   and so that you can have fun playing around across multiple processes.\n\n3. In a separate console, run the command emitted by the first program. E.g.:\n   ```shell\n   python -i src/main/python/receive-numpy-array-from-shm.py psm_4b597b2c '(85, 33, 512, 768)' uint8\n   ```\n   Where `psm_4b597b2c` is the name of the shared memory buffer,\n   `'(85, 33, 512, 768)'` is the shape of the image stored in that buffer,\n   and `uint8` is the image's dtype.\n\n   This second program will access the given shared memory, wrapping it up\n   the same way to provide an equivalent image backed by the same memory.\n   Changes to `arr_shared` in the first REPL will be reflected in `rai`\n   in the second REPL, and vice versa. Note that the respective indexing\n   orders of `arr_shared` (Python/NumPy) and `rai` (Java/ImgLib2) are\n   *backwards* from one another.\n\n## Planned components\n\n1. Structured data schema for exchange of typed input/output parameters\n   * this is the \"data model of Ops\" -- STANDS ALONE, LANGUAGE AGNOSTIC\n   * if written to XML, can be validated by an XSD\n   * if written to JSON, can be validated by a JSON-based schema validation mechanism\n   * if written to YAML, can be validated by a YAML-based schema validation mechanism\n   * analogous to takari.io's dialects\n   * we start with JSON, because it's easy to process in Python \u0026 JS \u0026 Java \u0026 Julia\n\n2. Named shared memory between processes -- LANGUAGE AND PROCESS AGNOSTIC\n   * Perfect for zero-copy sharing of ndarrays that fit in RAM\n   * Works across platforms\n     * POSIX has shm mechanism\n     * Windows has (Create|Open)FileHandle with `INVALID_FILE_POINTER` and non-null name\n   * Works across languages\n     * Python has `multiprocessing.shared_memory.SharedMemory`\n     * Java has LArray (needs a patch from Mark H)\n   * Works across ENVIRONMENTS\n     * Pythons in different conda environments still share ndarrays\n     * JVMs with different (class|module)paths still share ndarrays\n     * etc.\n   * We already have a working demo between Java ImgLib2 and Python numpy;\n     see [Demo](#demo) above.\n\n3. For Java-based apps: a process manager to exec processes from Java for Op execution\n   * Simple mode: each Op execution execs a process\n     * Feeds inputs in (1)'s JSON format to process's stdin\n     * Receives outputs in (1)'s JSON format from process's stdout\n     * Could also harvest status updates from process's stdout\n     * Later, could send I/O in fancier formats like Apache Arrow -- but for now, KISS\n   * Persistent mode: each environment has a persistent process,\n     accepting commands via JSON lines to stdin, and producing output via JSON\n     lines to stdout. In this way, we avoid the overhead of starting a new\n     process for every command execution, which is important if executing\n     commands thousands or millions of times in a loop from the other process.\n\n4. For Java-based apps: a Java library offering a conda environment manager layer\n   * Easily construct and update conda environments from Java code\n   * Separate GUI layer on top so users can do it from Java applications\n   * Coupled with (1) and (2), let's graphical Java applications execute\n     plugins in other heterogeneous environments! E.g. execute deep learning\n     models with divergent dependencies each in its own process from its own\n     conda environment.\n\n## Notes\n\n[numpy.memmap] - Create a memory-map to an array stored in a binary file on disk.\n\"This subclass of ndarray has some unpleasant interactions with some operations\"\n\"An alternative to using this subclass is to create the mmap object yourself\"\n\nMark K suggests looking into \"implementing Buffer Protocol in ImgLib2.\"\nhttps://numpy.org/doc/stable/reference/arrays.interface.html#the-array-interface-protocol\nWe don't need it explicitly in Java though, it turns out, with the plan above.\n\n## How to read .npy in Java\n\n* https://github.com/JetBrains-Research/npy\n* https://github.com/bioimage-io/model-runner-java/blob/main/src/main/java/io/bioimage/modelrunner/numpy/DecodeNumpy.java (not thoroughly tested)\n\n[numpy.memmap]: https://numpy.org/doc/stable/reference/generated/numpy.memmap.html\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscijava%2Fipc-shm-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscijava%2Fipc-shm-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscijava%2Fipc-shm-demo/lists"}