{"id":18499135,"url":"https://github.com/tensorpack/zmq_ops","last_synced_at":"2025-04-09T01:31:32.728Z","repository":{"id":152196679,"uuid":"116487610","full_name":"tensorpack/zmq_ops","owner":"tensorpack","description":"TensorFlow ZMQ ops.","archived":false,"fork":false,"pushed_at":"2021-01-31T11:11:11.000Z","size":636,"stargazers_count":42,"open_issues_count":3,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-23T20:06:01.956Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tensorpack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-06T14:04:48.000Z","updated_at":"2023-12-01T18:50:32.000Z","dependencies_parsed_at":null,"dependency_job_id":"10b9db0d-36bb-4a8e-9544-ff7c74c2dd83","html_url":"https://github.com/tensorpack/zmq_ops","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorpack%2Fzmq_ops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorpack%2Fzmq_ops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorpack%2Fzmq_ops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorpack%2Fzmq_ops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tensorpack","download_url":"https://codeload.github.com/tensorpack/zmq_ops/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247956542,"owners_count":21024568,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T13:44:47.818Z","updated_at":"2025-04-09T01:31:32.723Z","avatar_url":"https://github.com/tensorpack.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## TensorFlow ZMQ Op\n\nA convenient way to receive data from other processes. This small library can:\n\n+ Send a list of numpy arrays from python; serialization is written in C++ for efficiency.\n  + One copy in merging all the buffers; One copy in pybind11 overhead (TODO); One copy in ZMQ send.\n+ Receive a list of tensors from tensorflow;\n  + One copy in ZMQ recv; One copy to split the buffer into tensors.\n  + The op is stateful and safe to be evaluated multiple times in one `sess.run` call.\n+ Serialization is in a custom protocol for efficiency;\n\n## Why:\n\nSometimes for complicated large-scale tasks you would really want data processing to be separate from TensorFlow.\nHowever in TensorFlow there is no good way to receive data from other processes.\n\n## Build:\n\nRequire gcc\u003e=5.3, tensorflow\u003e=1.4, zeromq\u003e=4.\n\nRequire the `zmq.hpp` header from [cppzmq](https://github.com/zeromq/cppzmq) at\nyour compiler's include path, or under the `src` directory.\n\nAdd `/path/to/git/clone/zmq_ops` to `PYTHONPATH` to be able to import it.\nOr use `pip install .` to install it.\n\nOps will be compiled the first time it gets imported.\nNote that it usually requires recompilation after a TensorFlow reinstallation.\n\n## Use:\n\nSee `benchmark.py` for usage.\n\nOn my machine this script can achieve about 1.3GB/s throughput. Equivalent to about 2.3k float32 (or 9.2k uint8) imagenet images per second.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftensorpack%2Fzmq_ops","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftensorpack%2Fzmq_ops","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftensorpack%2Fzmq_ops/lists"}