{"id":15650989,"url":"https://github.com/patwie/tf_zmq","last_synced_at":"2025-04-30T18:21:35.966Z","repository":{"id":81506128,"uuid":"91605739","full_name":"PatWie/tf_zmq","owner":"PatWie","description":"TensorFlow operation for reading data from sockets (lightweight c++)","archived":false,"fork":false,"pushed_at":"2018-05-08T11:03:52.000Z","size":24,"stargazers_count":35,"open_issues_count":0,"forks_count":10,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-30T18:43:55.786Z","etag":null,"topics":["cpp11","data-reader","deep-learning","tensorflow","tensorflow-op","tensorflow-operation","tensorflow-ops"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PatWie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-17T18:02:36.000Z","updated_at":"2024-05-23T05:46:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"d8249553-38a4-46b2-8ee8-f66519dd3aa7","html_url":"https://github.com/PatWie/tf_zmq","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PatWie%2Ftf_zmq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PatWie%2Ftf_zmq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PatWie%2Ftf_zmq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PatWie%2Ftf_zmq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PatWie","download_url":"https://codeload.github.com/PatWie/tf_zmq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251758243,"owners_count":21639003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp11","data-reader","deep-learning","tensorflow","tensorflow-op","tensorflow-operation","tensorflow-ops"],"created_at":"2024-10-03T12:36:29.206Z","updated_at":"2025-04-30T18:21:35.943Z","avatar_url":"https://github.com/PatWie.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"ZeroMQ for TensorFlow\n======================\n\n[![Build Status](https://travis-ci.com/PatWie/tf_zmq.svg?token=EwCHGzKTTUEGS2rahMRy\u0026branch=master)](https://travis-ci.com/PatWie/tf_zmq)\n\nThis library contains methods to send data from your C++ application **directly** into the TensorFlow graph **without** any large dependency. Unfortunately, generating data within a C++ application for TensorFlow is not envisaged and communicated as not requested for the TF core. Even the discussion about the redesign of the input pipeline seems to converge to put all data-processing into the graph. \n\nThis is hilarious as most solutions for interesting problems are not based on a mindless iteration over a set of images. This small library helps to send data directly from your favorite C++ application (game-engine, renderer, emulator, [put here what you want]) to TensorFlow. Reading data is as simple as (see [read_tf.py](https://github.com/patwie/tf_zmq/blob/master/read_tf.py)):\n\n```python\nimport tensorflow as tf\nimport zmq_op\n\nimage, label = zmq_op.pull('ipc:///tmp/ipc-socket-0', [tf.float32, tf.int32])\nwith tf.Session() as sess:\n    print(sess.run([image, label]))\n```\n\nThe C++ code to send data is about 45 lines in [write.cpp](https://github.com/patwie/tf_zmq/blob/master/write.cpp). It uses [ZMQ](http://zeromq.org/) for an easy distributed messaging and [msgPack](http://msgpack.org/) for a fast serialization. You are free to change the shape of each tensor over time. If you want to receive the tensors in native python without TensorFlow dependencies, you can use [read_py.py](https://github.com/patwie/tf_zmq/blob/master/read_py.py). This library supports all combinations:\n\n\n| sender     | receiver                    |\n| ---------- | --------------------------- |\n| write.py   |  TF (read_tf.py)            |\n| write.cpp  |  TF (read_tf.py)            |\n| write.py   |  native python (read_py.py) |\n| write.cpp  |  native python (read_py.py) |\n| write.py   |  plain C++ (read.cpp)       |\n| write.cpp  |  plain C++ (read.cpp)       |\n\n\nInstall dependencies\n---------------------\n\nYou basically need two dependencies (see [.travis.yml](https://github.com/patwie/tf_zmq/blob/master/.travis.yml)).\n\n```bash\n# compile ZMQ library for c++\ncd /path/to/your_lib_folder\ngit clone https://github.com/zeromq/libzmq\ncd libzmq\n./autogen.sh\n./configure\n./configure --prefix=/path/to/your_lib_folder/libzmq/dist\nmake\nmake install\n# compile MSGPACK library for c++\ncd ../\ngit clone https://github.com/msgpack/msgpack-c\ncd msgpack-c\nmkdir build\ncd build\ncmake .. -DCMAKE_INSTALL_PREFIX:PATH=/path/to/your_lib_folder/msgpack-c/dist\nmake\nmake install\n```\n\nCompile TensorFlow op and example\n----------------------\n```bash\nexport PKG_CONFIG_PATH=/path/to/your_lib_folder/libzmq/dist/lib/pkgconfig/:$PKG_CONFIG_PATH\nexport PKG_CONFIG_PATH=/path/to/your_lib_folder/msgpack-c/dist/lib/pkgconfig/:$PKG_CONFIG_PATH\ngit clone https://github.com/PatWie/tf_zmq.git\ncd tf_zmq\n./compile.sh\n```\n\n\nrelated projects:\n-------------\n\nThere are 2 alternative attempts for solving this problem:\n- [Tensorpack](https://github.com/ppwwyyxx/tensorpack): This solution requires a python script to send data (it is based on TensorProto protobuf (currently, symbols for this are not exported by TF))\n- [pull-request TF](https://github.com/tensorflow/tensorflow/pull/8728): Here you need to specify the shape dimensions in advance (also pure-python because the TensorProto dependency)\n\nYou probably do not want to link a game engine or another large codebase with the TensorFlow core using `bazel`, given compiling TensorFlow alone can be a nightmare and is only recommended for adventurer.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatwie%2Ftf_zmq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpatwie%2Ftf_zmq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatwie%2Ftf_zmq/lists"}