{"id":13737627,"url":"https://github.com/jeffbass/imagehub","last_synced_at":"2025-04-12T15:22:18.300Z","repository":{"id":113357772,"uuid":"166520487","full_name":"jeffbass/imagehub","owner":"jeffbass","description":"Receive and save images from multiple Raspberry Pi's","archived":false,"fork":false,"pushed_at":"2024-07-24T00:04:53.000Z","size":1074,"stargazers_count":24,"open_issues_count":0,"forks_count":16,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-26T09:51:19.258Z","etag":null,"topics":["opencv-python","python","pyzmq","raspberry-pi","zmq"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jeffbass.png","metadata":{"files":{"readme":"README.rst","changelog":"HISTORY.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-19T07:46:10.000Z","updated_at":"2025-03-17T06:04:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"149f2d3c-6876-4ef8-b4f6-ff0f4d868735","html_url":"https://github.com/jeffbass/imagehub","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffbass%2Fimagehub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffbass%2Fimagehub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffbass%2Fimagehub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffbass%2Fimagehub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jeffbass","download_url":"https://codeload.github.com/jeffbass/imagehub/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248586531,"owners_count":21129058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["opencv-python","python","pyzmq","raspberry-pi","zmq"],"created_at":"2024-08-03T03:01:55.456Z","updated_at":"2025-04-12T15:22:18.278Z","avatar_url":"https://github.com/jeffbass.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"============================================================\nimagehub: Receive \u0026 save images from multiple Raspberry Pi's\n============================================================\n\nIntroduction\n============\n\n**imagehub** receives and stores images and event messages from multiple\nsources simultaneously. The sources are Raspberry Pi and other computers\nrunning **imaganode** to capture and send images and event messages.\n\nHere is a pair of images shown with their corresponding log entries. These were\ncaptured by a Raspberry Pi PiCamera and an infrared floodlight:\n\n.. image:: docs/images/coyote-events.png\n\nThe log shows the coyote motion events (in **bold**) from \"Barn\". It also shows\nevents from 2 other Raspberry Pi computers that were sending at the same time,\n\"WaterMeter\" and \"BackDeck\". Motion events create camera images like the\n2 shown above. Temperature events are gathered by sensors attached to the\nRaspberry Pi computers' GPIO pins. This log and these images were saved by\nthe imagehub program running on a Ubuntu computer that was receiving events and\nimages from a dozen imagenodes running on Raspberry Pi computers. The log\nsnippet shows the event messages from 3 of those imagenodes (WaterMeter,\nBackDeck and Barn) during an hour around midnight.\n\n.. contents::\n\nOverview\n========\n\n**imagehub** is the \"receive and store\" part of a distributed computer vision\npipeline that is run on multiple computers. Multiple Raspberry Pi\n(and other) computers run **imagenode** to capture images, detect motion, light,\ntemperature values, etc. **Imagenode** then sends event messages and selected\nimages to **imagehub**, which files the events and images for later\nanalysis.  My typical setup has 8 to 12 sending computers for each **imagehub**.\n\nBy design, **imagehub** is a very simple program. It does 2 things:\n\n1. It receives images and stores them.\n2. It receives event messages and logs them.\n\nIt does this from multiple sources simultaneously. The sources are typically a\nbunch of Raspberry Pi computers with PiCameras and temperature sensors. Keeping\n**imagehub** simple allows it to be fast enough to reliably store data from\nmultiple sources. Analysis of images and responses to queries\nare handled by other programs. See `Using imagenode in distributed computer vision projects \u003chttps://github.com/jeffbass/imagenode/blob/master/docs/imagenode-uses.rst\u003e`_\nfor a more detailed explanation of the overall project design. See the\n`Yin Yang Ranch project \u003chttps://github.com/jeffbass/yin-yang-ranch\u003e`_\nfor more details about the architecture of the\n**imagenode** \u003c--\u003e **imageZMQ** \u003c--\u003e **imagehub** system.\n\n\nimagehub Features\n=================\n\n- Receives and save images from multiple Raspberry Pi's simultaneously.\n- Receives and logs event messages from multiple RPi's simultaneously.\n- Uses threading for image writing to enhance responsiveness.\n- Threading can be replaced with multiprocessing with minimal code changes.\n\nDependencies and Installation\n=============================\n\n**imagehub** has been tested with:\n\n- Python 3.5, 3.6 and 3.7\n- OpenCV 3.3 and 4.0+\n- PyZMQ 16.0+\n- imageZMQ 1.1.1\n- Recent and current versions of MacOS, Raspberry Pi OS and Ubuntu\n\n  - NOT yet tested with Raspberry Pi OS Bullseye. Waiting for a production\n    replacement for the Python PiCamera module to update imageZMQ and imagehub.\n    I expect to test imagehub with Raspberry Pi OS Bullseye after imageZMQ and\n    imagehub have been tested with Bullseye.\n  - Not tested with Windows. I don't have a computer running Windows.\n    I have heard from some users that it is being used with Windows\n    successfully.\n\n\n**imagehub** uses **imageZMQ** to receive event messages and images that are\ncaptured and sent by **imagenode**. You will need to install and test both\n**imageZMQ** and **imagenode** before using **imagehub**.\nThe instructions for installing and testing **imageZMQ** are in the\n`imageZMQ GitHub repository \u003chttps://github.com/jeffbass/imagezmq.git\u003e`_.\nThe instructions for installing and testing **imagenode** are in the\n`imagenode GitHub repository \u003chttps://github.com/jeffbass/imagenode.git\u003e`_.\n\n**imagehub** is still in early development, so it is not yet in PyPI. Get it by\ncloning the GitHub repository::\n\n    git clone https://github.com/jeffbass/imagehub.git\n\nOnce you have cloned **imagehub** to a directory on your local machine,\nyou can run the tests using the instructions below. The instructions assume you\nhave cloned both **imagehub** to the user home directory. It\nis also important that you have successfully run all the tests for **imageZMQ**\nand for **imagenode**. The recommended testing arrangement is to run **imagehub**\non the same Mac (or other display computer) that you used to run the\n``imagezmq/tests/timing_receive_jpg_buf.py`` program when you tested **imagenode**.\n\nRunning the Tests\n=================\n\n**imagehub** should be tested after you have tested **imagenode**, because you\nwill be using **imagenode** to send test images and event messages to\n**imagehub**.\n\nBoth **imagehub** and **imagenode** use **imageZMQ** for sending and receiving\nimages and event messages. The **imageZMQ** package is pip installable. It is\nlikely that you already have it installed from your tests of **imagenode**. If\nnot, it should be pip installed in a virtual environment. For example,\nmy virtual environment is named **py3cv3**.\n\nTo install **imageZMQ** using pip:\n\n.. code-block:: bash\n\n    workon py3cv3  # use your own virtual environment name\n    pip install imagezmq\n\n\nTest **imagehub** in the same virtualenv that you installed **imagenZMQ** in.\nFor **imageZMQ** and **imagenode** testing, my virtualenv is called ``py3cv3``.\n\nTo test **imagehub**, you will use the same setup as Test 2 for **imagenode**.\nYou will run **imagenode** on a Raspberry Pi with a PiCamera, just as you did for\n**imagenode** Test 2. You will run **imagehub** on the same Mac (or other display\ncomputer) that you used to display the **imagenode** test images.\n\nDirectory Structure for running the imagehub tests\n--------------------------------------------------\nNeither **imagehub** or **imagenode** are far enough along in their development\nto be pip installable. So they should both be git-cloned to the computers that\nthey will each be running on. I recommend doing all testing in the user home\ndirectory. Here is a simplified directory layout for the computer that will be\nrunning **imagehub**::\n\n  ~ # user home directory of the computer running imagehub\n  +--- imagehub.yaml  # copied from imagenode/imagenode.yaml in this repository\n  |\n  +--- imagehub    # the git-cloned directory for imagehub\n  |    +--- sub directories include docs, imagehub, tests\n  |\n  +--- imagehub_data   # this directory will be created by imagehub\n       +--- images      # images will be saved here\n       +--- logs        # logs containing event messages will be saved here\n\nThe **imagehub** directory arrangement, including docs, **imagehub** code,\ntests, etc. is a common software development directory arrangement on GitHub.\nUsing ``git clone`` from your user home directory on your **imagehub** computer\n(either on a Mac, a RPi or other Linux computer) will put the **imagehub**\ndirectories in the right place for testing. When the **imagehub** program runs,\nit creates a directory (``imagehub_data``) with 2 subdirectories (``images`` and\n``logs``) to store the images and event messages it receives from **imagenode**\nrunning on one or more RPi's or other computers. Running **imagenode** requires\na settings file named ``imagehub.yaml``. To run the tests, copy the example\n``imagehub.yaml`` file from the ``imagehub`` directory to your home directory.\nThe ``imagehub.yaml`` settings file is expected to be in your home directory,\nbut you can specify another directory path using the --path optional argument.\nI recommend putting the ``imagehub.yaml`` file in your home directory for\ntesting. You can move the ``imagehub.yaml`` file to a different directory after\nyou have completed the tests.\n\nTest 1: Running **imagehub** with a single **imagenode** sender\n---------------------------------------------------------------\n**The first test** uses a single Raspberry Pi computer running **imagenode**\nwith **imagehub** running on Mac or other display computer.\nIt tests that the **imagehub** software is installed correctly and that the\n``imagehub.yaml`` file has been copied and edited in a way that works.\n\nTest 2: Running **imagehub** with 2 **imagenode** senders simultaneously\n------------------------------------------------------------------------\n**The second test** runs **imagenode** on 2 Raspberry Pi computers,\nwith **imagehub** receiving images and event messages from both RPi's at\nthe same time. The event logs and image files will record what is sent\nfrom both RPi's.\n\nFurther details of running the tests are `here \u003cdocs/testing.rst\u003e`_.\n\nRunning **imagehub** in production\n==================================\nRunning the test programs requires that you leave a terminal window open, which\nis helpful for testing, but not for production runs. I use systemctl / systemd\nto start **imagehub** in production. I have provided an example\n``imagehub.service`` unit configuration file that shows how I start **imagehub**\nfor the production programs observing my small farm. I have found the systemctl\n/ systemd system to be best way to start / stop / restart and check status of\n**imagehub** over several years of testing. For those who prefer using a shell\nscript to start **imagehub**, I have included an example ``imagehub.sh``. It is\nimportant to run **imagehub** in the right virtualenv in production, regardless\nof your choice of program startup tools.\n\nIn production, you would want to set the test options used to print settings\nto ``False``; they are only helpful during testing. All errors and **imagenode**\nevent messages are saved in the file ``imagehub.log`` which is located in the\ndirectory you specify in the ``imagenode.yaml`` setting ``data_directory``:\n\n.. code-block:: yaml\n\n    data_directory: imagehub_data\n\nThe ``imagehub.yaml`` settings file is expected to be in the users home\ndirectory by default. You can specify the path to a different directory\ncontaining ``imagehub.yaml`` by using the optional argument ``--path``:\n\n.. code-block:: bash\n\n    workon py3cv3  # use your own virtual environment name\n    python3 imagenode.py --path directory_name  # directory holding imagehub.yaml\n\nAdditional Documentation\n========================\n- `How imagehub works \u003cdocs/imagehub-details.rst\u003e`_.\n- `The imagehub Settings and the imagehub.yaml file \u003cdocs/settings-yaml.rst\u003e`_.\n- `Version History and Changelog \u003cHISTORY.md\u003e`_.\n- `Research and Development Roadmap \u003cdocs/research-roadmap.rst\u003e`_.\n- `The imageZMQ classes that allow transfer of images \u003chttps://github.com/jeffbass/imagezmq\u003e`_.\n- `The imagenode program that captures and sends images \u003chttps://github.com/jeffbass/imagenode\u003e`_.\n- `The larger farm automation / computer vision project \u003chttps://github.com/jeffbass/yin-yang-ranch\u003e`_.\n  This project shows the overall system architecture. It also contains\n  links to my **PyCon 2020** talk video and slides explaining the project.\n\nContributing\n============\n**imagehub** is in early development and testing. I welcome open issues and\npull requests, but because the code is still rapidly evolving, it is best\nto open an issue with some discussion before submitting any pull requests or\ncode changes.  We can exchange ideas about your potential pull request and how\nto best incorporate and test your code.\n\nAn Excellent Alternative Design for an imagehub \u0026 librarian combination\n=======================================================================\nAn ``imagenode`` \u0026 ``imagehub`` user and code contributor @sbkirby has designed\na completely different approach to building an imagehub and librarian\ncombination using a broad mix of tools in addition to Python including Node-Red,\nMQTT, MariaDB and OpenCV in Docker containers. He has posted it in this\n`Github repository \u003chttps://github.com/sbkirby/imagehub-librarian\u003e`_.\nI like his approach a lot, although I'm still working on a mostly Python\napproach to my own librarian.\n\nAcknowledgments\n===============\n- **ZeroMQ** is a great messaging library with great documentation\n  at `ZeroMQ.org \u003chttp://zeromq.org/\u003e`_.\n- **PyZMQ** serialization examples provided a starting point for **imageZMQ**.\n  See the\n  `PyZMQ documentation \u003chttps://pyzmq.readthedocs.io/en/latest/index.html\u003e`_.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeffbass%2Fimagehub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjeffbass%2Fimagehub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeffbass%2Fimagehub/lists"}