{"id":19401113,"url":"https://github.com/google-research/libsoftwaresync","last_synced_at":"2025-08-15T17:14:34.727Z","repository":{"id":41241900,"uuid":"227694672","full_name":"google-research/libsoftwaresync","owner":"google-research","description":":camera: :camera: :camera: :camera: :camera: Wireless software synchronization of multiple distributed smartphone cameras.","archived":false,"fork":false,"pushed_at":"2021-08-05T20:08:09.000Z","size":142,"stargazers_count":153,"open_issues_count":8,"forks_count":36,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-04-24T07:42:20.202Z","etag":null,"topics":["camera","phase-alignment","smartphone","time-synchronization"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1812.09366","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-12T20:54:23.000Z","updated_at":"2025-04-22T09:07:12.000Z","dependencies_parsed_at":"2022-08-10T01:43:16.733Z","dependency_job_id":null,"html_url":"https://github.com/google-research/libsoftwaresync","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/google-research/libsoftwaresync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flibsoftwaresync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flibsoftwaresync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flibsoftwaresync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flibsoftwaresync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/libsoftwaresync/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flibsoftwaresync/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270602456,"owners_count":24614260,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-15T02:00:12.559Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["camera","phase-alignment","smartphone","time-synchronization"],"created_at":"2024-11-10T11:17:14.661Z","updated_at":"2025-08-15T17:14:34.699Z","avatar_url":"https://github.com/google-research.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wireless Software Synchronization of Multiple Distributed Cameras\n\nReference code for the paper\n[Wireless Software Synchronization of Multiple Distributed Cameras](https://arxiv.org/abs/1812.09366).\n_Sameer Ansari, Neal Wadhwa, Rahul Garg, Jiawen Chen_, ICCP 2019.\n\nIf you use this code, please cite our paper:\n\n```\n@article{AnsariSoftwareSyncICCP2019,\n  author  = {Ansari, Sameer and Wadhwa, Neal and Garg, Rahul and Chen, Jiawen},\n  title   = {Wireless Software Synchronization of Multiple Distributed Cameras},\n  journal = {ICCP},\n  year    = {2019},\n}\n```\n\n_This is not an officially supported Google product._\n\n![Five smartphones synchronously capture a balloon filled with red water being popped.](https://i.imgur.com/rCkC5jW.gif)\n_Five smartphones synchronously capture a balloon filled with red water being popped to within 250 μs timing accuracy._\n\n## Android App to Capture Synchronized Images\n\nThe app has been tested on the Google Pixel 2, 3, and 4.\nIt may work on other Android phones with minor changes.\n\nNote: On Pixel 1 devices the viewfinder frame rate drops after a couple\ncaptures, which will likely cause time synchronization to be much\nlower in accuracy. This may be due to thermal throttling.\nDisabling saving to JPEG or lowering the frame rate may help.\n\n### Installation instructions:\n\n1.  Download [Android Studio](https://developer.android.com/studio). When you\n    install it, make sure to also install the Android SDK API 27.\n2.  Click \"Open an existing Android Studio project\". Select the \"CaptureSync\"\n    directory.\n3.  There will be a pop-up with the title \"Gradle Sync\" complaining about a\n    missing file called gradle-wrapper.properties. Click ok to recreate the\n    Gradle wrapper.\n4.  Plug in your Pixel smartphone. You will need to enable USB debugging. See\n    https://developer.android.com/studio/debug/dev-options for further\n    instructions.\n5.  Go to the \"Run\" menu at the top and click \"Run 'app'\" to compile and install\n    the app.\n\nNote: By default, the app will likely start in client mode, with no UI options.\n\n#### Setting up the Leader device\n\n1.  On the system pulldown menu of the leader device, disable WiFi.\n2.  [Start a hotspot](https://support.google.com/android/answer/9059108).\n3.  After this, opening the app on the leader device should show UI options, as\n    well as which clients are connected.\n\n#### Setting up the Client(s) device\n\n1.  Enable WiFi and connect to the leader's hotspot.\n2.  As client devices on the network start up, they will sync up with the\n    leader, which will show up on both the leader and client UIs.\n3.  (Optional) Go to wifi preferences and disable \"Turn on Wi-Fi automatically\"\n    and \"Connect to open networks\", this will keep devices from automatically\n    disconnecting from a hotspot without internet.\n\n#### Capturing images\n\n1.  (Optional) Press the phase align button to have each device synchronize\n    their phase, the phase error will show in real-time.\n2.  (Optional) Move the exposure and sensitivity slider on the leader device to\n    manually set 2A values.\n3.  Press the `Capture Still` button to request a synchronized image slightly in\n    the future on all devices.\n\nThis will save to internal storage, as well as show up under the Pictures\ndirectory in the photo gallery.\n\nNote: Fine-tuning the phase configuration JSON parameters in the `raw` resources\ndirectory will let you trade alignment-time for phase alignment accuracy.\n\nNote: AWB is used for simplicity, but could also be synchronized with devices.\n\n### Information about saved data\n\nSynchronized images are saved to the external files directory for this app,\nwhich is:\n\n```\n/storage/emulated/0/Android/data/com.googleresearch.capturesync/files\n```\n\nA JPEG version of the image will also populate in the photo gallery under the\n`Pictures` subdirectory under `Settings -\u003e Device Folders`.\n\nPulling data from individual phones using:\n\n```\nadb pull /storage/emulated/0/Android/data/com.googleresearch.capturesync/files /tmp/outputdir\n```\n\nThe images are also stored as a raw YUV file (in\n[packed NV21 format](https://wiki.videolan.org/YUV)) and a metadata file which\ncan be converted to PNG or JPG using the Python script in the `scripts/`\ndirectory.\n\n#### Example Workflow\n\n1. User sets up all devices on the same hotspot WiFi network of leader device.\n2. User starts app on all devices, uses exposure sliders and presses the\n`Phase Align` button on the leader device.\n3. User presses capture button on the leader device to collect captures.\n4. If JPEG is enabled (default) the user can verify captures by going to the\n`Pictures` photo directory on their phone through Google Photos or similar.\n5. After a capture session, the user pulls the data from each phone to the local\nmachine using `adb pull`.\n6. (Optional)The python script is used to convert the raw images using:\n```\npython3 yuv2rgb.py img_\u003ctimestamp\u003e.nv21 nv21_metadata_\u003ctimestamp\u003e.txt\nout.\u003cpng|jpg\u003e.\n```\n\n## How Networking and Communications work\n\nNote: Algorithm specifics can be found in our paper linked at the top.\n\nLeader and clients use heartbeats to connect with one another and keep track of\nstate. Simple NTP is used for clock synchronization. That, phase alignment and\n2A is used to make phones capture the same type of image as the same time.\nCapturing is done by sending a trigger time to all devices which will\nindependently capture at that time.\n\nAll of this requires communication. One component of this library is to provide\na method for sending messages (RPCs) between the leader device and client\ndevices, to allow for  synchronization as well as capture triggering, AWB,\nstate etc.\n\nThe network uses wifi with UDP messages for communication. The leader IP is\ndetermined automatically by client devices.\n\nA message is sent as an RPC byte sequence consisting of an integer method ID\n(defined in\n[`SyncConstants.java`](app/src/main/java/com/googleresearch/capturesync/softwaresync/SyncConstants.java)\nand the string message payload. (defined in\n[`SoftwareSyncBase.java`](app/src/main/java/com/googleresearch/capturesync/softwaresync/SoftwareSyncBase.java)\n`sendRpc()`)\n\nNote: This app has the leader set up a hotspot, through this client devices can\nautomatically determine the leader IP address from the connection, however one\ncould manually configure IP address with a different network configuration, such\nas using a router that all the phones connect to.\n\n\n### Capture\n\nThe leader sends a `METHOD_SET_TRIGGER_TIME` RPC (Method id located in\n[`SoftwareSyncController.java`](app/src/main/java/com/googleresearch/capturesync/SoftwareSyncController.java)\n) to all the clients containing a requested capture\nsynchronized timestamp far enough in the future to account for potential network\nlatency between devices. In practice network latency between devices is ~100ms\nor less, however the latency may be more or less depending on what devices or\nnetwork configuration is used.\n\nNote: In this case the future is 500ms, giving plenty of time for network\nlatency.\n\nEach client and leader receives the future timestamp and `CameraController.java`\nchecks the timestamp of each frame as it comes in and pulls the closest frame at\nor past the desired timestamp and saves it to disk. One advantage of this method\nis that if any delays happen in capturing, the synchronized capture timestamp\nwill show that the time offset between images without requiring looking at the\nimages.\n\nNote: Zero-shutter-lag capture is possible if each device is capable of storing\nframes in a ring buffer. Then when a desired current/past capture timestamp is\nprovided each device can check in the ring buffer for the closest frame\ntimestamp and save that one.\n\n### Heartbeat\n\nA leader listens for a heartbeat from any client, to determine if a client\nexists and whether starting the synchronization with that client is necessary.\nWhen it gets a heartbeat from a client that is not synchronized, it initiates an\nNTP handshake with the client to determine the clock offsets between the two\ndevices\n\nA client continuously sends out `METHOD_HEARTBEAT` RPC to the leader with it's\ncurrent boolean state for if it's already synchronized with the leader.\n\nA leader received `METHOD_HEARTBEAT` and responds with a `METHOD_HEARTBEAT_ACK`\nto the client. The leader uses this to keep track of a list of clients using the\n`ClientInfo` object for each client, which will also include sync information.\n\nThe client waits for a `METHOD_OFFSET_UPDATE` from the leader which contains the\ntime offset needed to get to a synchronized clock domain with the leader, after\nwhich it's heartbeat messages will show that it is synced to the leader.\n\nWhenever a client gets desynchronized, the heartbeats will notify the leader of\nit and they will re-initiate synchronization. Through this mechanism automated\nclock synchronization and maintenance is achieved.\n\n### Simple NTP Handshake\n\nThe\n[`SimpleNetworkTimeProtocol.java`](app/src/main/java/com/googleresearch/capturesync/softwaresync/SimpleNetworkTimeProtocol.java)\nis used to perform an NTP handshake between the leader and client. The local\ntime domain of the devices is used, using the\n[`Ticker.java`](app/src/main/java/com/googleresearch/capturesync/softwaresync/Ticker.java)\nmethod for getting local nanosecond time.\n\nAn NTP handshake consists of the leader sending a message containing the current\nleader timestamp t0. The client receives and appends it's receiving local\ntimestamp t1, as well as the timestamp it sends a return message to the leader\nt2. The leader receives this at timestamp t3, and using these 4 times estimates\nthe clock offset between the two devices, accounting for network latency.\n\nThis result is encapsulated in\n[`SntpOffsetResponse.java`](app/src/main/java/com/googleresearch/capturesync/softwaresync/SntpOffsetResponse.java)\nwhich also contains the hard upper bound timing error on the offset. In practice\nthe timing error is an order of magnitude smaller since wifi network\ncommunication is mostly symmetric with the bias accounted for by choosing the\nsmallest sample(s).\n\nMore information can be found in our paper on this topic.\n\n### Phase Alignment\n\nThe leader sends out a `METHOD_DO_PHASE_ALIGN` RPC (Method id located in\n`SoftwareSyncController.java`) to all the clients whenever the Align button is\npressed. Each client on receipt then starts a phase alignment process (handled\nby `PhaseAlignController.java`) which may take a couple frames to settle.\n\nNote: The leader could instead send its current phase to all devices, and the\ndevices could align to that, reducing the total potential error. For simplicity\nthis app uses a hard-coded goal phase.\n\n### Exposure / White Balance / Focus\n\nFor simplicity, this app uses manual exposure, hard-coded white balance, and\nauto-focus. The leader uses UI sliders to set exposure and sensitivity, which\nautomatically sends out a `METHOD_SET_2A` RPC (Method id located in\n[`SoftwareSyncController.java`](app/src/main/java/com/googleresearch/capturesync/SoftwareSyncController.java)\n) to all the clients, which update their 2A as\nwell. Technically 2A is a misnomer here as it is only setting exposure and\nsensitivity, not white balance.\n\nIt is possible to use auto exposure/sensitivity and white balance, and have the\nleader lock and send the current 2A using the same RPC mechanism to other\ndevices which can then set theirs manually to the same.\n\nNote: One could try synchronizing focus values as well, though in practice we\nfound the values were not accurate enough to provide sharp focus across devices.\nHence we keep auto-focus.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Flibsoftwaresync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Flibsoftwaresync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Flibsoftwaresync/lists"}