{"id":22382257,"url":"https://github.com/googleinterns/instant_motion_tracking","last_synced_at":"2025-03-26T19:40:28.313Z","repository":{"id":93894715,"uuid":"268877685","full_name":"googleinterns/instant_motion_tracking","owner":"googleinterns","description":null,"archived":false,"fork":false,"pushed_at":"2020-08-28T20:37:46.000Z","size":236645,"stargazers_count":3,"open_issues_count":2,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-02-01T01:41:25.423Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/googleinterns.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":"docs/code-of-conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-02T18:25:20.000Z","updated_at":"2021-01-30T21:53:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"e1ec993c-7107-40ab-8566-f5097dbf4a83","html_url":"https://github.com/googleinterns/instant_motion_tracking","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/googleinterns%2Finstant_motion_tracking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/googleinterns%2Finstant_motion_tracking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/googleinterns%2Finstant_motion_tracking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/googleinterns%2Finstant_motion_tracking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/googleinterns","download_url":"https://codeload.github.com/googleinterns/instant_motion_tracking/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245726698,"owners_count":20662542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-05T00:12:20.036Z","updated_at":"2025-03-26T19:40:28.279Z","avatar_url":"https://github.com/googleinterns.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Instant Motion Tracking\n\n**This is not an officially supported Google product.**\n\nThis repository contains the Instant Motion Tracking project\nusing MediaPipe.\n\nThis document focuses on the [below graph](#main-graph) that performs instant motion tracking.\n\nFor overall context on instant motion tracking, please read this [paper](https://arxiv.org/abs/1907.06796).\n\n![instant_motion_tracking_android_gif1](images/demo_1.gif) ![instant_motion_tracking_android_gif2](images/demo_2.gif)\n\n## Android\n\n[Source](mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking)\n\nTo build and install the app, the MediaPipe repository must first be cloned from Github:\n\n```bash\ngit clone https://github.com/google/mediapipe.git\n```\n\nOnce you have the MediaPipe repository, clone this project as well:\n\n```bash\ngit clone https://github.com/googleinterns/instant_motion_tracking.git\n```\n\nNext, run the following command to move the instantmotiontracking source graphs and calculators into the MediaPipe directory:\n\n```bash\nmv instant_motion_tracking/mediapipe/graphs/instantmotiontracking mediapipe/mediapipe/graphs/instantmotiontracking\n```\n\nIt is also necessary move the Android source application to Mediapipe:\n\n```bash\nmv instant_motion_tracking/mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking mediapipe/mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking\n```\n\nDue to BUILD dependency issues with related MediaPipe projects, you must change the visibility permissions of the object_tracking_3d calculators. Run the command below to copy the modified BUILD file to fix privacy issues between projects:\n\n```base\nmv instant_motion_tracking/installation/BUILD mediapipe/mediapipe/graphs/object_detection_3d/calculators/BUILD\n```\n\nNavigate to the mediapipe directory containing your WORKSPACE file and use the following commands to add Glide Image Loading dependencies to the Maven installation section: \n\n```bash\ncd mediapipe\n\nsed -i '/\"com.google.guava:guava:27.0.1-android\",/a \"com.github.bumptech.glide:glide:4.11.0\",' WORKSPACE \n\nsed -i '/\"com.github.bumptech.glide:glide:4.11.0\",/a \"com.github.bumptech.glide:gifdecoder:4.11.0\",' WORKSPACE \n\n# If you wish to avoid running the commands above (or run into issues), you can alternatively copy the following two dependencies to the 'artifacts' portion of maven_install:\n#\n# artifacts = [\n# \t...\n# \t\"com.github.bumptech.glide:glide:4.11.0\",\n# \t\"com.github.bumptech.glide:gifdecoder:4.11.0\",\n# ]\n```\n\nNow the application can be built via Bazel:\n\n```bash\nbazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking\n```\n\nOnce the app is built, install it on an Android device by transferring the apk via ADB:\n\n```bash\nadb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking/instantmotiontracking.apk\n```\n\n## Graph\n\nThe instant motion tracking [main graph](#main-graph) internally utilizes a [region tracking subgraph](#region-tracking-subgraph)\nin order to perform anchor tracking for each individual 3d render.\n\n### Main Graph\n\n[Source pbtxt file](mediapipe/graphs/instantmotiontracking/instantmotiontracking.pbtxt)\n\n![main-graph-png](images/main-graph.png)\n\n```bash\n# MediaPipe graph that performs region tracking and 3d object (AR sticker) rendering.\n\n# Images in/out of graph with sticker data and IMU information from device\ninput_stream: \"input_video\"\ninput_stream: \"sticker_sentinel\"\ninput_stream: \"sticker_proto_string\"\ninput_stream: \"imu_rotation_matrix\"\ninput_stream: \"gif_texture\"\noutput_stream: \"output_video\"\n\n# Converts sticker data into user data (rotations/scalings), render data, and\n# initial anchors.\nnode {\n  calculator: \"StickerManagerCalculator\"\n  input_stream: \"PROTO:sticker_proto_string\"\n  output_stream: \"ANCHORS:initial_anchor_data\"\n  output_stream: \"USER_ROTATIONS:user_rotation_data\"\n  output_stream: \"USER_SCALINGS:user_scaling_data\"\n  output_stream: \"RENDER_DATA:sticker_render_data\"\n}\n\n# Uses box tracking in order to create 'anchors' for associated 3d stickers.\nnode {\n  calculator: \"RegionTrackingSubgraph\"\n  input_stream: \"VIDEO:input_video\"\n  input_stream: \"SENTINEL:sticker_sentinel\"\n  input_stream: \"ANCHORS:initial_anchor_data\"\n  output_stream: \"ANCHORS:tracked_anchor_data\"\n}\n\n# Concatenates all transformations to generate model matrices for the OpenGL\n# animation overlay calculator.\nnode {\n  calculator: \"MatricesManagerCalculator\"\n  input_stream: \"ANCHORS:tracked_anchor_data\"\n  input_stream: \"IMU_ROTATION:imu_rotation_matrix\"\n  input_stream: \"USER_ROTATIONS:user_rotation_data\"\n  input_stream: \"USER_SCALINGS:user_scaling_data\"\n  input_stream: \"RENDER_DATA:sticker_render_data\"\n  output_stream: \"MATRICES:0:gif_matrices\"\n  output_stream: \"MATRICES:1:asset_3d_matrices\"\n  input_side_packet: \"FOV:vertical_fov_radians\"\n  input_side_packet: \"ASPECT_RATIO:aspect_ratio\"\n}\n\n# Renders the final 3d stickers and overlays them on input image.\nnode {\n  calculator: \"GlAnimationOverlayCalculator\"\n  input_stream: \"VIDEO:input_video\"\n  input_stream: \"MODEL_MATRICES:gif_matrices\"\n  input_stream: \"TEXTURE:gif_texture\"\n  input_side_packet: \"ANIMATION_ASSET:gif_asset_name\"\n  output_stream: \"asset_gif_rendered\"\n}\n\n# Renders the final 3d stickers and overlays them on input image.\nnode {\n  calculator: \"GlAnimationOverlayCalculator\"\n  input_stream: \"VIDEO:asset_gif_rendered\"\n  input_stream: \"MODEL_MATRICES:asset_3d_matrices\"\n  input_side_packet: \"TEXTURE:texture_3d\"\n  input_side_packet: \"ANIMATION_ASSET:asset_3d\"\n  output_stream: \"output_video\"\n}\n```\n\n### Region Tracking Subgraph\n\n![region_tracking_subgraph](images/anchor-tracking-subgraph.png)\n\n[Source pbtxt file](mediapipe/graphs/instantmotiontracking/subgraphs/region_tracking.pbtxt)\n\n```bash\n# MediaPipe graph that performs region tracking on initial anchor positions\n# provided by the application\n\n# Images in/out of graph with tracked and scaled normalized anchor data\ntype: \"RegionTrackingSubgraph\"\ninput_stream: \"VIDEO:input_video\"\ninput_stream: \"SENTINEL:sticker_sentinel\"\ninput_stream: \"ANCHORS:initial_anchor_data\"\noutput_stream: \"ANCHORS:tracked_scaled_anchor_data\"\n\n# Manages the anchors and tracking if user changes/adds/deletes anchors\nnode {\n calculator: \"TrackedAnchorManagerCalculator\"\n input_stream: \"SENTINEL:sticker_sentinel\"\n input_stream: \"ANCHORS:initial_anchor_data\"\n input_stream: \"BOXES:boxes\"\n input_stream_info: {\n   tag_index: 'BOXES'\n   back_edge: true\n }\n output_stream: \"START_POS:start_pos\"\n output_stream: \"CANCEL_ID:cancel_object_id\"\n output_stream: \"ANCHORS:tracked_scaled_anchor_data\"\n}\n\n# Subgraph performs anchor placement and tracking\nnode {\n  calculator: \"BoxTrackingSubgraph\"\n  input_stream: \"VIDEO:input_video\"\n  input_stream: \"START_POS:start_pos\"\n  input_stream: \"CANCEL_ID:cancel_object_id\"\n  output_stream: \"BOXES:boxes\"\n}\n```\n\n### Box Tracking Subgraph\n\n![box_tracking_subgraph](images/box-tracking-subgraph.png)\n\n[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/subgraphs/box_tracking_gpu.pbtxt)\n\n```bash\n# MediaPipe box tracking subgraph.\n\ntype: \"BoxTrackingSubgraph\"\n\ninput_stream: \"VIDEO:input_video\"\ninput_stream: \"BOXES:start_pos\"\ninput_stream: \"CANCEL_ID:cancel_object_id\"\noutput_stream: \"BOXES:boxes\"\n\nnode: {\n  calculator: \"ImageTransformationCalculator\"\n  input_stream: \"IMAGE_GPU:input_video\"\n  output_stream: \"IMAGE_GPU:downscaled_input_video\"\n  node_options: {\n    [type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {\n      output_width: 240\n      output_height: 320\n    }\n  }\n}\n\n# Converts GPU buffer to ImageFrame for processing tracking.\nnode: {\n  calculator: \"GpuBufferToImageFrameCalculator\"\n  input_stream: \"downscaled_input_video\"\n  output_stream: \"downscaled_input_video_cpu\"\n}\n\n# Performs motion analysis on an incoming video stream.\nnode: {\n  calculator: \"MotionAnalysisCalculator\"\n  input_stream: \"VIDEO:downscaled_input_video_cpu\"\n  output_stream: \"CAMERA:camera_motion\"\n  output_stream: \"FLOW:region_flow\"\n\n  node_options: {\n    [type.googleapis.com/mediapipe.MotionAnalysisCalculatorOptions]: {\n      analysis_options {\n        analysis_policy: ANALYSIS_POLICY_CAMERA_MOBILE\n        flow_options {\n          fast_estimation_min_block_size: 100\n          top_inlier_sets: 1\n          frac_inlier_error_threshold: 3e-3\n          downsample_mode: DOWNSAMPLE_TO_INPUT_SIZE\n          verification_distance: 5.0\n          verify_long_feature_acceleration: true\n          verify_long_feature_trigger_ratio: 0.1\n          tracking_options {\n            max_features: 500\n            adaptive_extraction_levels: 2\n            min_eig_val_settings {\n              adaptive_lowest_quality_level: 2e-4\n            }\n            klt_tracker_implementation: KLT_OPENCV\n          }\n        }\n      }\n    }\n  }\n}\n\n# Reads optical flow fields defined in\n# mediapipe/framework/formats/motion/optical_flow_field.h,\n# returns a VideoFrame with 2 channels (v_x and v_y), each channel is quantized\n# to 0-255.\nnode: {\n  calculator: \"FlowPackagerCalculator\"\n  input_stream: \"FLOW:region_flow\"\n  input_stream: \"CAMERA:camera_motion\"\n  output_stream: \"TRACKING:tracking_data\"\n\n  node_options: {\n    [type.googleapis.com/mediapipe.FlowPackagerCalculatorOptions]: {\n      flow_packager_options: {\n        binary_tracking_data_support: false\n      }\n    }\n  }\n}\n\n# Tracks box positions over time.\nnode: {\n  calculator: \"BoxTrackerCalculator\"\n  input_stream: \"TRACKING:tracking_data\"\n  input_stream: \"TRACK_TIME:input_video\"\n  input_stream: \"START_POS:start_pos\"\n  input_stream: \"CANCEL_OBJECT_ID:cancel_object_id\"\n  input_stream_info: {\n    tag_index: \"CANCEL_OBJECT_ID\"\n    back_edge: true\n  }\n  output_stream: \"BOXES:boxes\"\n\n  input_stream_handler {\n    input_stream_handler: \"SyncSetInputStreamHandler\"\n    options {\n      [mediapipe.SyncSetInputStreamHandlerOptions.ext] {\n        sync_set {\n          tag_index: \"TRACKING\"\n          tag_index: \"TRACK_TIME\"\n        }\n        sync_set {\n          tag_index: \"START_POS\"\n        }\n        sync_set {\n          tag_index: \"CANCEL_OBJECT_ID\"\n        }\n      }\n    }\n  }\n\n  node_options: {\n    [type.googleapis.com/mediapipe.BoxTrackerCalculatorOptions]: {\n      tracker_options: {\n        track_step_options {\n          track_object_and_camera: true\n          tracking_degrees: TRACKING_DEGREE_OBJECT_SCALE\n          inlier_spring_force: 0.0\n          static_motion_temporal_ratio: 3e-2\n        }\n      }\n      visualize_tracking_data: false\n      streaming_track_data_cache_size: 100\n    }\n  }\n}\n```\n\n## Licensing \u0026 Use\n\n    Copyright 2020 Google LLC\n\n    Licensed under the Apache License, Version 2.0 (the \"License\");\n    you may not use this file except in compliance with the License.\n    You may obtain a copy of the License at\n\n        https://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS,\n    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n    See the License for the specific language governing permissions and\n    limitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogleinterns%2Finstant_motion_tracking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogleinterns%2Finstant_motion_tracking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogleinterns%2Finstant_motion_tracking/lists"}