{"id":19401190,"url":"https://github.com/google-research/pangea","last_synced_at":"2025-07-11T15:41:09.655Z","repository":{"id":68171555,"uuid":"331371706","full_name":"google-research/pangea","owner":"google-research","description":"Panoramic Graph Environment Annotation toolkit, for collecting audio and text annotations in panoramic graph environments such as Matterport3D and StreetLearn.","archived":false,"fork":false,"pushed_at":"2021-03-05T20:35:57.000Z","size":2017,"stargazers_count":18,"open_issues_count":0,"forks_count":7,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-24T07:42:22.540Z","etag":null,"topics":["annotation-tool","computer-vision","crowdsourcing","nlp"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-01-20T16:54:36.000Z","updated_at":"2024-07-11T15:36:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"4c0e2d8e-413d-423d-8fea-0b471319bd44","html_url":"https://github.com/google-research/pangea","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/google-research/pangea","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fpangea","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fpangea/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fpangea/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fpangea/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/pangea/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fpangea/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264842226,"owners_count":23671957,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["annotation-tool","computer-vision","crowdsourcing","nlp"],"created_at":"2024-11-10T11:17:30.542Z","updated_at":"2025-07-11T15:41:09.645Z","avatar_url":"https://github.com/google-research.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PanGEA\n\nThe Panoramic Graph Environment Annotation toolkit, abbreviated as PanGEA, is a\nlightweight and customizable codebase for collecting audio and text\nannotations in panoramic graph environments, such as\n[Matterport3D](https://niessner.github.io/Matterport/) and\n[StreetLearn](https://sites.google.com/corp/view/streetlearn). PanGEA has been\nused to collect the\n[RxR dataset](https://github.com/google-research-datasets/RxR) of multilingual\nnavigation instructions, and to perform human wayfinding evaluations of\nmachine-generated navigation instructions.\n\n## Organization\n\nThe `src` directory contains the core components used to create a plugin.\n\n*   `environment.js` contains the `Environment` class for manipulating a graph\n    of panoramic viewpoints. Environments maintain a\n    [Three.js](https://threejs.org/) camera and scene. Also contains the\n    `Navigator` class, which is a strict superclass of an environment.\n    Navigators maintain a WebGL renderer, cursor, and first-person controls.\n*   `animation.js` contains the `Animation` class, which is a high-level wrapper\n    around the\n    [Web Animations API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Animations_API).\n*   `microphone.js` contains the `Microphone` class, which is a high-level\n    wrapper around the\n    [Web Audio API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API)\n    as well as an `RmsMeter` and `Histogram` class to visualize the audio\n    stream.\n*   `drawing.js` contains the `LineDrawing` class which renders a line in an\n    environment. This is useful for rendering paths to be annotated.\n*   `firebase.js` contains Firebase upload related utilities.\n*   `utils.js` contains common utility functions.\n*   `data_adapter.js` contains code for loading Matterport3D environments.\n\nThe `examples` directory contains examples of PanGEA in action.\n\n*   `environment_demo.html` demonstrates loading a navigator, recording a pose\n    trace, and replaying the recorded pose trace. A pose trace is a timestamped\n    sequence of camera poses that captures everything the annotator sees.\n*   `microphone_demo.html` demonstrates recording audio from the microphone.\n*   `guide_plugin.html` is the Guide plugin for an RxR annotation task.\n*   `follower_plugin.html` is the Follower plugin for an RxR annotation task.\n\nThe `environment_demo.html` and `microphone_demo.html` pages are minimal\nself-contained examples that don't require downloading any additional data. Just\nlaunch a local server to see them, e.g.\n\n```bash\npython3 -m http.server 8000\n```\n\nand browse to\n\n```bash\nhttp://localhost:8000/examples/environment_demo.html\nhttp://localhost:8000/examples/microphone_demo.html\n```\n\nThe `guide_plugin.html` and `follower_plugin.html` pages require a dataset to\nwork, as described below.\n\n## Guide and Follower plugins\n\nThe Guide and Follower plugins were used to collect the\n[RxR dataset](https://github.com/google-research-datasets/RxR) of multilingual\nnavigation instructions in the Matterport3D environment. Both plugins can be\nconfigured to save annotation outputs to [Firebase](https://firebase.google.com/).\n\n### Guide Task\n\nIn the Guide task, annotators are given a path in the environment. Their\nvoice is recorded as they freely move along the path and attempt to generate\nspoken navigation instructions others can follow. After they're satisfied with\ntheir instructions, they'll be asked to manually transcribe their own voice\nrecording into text.\n\nInput Args (currently loaded from `testdata/args.json`):\n\n- language: currently supports English, Hindi or Telugu. Used to configure\n[Google Input Tools](https://www.google.com/inputtools/) in the transcription\nstep\n- scan: environment identifer\n- path: sequence of viewpoint ids indicating the path to be annotated\n- heading: starting heading\n- path_id: your path identifier\n\nOutputs:\n\n- a pose trace (timestamped virtual camera poses, capturing the annotator's\nmovements and everything they see along the path)\n- the audio file of the annotator's voice recording, and\n- the annotator's manual transcript of their voice recording.\n\n### Follower Task\n\nIn the Follower task, annotators begins at the start of an unknown path and\ntry to follow a Guide’s instruction. They observe the environment and navigate\nin the simulator as the Guide’s audio plays. Once they believe they have reached\nthe the end of the path, or give up, they indicate they are done.\n\nInput Args (currently loaded from `testdata/args.json`):\n\n- audio: path to a Guide's audio recording\n- scan: environment identifer\n- path: a sequence of viewpoint ids (annotator starts at the first viewpoint)\n- heading: starting heading\n- path_id: your path identifier\n\nOutputs:\n\n- a pose trace (timestamped virtual camera poses, capturing the annotator's\nmovements and everything they see along the path)\n\n### Post-Processing\n\nIn the RxR dataset, all words in the manually-transcribed navigation\ninstructions are timestamped and aligned with both the Guide and Follower pose\ntraces. In a future release we will provide tooling to perform this alignment\nautomatically following the same approach as\n[Localized Narratives](https://google.github.io/localized-narratives/).\n\n### Running locally\n\nIn order to simulate an environment, PanGEA requires a dataset of 360-degree\npanoramic images, plus navigation graphs encoding the position of these\nviewpoints and the navigable connections between them. To use the Matterport3D\ndataset, download the Matterport3D skybox images from\n[here](https://niessner.github.io/Matterport/) and the navigation graphs from\n[here](https://github.com/peteanderson80/Matterport3DSimulator/tree/master/connectivity).\nPlease note that the Matterport3D data is governed by the following\n[Terms of Use](http://kaldir.vc.in.tum.de/matterport/MP_TOS.pdf). Alternatively,\na different dataset such as StreetLearn could be used by writing a new\ndataloader class in `data_adapter.js`.\n\nAssuming the data is saved in a directory as follows:\n\n```bash\n\u003cdata_dir\u003e/data/v1/scans/\u003cscan_id\u003e/matterport_skybox_images/\u003cpano_id\u003e_skybox\u003cface_id\u003e_sami.jpg\n\u003cdata_dir\u003e/connectivity/\u003cscan_id\u003e_connectivity.json\n```\n\ncreate a symlink to the data and start a local server\n\n```bash\nln -s \u003cdata_dir\u003e symdata\npython3 -m http.server 8000\n```\n\nthen browse to\n\n```bash\nhttp://localhost:8000/examples/guide_plugin.html\nhttp://localhost:8000/examples/follower_plugin.html\n```\n\n### Deploying remotely\n\n\n#### Environment data hosting\n\nTo deploy a plugin remotely (i.e., for a genuine data collection effort), you'll\nneed to host your environment data and javascript on a\n[Google Cloud Storage bucket](https://cloud.google.com/storage/). The\nenvironment data file paths should be as follows:\n\n```bash\ngs://\u003cbucket_id\u003e/data/v1/scans/\u003cscan_id\u003e/matterport_skybox_images/\u003cpano_id\u003e_skybox\u003cface_id\u003e_sami.jpg\ngs://\u003cbucket_id\u003e/connectivity/\u003cscan_id\u003e_connectivity.json\n```\n\nIn `guide_plugin.html` and `follower_plugin.html`, swap the argument to the\n`Matterport3D` function from `../symdata` to\n`https://storage.googleapis.com/\u003cbucket_id\u003e`, and update the javascript urls\nbeginning with `../src/` to point to the new location in your bucket.\n\nDon't forget to grant your plugin\n[read access](https://cloud.google.com/storage/docs/access-control) and\n[enable CORS](https://cloud.google.com/storage/docs/configuring-cors).\n\n#### Firebase integration\n\nThe Guide and Follower plugins are setup to use (1) a storage bucket to upload\nraw annotations (e.g., speech recordings and pose traces), and (2) a database to\nstore and organize additional metadata (e.g., audio transcriptions). Both are\nsetup using\n[Firebase](https://firebase.google.com/).\n\n*   First, create an app from the\n    [console](https://console.firebase.google.com/).\n*   Go to the Storage tab and add a bucket. This can either be the same bucket\n    used to host your data or a brand new one. We recommend the latter. Don't\n    forget to grant your plugin\n    [write access](https://cloud.google.com/storage/docs/access-control).\n*   Go to the Cloud Firestore tab and create your\n    [database](https://firebase.google.com/docs/firestore). Don't forget to\n    grant your plugin\n    [write access](https://firebase.google.com/docs/firestore/security/rules-structure).\n*   Firebase integration is built into `examples/guide_plugin.html` and\n    `examples/follower_plugin.html`, but you'll need to make some modifications.\n    *   Set `USE_FIREBASE` to true.\n    *   Set `FIREBASE_APP_CONFIG` to the app config under the Project Settings\n        tab.\n    *   Set `FIREBASE_COLLECTION` to the name of the database collection you\n        want the plugin to write documents to.\n    *   And optionally set `USE_GOOGLE_LOGIN` to true if you want to\n        authenticate the user via their Google account. This is useful if your\n        [rules](https://firebase.google.com/docs/firestore/security/rules-structure)\n        only allow specific users to write to the database. Make sure to enable\n        popups and redirects.\n\nYour app config should look something like this:\n\n```javascript\n {\n  apiKey: \"AIza....\",                             // Auth / General Use\n  appId: \"1:27992087142:web:ce....\",              // General Use\n  projectId: \"my-firebase-project\",               // General Use\n  authDomain: \"YOUR_APP.firebaseapp.com\",         // Auth with popup/redirect\n  databaseURL: \"https://YOUR_APP.firebaseio.com\", // Realtime Database\n  storageBucket: \"YOUR_APP.appspot.com\",          // Storage\n  messagingSenderId: \"123456789\",                 // Cloud Messaging\n  measurementId: \"G-12345\"                        // Analytics\n}\n```\n\n#### Crowdsourcing platform\n\nIntegration with crowdsourcing platforms (e.g., Amazon Mechanical Turk) is left\nto the user. Currently, both `examples/guide_plugin.html` and\n`examples/follower_plugin.html` load the required metadata (e.g., the path to be\nannotated) from `testdata/args.json`. To deploy a plugin, args.json should be\nreplaced with data from a crowdsourcing platform.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fpangea","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Fpangea","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fpangea/lists"}