{"id":50939994,"url":"https://github.com/mandarwagh9/egomem","last_synced_at":"2026-06-17T13:03:19.626Z","repository":{"id":364642463,"uuid":"1268692130","full_name":"mandarwagh9/egomem","owner":"mandarwagh9","description":"A model-agnostic memory layer for robotics from egocentric human video","archived":false,"fork":false,"pushed_at":"2026-06-13T20:37:34.000Z","size":141,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-13T22:22:03.230Z","etag":null,"topics":["egocentric-video","embodied-ai","lerobot","memory","robotics","vla","world-models"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mandarwagh9.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-13T20:34:54.000Z","updated_at":"2026-06-13T20:46:02.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mandarwagh9/egomem","commit_stats":null,"previous_names":["mandarwagh9/egomem"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/mandarwagh9/egomem","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mandarwagh9%2Fegomem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mandarwagh9%2Fegomem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mandarwagh9%2Fegomem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mandarwagh9%2Fegomem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mandarwagh9","download_url":"https://codeload.github.com/mandarwagh9/egomem/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mandarwagh9%2Fegomem/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34449283,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-17T02:00:05.408Z","response_time":127,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["egocentric-video","embodied-ai","lerobot","memory","robotics","vla","world-models"],"created_at":"2026-06-17T13:03:18.277Z","updated_at":"2026-06-17T13:03:19.618Z","avatar_url":"https://github.com/mandarwagh9.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# EgoMem\n\n[![ci](https://github.com/mandarwagh9/egomem/actions/workflows/ci.yml/badge.svg)](https://github.com/mandarwagh9/egomem/actions/workflows/ci.yml)\n\n**A model-agnostic memory layer for robotics from egocentric human video.** One\nneutral `write(obs) -\u003e mem` / `query(state) -\u003e retrieved_context` store that\n**both a world model and a VLA policy** can call — written once from egocentric\nobservations (RGB-derived detections + depth + camera pose), queried unchanged by\nany consumer. In the literature, memory is always baked inside a specific VLA, a\nspecific world model, or an LLM agent; EgoMem is the buyer-side, paradigm-neutral\nlayer that seam is missing (see `paper/paper.md` §2).\n\n## Headline result\n\nOn a controlled egomotion testbed, on an **out-of-view object recall** task\n(\"object permanence under egomotion\"), a single non-parametric EgoMem store —\nqueried unchanged by both a world-model position predictor and a VLA direction\npolicy — beats a no-memory baseline and a naive raw-frame buffer, across 3 seeds:\n\n| Consumer | no-memory | naive buffer | **EgoMem** |\n|---|---|---|---|\n| World-model (pos recall @0.5 m) | 0.011 | 0.018 | **0.999** |\n| VLA (direction @30°) | 0.244 | 0.317 | **1.000** |\n\n(means of seeds 0–2; every cell is a row in [`RESULTS.md`](RESULTS.md).) The\nhypothesis is **confirmed for both consumers** from one unchanged memory — the\ncross-paradigm transfer is demonstrated, not assumed.\n\n**Replicated on real data.** The *unchanged* library reproduces this on real\negocentric scans (ARKitScenes: real ARKit VIO poses + 3D layouts, 14 scenes):\nEgoMem **0.70–1.00** world-model and **1.00** VLA recall vs **≤ 0.03** for both\nbaselines, across 2 seeds (projection gate passed 14/14). See `paper/paper.md` §7.\nIt also **survives heavy detection noise + dropout** (§7.1): EgoMem stays above\nthe baselines across a noise×miss grid (worst-corner 0.53 / 0.83 vs ≤ 0.10), and\nits margin over a raw buffer *widens* as detections worsen — the cross-frame\naveraging is the robustness.\n\n**Honest caveat (the boundary):** the advantage is *localization-quality\ndependent*. Under injected camera-pose drift the precise world-model consumer\ndegrades and, at heavy drift, **collapses to the baseline** (H1 rejected for that\nconsumer), while the coarser VLA direction consumer survives. See the ablation in\n`paper/paper.md` §6. The current testbed is **synthetic** (no real human video\nyet) with oracle data association — real-clip validation is the next step.\n\n## EgoMem augments a *real, frozen* VLM (OpenEQA-style spatial QA)\n\n![EgoMem augments a frozen VLM on episodic-spatial QA](paper/figures/qa_results.png)\n\nThe recognized 2026 frontier is episodic-spatial QA, where VLMs are documented \"nearly\nblind\" (OpenEQA: best VLM ~0.50). EgoMem is model-agnostic in practice: hand its spatial\nsummary to a **frozen, off-the-shelf Gemini 2.5-flash** (no fine-tuning) and its accuracy on\nreal-ARKitScenes episodic-spatial questions jumps — **beating the same VLM that is actually\nshown egocentric photos of the room** (the true OpenEQA setting):\n\n| condition (frozen VLM) | spatial-QA accuracy |\n|---|---|\n| no-memory | 0.273 |\n| **VLM shown 5 egocentric photos** | 0.491 |\n| current-view object list (text) | 0.727 |\n| **+ EgoMem spatial summary** | **0.927** |\n\n**+43.6 points over the VLM that sees the room**, +65.5 over no-memory (6 scenes, 55 Qs);\ngains concentrate on counting (2/12 → 11/12) and spatial relations (4/13 → 13/13) — exactly\nwhere VLMs fail. Any VLM gains episodic-spatial competence by reading EgoMem. See\n`paper/paper.md` §9; rows in [`RESULTS.md`](RESULTS.md) (`exp_id = spatial-qa H11b`).\n\n## Install \u0026 reproduce\n\n```bash\npip install -e lib            # Python \u003e=3.10, numpy + torch (CPU is fine)\n\negomem demo                   # one-command demo: small synthetic run, prints the 3-arm comparison\negomem sim --seed 0           # full benchmark — reproduces the seed-0 numbers in RESULTS.md\negomem sim --seed 0 --pose-drift 0.15   # the heavy-drift ablation (H1 rejected for world-model)\n```\n\n**Real-data run (ARKitScenes).** The loader at\n`experiments/2026-06-13_arkit-oov/arkit_loader.py` downloads scene zips from the\npublic ARKitScenes CDN and drives the unchanged library; see `research/design_h2.md`\nfor the protocol and `experiments/2026-06-13_arkit-oov/config.json` for the scene ids.\n\n`egomem sim --seed 0` prints no-memory 0.016 / 0.280, naive 0.027 / 0.371, EgoMem\n1.000 / 1.000 — identical to the committed experiment\n(`experiments/2026-06-13_oov-recall/stdout_lib_seed0.log`).\n\n**VLM spatial-QA run.** `experiments/2026-06-15_spatial-qa/spatial_qa.py` builds EgoMem over\nreal ARKitScenes scenes and queries a frozen VLM under no-memory / vision-frames / EgoMem\ncontexts. Needs a Gemini key (`GEMINI_API_KEY`) or Vertex ADC (`EGOMEM_VERTEX_PROJECT`).\n\nUse the layer directly (write once, query from any consumer): see `lib/README.md`.\n\n## Repo map\n\n| Path | What |\n|---|---|\n| [`paper/paper.md`](paper/paper.md) | the paper: method, results, ablations (§6 drift, §7 real data, §8 task success, §9 VLM-QA), limitations |\n| [`RESULTS.md`](RESULTS.md) | append-only experiment log — every reported number |\n| [`BIBLIOGRAPHY.md`](BIBLIOGRAPHY.md) | verified sources + 2026 SOTA refs (OpenEQA, MemoryVLA, …) |\n| [`research/`](research) | `landscape.md` (lit map) + falsifiable hypotheses H1–H11 + designs |\n| [`lib/`](lib) | the installable `egomem` package + CLI (5 memory classes, 9 tests) |\n| [`experiments/`](experiments) | raw logs, configs, metrics per run (synthetic, ARKitScenes, task, spatial-QA) |\n| [`PROGRESS.md`](PROGRESS.md) | the research-loop state and run log |\n\n## How it works (one paragraph)\n\n`EgoMem.write` transforms each detection's camera-frame 3D position into a world\nframe via the observation pose and accumulates a running per-object estimate;\n`EgoMem.query` transforms those world positions back into the *querying* camera\nframe, so every consumer receives an egocentric answer (`pos_cam_now`) without\nknowing the memory's internals. The store is non-parametric — written once,\nqueried unchanged by a state predictor and an action policy alike. The two naive\nbaselines (no memory; raw buffer with no pose transform) isolate that the win\ncomes from pose-aware world-frame integration.\n\n## License\n\nMIT.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmandarwagh9%2Fegomem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmandarwagh9%2Fegomem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmandarwagh9%2Fegomem/lists"}