{"id":51411586,"url":"https://github.com/alvinmurimi/agent-memory-observations","last_synced_at":"2026-07-04T15:02:03.453Z","repository":{"id":366427030,"uuid":"1276220956","full_name":"alvinmurimi/agent-memory-observations","owner":"alvinmurimi","description":"Empirical study of how LLM agents retrieve and use information in context; includes controlled experiments on retrieval, coreference, and scaling effects.","archived":false,"fork":false,"pushed_at":"2026-06-21T18:33:57.000Z","size":26,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-21T20:14:08.704Z","etag":null,"topics":["agents","benchmarking","coreference","evaluation","information-retrieval","llm","retrieval","synthetic-data"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alvinmurimi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-21T17:37:38.000Z","updated_at":"2026-06-21T18:34:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/alvinmurimi/agent-memory-observations","commit_stats":null,"previous_names":["alvinmurimi/agent-memory-observations"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/alvinmurimi/agent-memory-observations","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alvinmurimi%2Fagent-memory-observations","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alvinmurimi%2Fagent-memory-observations/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alvinmurimi%2Fagent-memory-observations/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alvinmurimi%2Fagent-memory-observations/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alvinmurimi","download_url":"https://codeload.github.com/alvinmurimi/agent-memory-observations/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alvinmurimi%2Fagent-memory-observations/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35125718,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-04T02:00:05.987Z","response_time":113,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","benchmarking","coreference","evaluation","information-retrieval","llm","retrieval","synthetic-data"],"created_at":"2026-07-04T15:01:58.215Z","updated_at":"2026-07-04T15:02:03.447Z","avatar_url":"https://github.com/alvinmurimi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Agent memory observations\n\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.20787349.svg)](https://doi.org/10.5281/zenodo.20787349)\n\nAn exploratory study that began from a practical question, \"can structuring a knowledge base make an LLM agent answer more accurately,\" and is frozen here at observations only.\n\nStart with these two files:\n\n- [OBSERVATIONS.md](OBSERVATIONS.md). What was measured, what held, what failed under controls, and what is explicitly unknown. This is the canonical record.\n- [JOURNEY.md](JOURNEY.md). How the study started, what it found, where it over-reached, and why it is paused.\n\n## Summary\n\nGiven the relevant evidence inside its context window, a capable LLM reader already resolves coreference, time, and source authority on its own, so structuring that evidence for reasoning did not improve answer accuracy on the tasks tested (stable across two model tiers). The most useful by-product is methodological: LLM reader accuracy showed large run-to-run variance (0.53 to 0.93 on identical inputs), so low-sample single-run comparisons are unreliable. A separate set of synthetic retrieval experiments observed that recall is sensitive to whether a text unit contains an explicit lexical anchor for the queried entity. That is a known information-retrieval phenomenon, measured here in a synthetic regime and never validated against a production pipeline. Nothing here establishes a memory architecture principle.\n\n## Experiments (deterministic, no API keys)\n\n- [recall_ablation.py](recall_ablation.py): recall ablations (E1)\n- [e1b_multihop_fair.py](e1b_multihop_fair.py): de-rigged multi-hop re-test (E1b)\n- [e1d_recall_boundary.py](e1d_recall_boundary.py): single-shot recall boundary (E1d)\n- [e7_scale_retrieval.py](e7_scale_retrieval.py): lexical recall@B vs scale (E7)\n- [dense_local.py](dense_local.py): static-dense recall@B vs scale\n- [e8_hybrid.py](e8_hybrid.py): lexical + dense RRF hybrid\n- [e7b_extraction_curve.py](e7b_extraction_curve.py): write-time locality recovery curve\n- [e9_recallK.py](e9_recallK.py): burial-depth / recall@K sweep\n- [requirements.txt](requirements.txt), aggregated outputs in [results/](results)\n\n## Reproduce\n\n```\npip install -r requirements.txt\npython recall_ablation.py\npython e1b_multihop_fair.py\npython e1d_recall_boundary.py\npython e7_scale_retrieval.py\npython dense_local.py\npython e8_hybrid.py\npython e7b_extraction_curve.py\npython e9_recallK.py\n```\n\nThe LLM-in-the-loop experiments (E2 to E6, run across two model tiers) are not reproducible without a model endpoint; their aggregated metrics are in [results/](results) and their design is described in [OBSERVATIONS.md](OBSERVATIONS.md).\n\n## Related prior work (provenance only)\n\nListed for lineage. This study does not extend, unify, or refute them.\n\n- [CORE-RT](https://github.com/alvinmurimi/core-rt): a contamination- and tautology-controlled corrective-QA benchmark (read-time resolution and recency effects).\n- [StaleRAG](https://github.com/alvinmurimi/StaleRAG): an earlier exploration of temporal staleness in retrieval.\n\n## Status\n\nPaused, on purpose. The useful results are small and engineering-ready. A broader claim would require a deployed-system study that was not done. See [OBSERVATIONS.md](OBSERVATIONS.md) section D for the explicit unknowns.\n\n## License\n\nCode is released under the MIT License (see [LICENSE](LICENSE)). Documentation and aggregated results may be reused with attribution.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falvinmurimi%2Fagent-memory-observations","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falvinmurimi%2Fagent-memory-observations","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falvinmurimi%2Fagent-memory-observations/lists"}