{"id":47663588,"url":"https://github.com/basicmachines-co/basic-memory-benchmarks","last_synced_at":"2026-04-02T11:47:40.534Z","repository":{"id":340695275,"uuid":"1167194152","full_name":"basicmachines-co/basic-memory-benchmarks","owner":"basicmachines-co","description":"Reproducible benchmark suite for Basic Memory and competitor memory systems","archived":false,"fork":false,"pushed_at":"2026-03-15T02:46:46.000Z","size":369,"stargazers_count":0,"open_issues_count":11,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-15T13:52:00.701Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/basicmachines-co.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-26T03:15:22.000Z","updated_at":"2026-02-27T15:52:17.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/basicmachines-co/basic-memory-benchmarks","commit_stats":null,"previous_names":["basicmachines-co/basic-memory-benchmarks"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/basicmachines-co/basic-memory-benchmarks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basicmachines-co%2Fbasic-memory-benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basicmachines-co%2Fbasic-memory-benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basicmachines-co%2Fbasic-memory-benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basicmachines-co%2Fbasic-memory-benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/basicmachines-co","download_url":"https://codeload.github.com/basicmachines-co/basic-memory-benchmarks/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basicmachines-co%2Fbasic-memory-benchmarks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31305809,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T09:48:21.550Z","status":"ssl_error","status_checked_at":"2026-04-02T09:48:19.196Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-02T11:47:38.561Z","updated_at":"2026-04-02T11:47:40.526Z","avatar_url":"https://github.com/basicmachines-co.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# basic-memory-benchmarks\n\nStandalone, reproducible benchmark suite for comparing Basic Memory against competitor memory systems.\n\n## Goals\n\n- Deterministic retrieval benchmarks (Recall@5/10, MRR, Precision@5, content-hit, latency)\n- Optional LLM-as-judge scoring (Pydantic Evals)\n- Public artifacts with provenance and reproducibility metadata\n- Clean dependency isolation from the core `basic-memory` repository\n\n## Current v1 Scope\n\n- Providers:\n  - `bm-local` (warm `bm mcp` stdio session)\n  - `bm-cloud` (optional, credential-gated)\n  - `mem0-local`\n  - `zep-reference` (reference-only in v1)\n- Datasets:\n  - LoCoMo (primary)\n  - LongMemEval scaffold (placeholder)\n  - Built-in synthetic smoke corpus\n\n## Installation\n\n```bash\nuv sync --group dev\n```\n\nOptional judge dependencies:\n\n```bash\nuv sync --group dev --extra judge\n```\n\n## Quickstart\n\n### 1) Fetch LoCoMo dataset\n\n```bash\nuv run bm-bench datasets fetch --dataset locomo\n```\n\n### 2) Convert LoCoMo into benchmark corpus\n\n```bash\nuv run bm-bench convert locomo\n```\n\n### 3) Run retrieval benchmark\n\n```bash\nuv run bm-bench run retrieval \\\n  --providers bm-local,mem0-local \\\n  --corpus-dir benchmarks/generated/locomo/docs \\\n  --queries-path benchmarks/generated/locomo/queries.json\n```\n\n### 4) Optional judge benchmark\n\n```bash\nuv run bm-bench run judge --run-dir benchmarks/runs/\u003crun-id\u003e\n```\n\n### 5) Publish run artifacts\n\n```bash\nuv run bm-bench publish --run-dir benchmarks/runs/\u003crun-id\u003e\n```\n\n## Basic Memory source policy\n\nBy default this project tracks Basic Memory from `main`.\n\nEach run manifest stores:\n- BM source (`github main` or local path override)\n- resolved BM commit SHA\n\nLocal override:\n\n```bash\nuv run bm-bench run retrieval \\\n  --bm-local-path /path/to/basic-memory\n```\n\n## Mem0 local requirements\n\n`mem0-local` requires model credentials available in environment.\n\nAt minimum, set:\n\n```bash\nexport OPENAI_API_KEY=...\n```\n\nIf unavailable, provider status will be recorded as `SKIPPED(reason)`.\n\n## BM indexing readiness\n\n`bm-local` verifies index readiness before querying.\n\n- If the installed `bm` supports `bm status --json`, readiness is polled from that output.\n- If `--json` is not available in the installed `bm`, the benchmark proceeds after reindex.\n\n## Run Artifacts\n\nPer run (`benchmarks/runs/\u003crun-id\u003e/`):\n\n- `manifest.json`\n- `provider-status.json`\n- `per-query-retrieval.jsonl`\n- `retrieval-summary.json`\n- `per-query-judge.jsonl` (optional)\n- `judge-summary.json` (optional)\n- `summary.md`\n\n## Just commands\n\n```bash\njust bench-smoke\njust bench-fetch-locomo\njust bench-convert-locomo\njust bench-run-bm-local\njust bench-run-mem0-local\njust bench-run-full\njust bench-judge\njust bench-publish RUN_DIR=benchmarks/runs/\u003crun-id\u003e\n```\n\n## Notes on dataset publication\n\nDataset publication follows licensing constraints:\n- If redistribution is permitted: snapshot + checksum may be published.\n- If not: canonical source links + downloader + checksum verification are published.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbasicmachines-co%2Fbasic-memory-benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbasicmachines-co%2Fbasic-memory-benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbasicmachines-co%2Fbasic-memory-benchmarks/lists"}