{"id":51492116,"url":"https://github.com/evermind-ai/evermembench","last_synced_at":"2026-07-07T12:02:10.325Z","repository":{"id":323961734,"uuid":"1085720709","full_name":"EverMind-AI/EverMemBench","owner":"EverMind-AI","description":null,"archived":false,"fork":false,"pushed_at":"2026-02-13T16:06:26.000Z","size":130,"stargazers_count":30,"open_issues_count":1,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-28T16:41:30.313Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EverMind-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-29T12:28:50.000Z","updated_at":"2026-03-28T03:18:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/EverMind-AI/EverMemBench","commit_stats":null,"previous_names":["evermind-ai/evermembench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/EverMind-AI/EverMemBench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EverMind-AI%2FEverMemBench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EverMind-AI%2FEverMemBench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EverMind-AI%2FEverMemBench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EverMind-AI%2FEverMemBench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EverMind-AI","download_url":"https://codeload.github.com/EverMind-AI/EverMemBench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EverMind-AI%2FEverMemBench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35226918,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-07T12:02:08.010Z","updated_at":"2026-07-07T12:02:10.307Z","avatar_url":"https://github.com/EverMind-AI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multi-Person Group Chat Evaluation Framework\n\n[![arXiv](https://img.shields.io/badge/arXiv-2602.01313-b31b1b.svg)](https://arxiv.org/pdf/2602.01313)\n[![Dataset](https://img.shields.io/badge/🤗%20Dataset-EverMemBench--Dynamic-yellow)](https://huggingface.co/datasets/EverMind-AI/EverMemBench-Dynamic)\n\nA comprehensive evaluation framework for multi-person group chat datasets, supporting **Memory Systems** (Memos, Mem0, Memobase, EverMemOS, Zep) and **LLM Long-Context Evaluation**.\n\n📄 **Paper**: [EverMemBench: A Comprehensive Benchmark for Long-Term Memory in Conversational AI](https://arxiv.org/pdf/2602.01313)\n\n🤗 **Dataset**: [EverMind-AI/EverMemBench-Dynamic](https://huggingface.co/datasets/EverMind-AI/EverMemBench-Dynamic)\n\n## Features\n\n- **Multi-person group chat support**: Handles datasets with multiple speakers across multiple groups and days\n- **5 Memory Systems**: Memos, Mem0, Memobase, EverMemOS, Zep\n- **LLM Long-Context Evaluation**: Direct LLM evaluation using full dialogue as context\n- **Full Evaluation Pipeline**: Add → Search → Answer → Evaluate\n- **Two Question Types**: Multiple choice (direct comparison) and open-ended (LLM judge)\n- **Unified message format**: All messages include group/speaker attribution\n- **LLM Integration**: Uses OpenRouter for answer generation and evaluation\n- **Batch processing**: Efficient API calls with configurable batch sizes and rate limiting\n- **Smoke test mode**: Quick validation with limited data\n\n## Pipeline Stages\n\n```\n┌─────────┐    ┌──────────┐    ┌──────────┐    ┌───────────┐\n│   Add   │ -\u003e │  Search  │ -\u003e │  Answer  │ -\u003e │ Evaluate  │\n└─────────┘    └──────────┘    └──────────┘    └───────────┘\n     │              │               │               │\n     v              v               v               v\n  Ingest       Retrieve LLM      Generate       Assess\n memories     memories        answers       accuracy\n```\n\n| Stage | Description | Output |\n|-------|-------------|--------|\n| **Add** | Ingest conversation data into memory system | - |\n| **Search** | Retrieve relevant memories for QA questions | `search_results_{user_id}.json` |\n| **Answer** | Generate answers using LLM with retrieved context | `answer_results_{user_id}.json` |\n| **Evaluate** | Assess answer quality (MC: direct, OE: LLM judge) | `evaluation_results_{user_id}.json` |\n\n## Supported Systems\n\n### Memory Systems\n\n| System | Timestamp Support | Message Format | Environment Variables |\n|--------|-------------------|----------------|----------------------|\n| **Memos** | Native `chat_time` | `[Group: X][Speaker: Y]content` | `MEMOS_API_KEY`, `MEMOS_BASE_URL` |\n| **Mem0** | Native `timestamp` (Unix, per-batch) | `run_id=\"${user_id}_${groupId}\"`, `name=\u003cSpeaker\u003e` | `MEM0_API_KEY` |\n| **Memobase** | Native `created_at` | `[Group: X][Speaker: Y]content`, `alias=\u003cSpeaker\u003e` | `MEMOBASE_BASE_URL`, `MEMOBASE_API_TOKEN` |\n| **EverMemOS** | Native `create_time` | `sender=\u003cSpeaker\u003e`, `group_id=${user_id}_${groupId}` | `EVERMEMOS_BASE_URL`, `EVERMEMOS_API_KEY` |\n| **Zep** | Native `created_at` | `[Group: X][Speaker: Y]content` | `ZEP_API_KEY` |\n\n### LLM System\n\n| System | Context | Use Case | Environment Variables |\n|--------|---------|----------|----------------------|\n| **LLM** | Full dialogue (no retrieval) | Test LLM long-context comprehension | `LLM_BASE_URL`, `LLM_API_KEY` |\n\n**Key Differences: Memory Systems vs LLM System**\n\n| Aspect | Memory Systems | LLM System |\n|--------|---------------|------------|\n| Context | Retrieved memories (top-k) | Full dialogue |\n| Add Stage | Ingest into memory system | No-op (stores dialogue) |\n| Search Stage | Query memory system | Returns full dialogue |\n| Answer Stage | Answer with retrieved context | Answer with full dialogue |\n| Use Case | Test memory retrieval | Test LLM long-context |\n\n## Directory Structure\n\n```\neval/\n├── cli.py                    # CLI entry point\n├── config/\n│   ├── pipeline.yaml        # Pipeline settings (answer/evaluate/search/retry/debug)\n│   ├── prompts.yaml         # LLM prompts for answer/evaluate\n│   ├── memos.yaml           # Memos configuration (connection + add + search)\n│   ├── mem0.yaml            # Mem0 configuration (connection + add + search)\n│   ├── memobase.yaml        # Memobase configuration (connection + add + search)\n│   ├── evermemos.yaml       # EverMemOS configuration (connection + add + search)\n│   └── zep.yaml             # Zep configuration (connection + add + search)\n├── src/\n│   ├── core/\n│   │   ├── data_models.py   # Data classes (QAItem, SearchResult, etc.)\n│   │   ├── loaders.py       # Dataset loading utilities\n│   │   ├── qa_loader.py     # QA data loader\n│   │   ├── pipeline.py      # Evaluation pipeline orchestrator\n│   │   ├── answerer.py      # Answer generation with LLM\n│   │   └── evaluator.py     # Evaluation with LLM judge\n│   ├── adapters/\n│   │   ├── base.py          # Base adapter abstract class\n│   │   ├── memos_adapter.py # Memos implementation\n│   │   ├── mem0_adapter.py  # Mem0 implementation\n│   │   ├── memobase_adapter.py   # Memobase implementation\n│   │   ├── evermemos_adapter.py  # EverMemOS implementation\n│   │   ├── zep_adapter.py   # Zep Graph API implementation\n│   │   └── llm_adapter.py   # LLM system adapter (full dialogue as context)\n│   └── utils/\n│       ├── config.py        # YAML config loader with env var support\n│       └── logger.py        # Rich console logging\n└── results/{system}/        # Output: eval/results/{system}/*.json\n│                            #   LLM: eval/results/llm/{model}/*.json\ntools/\n└── analyze_results.py       # Analyze evaluation results by category\n```\n\n## Installation\n\n**Requires Python \u003e= 3.11**.\n\n```bash\npip install -r requirements.txt\n```\n\n## Configuration\n\n### Environment Variables\n\nCopy the template and fill in your API keys:\n\n```bash\ncp env.template .env\n```\n\nThe LLM variables (OpenRouter) are required for answer generation and evaluation across all systems. Memory system variables only need to be configured for the systems you intend to use. See `env.template` for details.\n\n### Pipeline Configuration\n\nPipeline settings are in `eval/config/pipeline.yaml`.\n\n```yaml\n# eval/config/pipeline.yaml\n\n# Answer generation (answerer.py)\nanswer:\n  model: \"openai/gpt-4.1-mini\"\n  provider:\n    order: [\"openai\"]\n    allow_fallbacks: false\n  temperature: 0\n  max_tokens: 1000\n  timeout: 300\n  concurrency: 1\n\n# LLM judge evaluation (evaluator.py)\nevaluate:\n  model: \"google/gemini-3-flash-preview\"\n  provider:\n    order: [\"google-ai-studio\"]\n    allow_fallbacks: false\n  concurrency: 20\n\n# Search stage (pipeline.py)\nsearch:\n  concurrency: 3\n  timeout: 120\n\n# Retry (shared)\nretry:\n  max_retries: 20\n  retry_delay: 1.0\n  max_delay: 300\n\n# Debug\ndebug:\n  show_usage: true\n\n# Cache warmup (LLM system only)\nwarmup:\n  enabled: true\n  delay_seconds: 15\n```\n\n### System Search Configuration\n\nEach memory system has its own config file (`eval/config/{system}.yaml`) with a `search:` section for system-specific search parameters. CLI `--top-k` overrides the config `top_k` when provided.\n\n```yaml\n# eval/config/memos.yaml\nsearch:\n  top_k: 10                        # Number of memories to retrieve\n  preference_limit_number: 6        # Number of preference memories\n\n# eval/config/mem0.yaml\nsearch:\n  top_k: 10\n  group_ids: [\"1\", \"2\", \"3\"]       # Group IDs to search across\n\n# eval/config/memobase.yaml\nsearch:\n  max_token_size: 3000              # Max token size for search results\n  event_similarity_threshold: 0.2   # Similarity threshold for event matching\n\n# eval/config/evermemos.yaml\nsearch:\n  top_k: 10\n  retrieve_method: \"hybrid\"         # Retrieval method: hybrid/semantic/keyword\n\n# eval/config/zep.yaml\nsearch:\n  top_k: 10\n  reranker_edges: \"cross_encoder\"   # Edge reranking strategy\n  reranker_nodes: \"rrf\"             # Node reranking strategy\n  max_query_length: 400             # Max query length for search\n```\n\n### Prompt Templates\n\n```yaml\n# eval/config/prompts.yaml\nllm_answer:\n  multiple_choice: |\n    ...\n  open_ended: |\n    ...\nllm_judge:\n  system_prompt: |\n    ...\n  user_prompt: |\n    ...\n```\n\n## Usage\n\n### Memory Systems Evaluation\n\nMemory systems follow a two-phase workflow: **Add** (ingest data), then **Search → Answer → Evaluate** (run evaluation).\n\n#### Memos\n\n```bash\n# Add\npython -m eval.cli \\\n    --dataset dataset/004/dialogue.json \\\n    --system memos \\\n    --user-id 004 \\\n    --stages add\n\n# Search -\u003e Answer -\u003e Evaluate\npython -m eval.cli \\\n    --dataset dataset/004/dialogue.json \\\n    --qa dataset/004/qa_004.json \\\n    --system memos \\\n    --user-id 004 \\\n    --stages search answer evaluate \\\n    --top-k 10\n```\n\n#### Mem0\n\n```bash\n# Add\npython -m eval.cli \\\n    --dataset dataset/004/dialogue.json \\\n    --system mem0 \\\n    --user-id 004 \\\n    --stages add\n\n# Search -\u003e Answer -\u003e Evaluate\npython -m eval.cli \\\n    --dataset dataset/004/dialogue.json \\\n    --qa dataset/004/qa_004.json \\\n    --system mem0 \\\n    --user-id 004 \\\n    --stages search answer evaluate \\\n    --top-k 10\n```\n\n#### Memobase\n\n```bash\n# Add\npython -m eval.cli \\\n    --dataset dataset/004/dialogue.json \\\n    --system memobase \\\n    --user-id 004 \\\n    --stages add\n\n# Search -\u003e Answer -\u003e Evaluate\npython -m eval.cli \\\n    --dataset dataset/004/dialogue.json \\\n    --qa dataset/004/qa_004.json \\\n    --system memobase \\\n    --user-id 004 \\\n    --stages search answer evaluate\n```\n\n#### EverMemOS\n\nEverMemOS requires **separate data isolation per batch** (user ID):\n- **Cloud service**: Create a new memspace for each batch via the EverMemOS dashboard, then use the corresponding `--base-url`.\n- **Local deployment**: Start a separate service instance per batch, each on its own port (e.g., port `19004` for user `004`, port `19005` for user `005`). API key is not required for local deployment.\n\n```bash\n# Add (local deployment, port per batch)\npython -m eval.cli \\\n    --dataset dataset/004/dialogue.json \\\n    --system evermemos \\\n    --user-id 004 \\\n    --stages add \\\n    --base-url http://0.0.0.0:19004\n\n# Search -\u003e Answer -\u003e Evaluate\npython -m eval.cli \\\n    --dataset dataset/004/dialogue.json \\\n    --qa dataset/004/qa_004.json \\\n    --system evermemos \\\n    --user-id 004 \\\n    --stages search answer evaluate \\\n    --top-k 10 \\\n    --base-url http://0.0.0.0:19004\n```\n\n#### Zep\n\n```bash\n# Add\npython -m eval.cli \\\n    --dataset dataset/004/dialogue.json \\\n    --system zep \\\n    --user-id 004 \\\n    --stages add\n\n# Search -\u003e Answer -\u003e Evaluate\npython -m eval.cli \\\n    --dataset dataset/004/dialogue.json \\\n    --qa dataset/004/qa_004.json \\\n    --system zep \\\n    --user-id 004 \\\n    --stages search answer evaluate \\\n    --top-k 10\n```\n\n### LLM Long-Context Evaluation\n\nThe LLM system uses the **full dialogue** as context (no memory retrieval). Add/search stages are auto-injected.\n\n```bash\npython -m eval.cli \\\n    --dataset dataset/004/dialogue.json \\\n    --qa dataset/004/qa_004.json \\\n    --system llm \\\n    --user-id 004 \\\n    --stages answer evaluate\n```\n\n### Evaluate Only (re-evaluate existing answer results)\n\n```bash\npython -m eval.cli \\\n    --qa dataset/004/qa_004.json \\\n    --system mem0 \\\n    --user-id 004 \\\n    --stages evaluate\n```\n\n### Smoke Test\n\n```bash\n# Smoke test add stage\npython -m eval.cli --dataset dataset/004/dialogue.json --system memos --smoke\n\n# Smoke test with specific date\npython -m eval.cli --dataset dataset/004/dialogue.json --system memos --smoke --smoke-date 2025-01-16\n\n# LLM smoke test with limited questions\npython -m eval.cli \\\n    --dataset dataset/004/dialogue.json \\\n    --qa dataset/004/qa_004.json \\\n    --system llm \\\n    --user-id 004 \\\n    --stages answer evaluate \\\n    --qa-limit 3\n```\n\n## CLI Options\n\n| Option | Description | Default |\n|--------|-------------|---------|\n| `--dataset` | Path to dataset JSON file (required for add stage) | - |\n| `--system` | System (memos/mem0/memobase/evermemos/zep/llm) | Required |\n| `--stages` | Stages to run: add, search, answer, evaluate | `[\"add\"]` |\n| `--qa` | Path to QA JSON file (required for search/answer/evaluate) | - |\n| `--user-id` | User ID for memory system | Auto-generated |\n| `--top-k` | Number of memories to retrieve | From system config |\n| `--output-dir` | Results base directory (output goes to `{output-dir}/{system}/`) | `eval/results` |\n| `--base-url` | Override base URL for memory system | - |\n| `--start-date` | Resume add from this date (YYYY-MM-DD) | - |\n| `--smoke` | Enable smoke test mode | False |\n| `--smoke-days` | Days to process in smoke test | 1 |\n| `--smoke-date` | Specific date for smoke test (YYYY-MM-DD) | - |\n| `--qa-limit` | Limit number of QA questions | - |\n\n## Output Structure\n\nResults are organized by system under `eval/results/`:\n\n```\neval/results/\n├── memos/\n│   ├── search_results_004.json\n│   ├── answer_results_004.json\n│   └── evaluation_results_004.json\n├── mem0/\n│   └── ...\n├── memobase/\n│   └── ...\n├── evermemos/\n│   └── ...\n├── zep/\n│   └── ...\n└── llm/\n    └── openai/\n        └── gpt-4.1-mini/          # LLM results include model name in path\n            ├── answer_results_004.json\n            └── evaluation_results_004.json\n```\n\n## Analysis Tools\n\n`tools/analyze_results.py` analyzes evaluation results by question_id categories (major/minor/hierarchical). Supports single-file analysis and multi-batch aggregation.\n\n```bash\n# Single file analysis\npython tools/analyze_results.py eval/results/evermemos/evaluation_results_004.json\n\n# Aggregate all batches for a system\npython tools/analyze_results.py --system mem0\n\n# Specify results directory directly\npython tools/analyze_results.py --results-dir eval/results/memos/\n\n# Save JSON report\npython tools/analyze_results.py --system evermemos -o report.json\n\n# Quiet mode (JSON output only)\npython tools/analyze_results.py --system zep -o report.json -q\n```\n\n## Dataset Batches\n\nSupported user IDs: `004`, `005`, `010`, `011`, `016`\n\nEach batch has:\n- `dataset/{batch_id}/dialogue.json` - Conversation data\n- `dataset/{batch_id}/qa_{batch_id}.json` - QA questions for evaluation\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevermind-ai%2Fevermembench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevermind-ai%2Fevermembench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevermind-ai%2Fevermembench/lists"}