{"id":13497010,"url":"https://github.com/spyglass-search/memex","last_synced_at":"2025-04-23T21:51:48.213Z","repository":{"id":173839493,"uuid":"651364737","full_name":"spyglass-search/memex","owner":"spyglass-search","description":"Super-simple, fully Rust powered \"memory\" (doc store + semantic search) for LLM projects, semantic search, etc.","archived":false,"fork":false,"pushed_at":"2023-10-09T20:56:08.000Z","size":4471,"stargazers_count":58,"open_issues_count":0,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-23T21:51:35.828Z","etag":null,"topics":["llm","rust","rust-lang","semantic-search"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spyglass-search.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-09T04:49:39.000Z","updated_at":"2025-03-26T00:57:50.000Z","dependencies_parsed_at":"2024-10-31T13:31:38.338Z","dependency_job_id":"4d9b3025-dabf-4f33-80b4-3ce6e70bc8b1","html_url":"https://github.com/spyglass-search/memex","commit_stats":null,"previous_names":["spyglass-search/memex"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spyglass-search%2Fmemex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spyglass-search%2Fmemex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spyglass-search%2Fmemex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spyglass-search%2Fmemex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spyglass-search","download_url":"https://codeload.github.com/spyglass-search/memex/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250522292,"owners_count":21444510,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm","rust","rust-lang","semantic-search"],"created_at":"2024-07-31T20:00:20.899Z","updated_at":"2025-04-23T21:51:48.182Z","avatar_url":"https://github.com/spyglass-search.png","language":"Rust","funding_links":[],"categories":["LLM Memory","rust","Memory"],"sub_categories":[],"readme":"# memex\n\nSuper simple \"memory\" and common functionality for LLM projects, semantic search, etc.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/memex-in-action.gif\"\u003e\n\u003c/p\u003e\n\n## Running the service\n\nNote that if you're running on Apple silicon (M1/M2/etc.), it's best to run natively (and faster)\nsince Linux ARM builds are very finicky.\n\n``` bash\n# Build and run the docker image. This will build \u0026 run memex as well as an opensearch\n# node for document storage + search.\n\u003e docker-compose up\n# OR run natively in you have the rust toolchain installed. This uses the default file\n# based vector store instead of opensearch which may yield worse results.\n\u003e cp .env.template .env\n\u003e cargo run --release -p memex serve\n# If everything is running correctly, you should see something like:\n2023-06-13T05:04:21.518732Z  INFO memex: starting server with roles: [Api, Worker]\n```\n\n## Using a LLM\nYou can use either OpenAI or a local LLM for LLM based functionality (such as the\nsummarization or extraction APIs).\n\nSet `OPENAI_API_KEY` to your API key in the `.env` file or set `LOCAL_LLM_CONFIG` to\na LLM configuration file. See `resources/config.llama2.toml` for an example. By\ndefault, a base memex will use the llama-2 configuration file.\n\n### Supported local models\n\nCurrently we have supported (and have tested) the following models:\n- Llama based models (llama 1 \u0026 2, Mistral, etc.) - *recommended*\n- Gptj (e.g. GPT4All)\n\n\n## Adding a document\n\nNOTE: If the `test` collection does not initially exist, it'll be created.\n\n``` bash\n\u003e curl http://localhost:8181/api/collections/test \\\n    -H \"Content-Type: application/json\" \\\n    --data @example_docs/state_of_the_union_2023.json\n{\n    \"time\": 0.123,\n    \"status\": \"ok\",\n    \"result\": {\n        \"taskId\": 1,\n        \"collection\": \"test\",\n        \"status\": \"Queued\",\n        ...\n    }\n}\n```\n\nFeel free to add as many documents as you want. Each one will be enqueued and processed\nas they are added.\n\nWait a couple seconds per document to be processed. You can check the status\nusing the `task_id` above like so:\n\n## Check task status\n\n``` bash\n\u003e curl http://localhost:8181/api/tasks/1\n{\n    \"time\": 0.123,\n    \"status\": \"ok\",\n    \"result\": {\n        \"taskId\": 1,\n        \"status\": \"Processing\"\n    }\n}\n```\n\nOr if it's finished, something like so:\n```bash\n{\n    \"time\": 0.123,\n    \"status\": \"ok\",\n    \"result\": {\n        \"taskId\": 1,\n        \"collection\": \"test\"\n        \"status\": \"Completed\",\n        \"createdAt\": \"2023-09-19T00:00:00Z\"\n    }\n}\n```\n\nOne the task is shown as \"Completed\", you can now run a query against the doc(s)\nyou've just added.\n\n## Run a search query\n\n``` bash\n\u003e curl http://localhost:8181/api/collections/test/search \\\n    -H \"Content-Type: application/json\" \\\n    -X GET \\\n    -d \"{\\\"query\\\": \\\"what does Biden say about taxes?\\\", \\\"limit\\\": 3}\"\n{\n    \"time\": 1.234,\n    \"status\": \"ok\",\n    \"result\": [{\n        \"_id\": \u003cinternal_id\u003e, // reference to this particular segment text.\n        \"document_id\": \u003cdocument UUID\u003e, // The original document that this came from.\n        \"segment\": \u003cdocument section\u003e,\n        \"content\": \u003ccontent block\u003e,\n        \"score\": \u003crelevancy score\u003e\n    }, ...]\n}\n```\n\n## Ask a question\n```bash\n\u003e curl http://localhost:8181/api/action/ask \\\n    -H \"Content-Type: application/json\" \\\n    -X POST \\\n    -d \"{\\\"text\\\": \\\"\u003ccontext if any\u003e\\\", \\\"query\\\": \\\"What is the airspeed velocity of an unladen swallow?\\\", \"json_schema\": { .. }}\"\n{\n    \"time\": 1.234,\n    \"status\": \"ok\",\n    \"result\": {\n        \"answer\": \"The airspeed velocity of an unladen swallow is...\"\n    }\n}\n\n```\n\n## Env variables\n\n- `HOST`: Defaults to `127.0.0.1`\n- `PORT`: Defaults to `8181`\n- `DATABASE_CONNECTION`: Connection URI for either an sqlite or postgres database\n- `VECTOR_CONNECTION`: Either `hnsw://\u003cpath\u003e` for a file-based vector store (but _very_ limited) or `opensearch+https://\u003curi\u003e` for OpenSearch support.\n\n## Examples\n\nFor any of these examples, make sure you have `memex` running in the background.\n\n### Clippy\n\n#### Ask questions about a single or many document(s)\n\n``` bash\n# In a different terminal, run memex\n\u003e cargo run --release -p memex serve\n# Make sure the LLM model is downloaded\n\u003e make setup-examples\n# In your main terminal\n\u003e cd examples/clippy\n# NOTE: there is no duplicate detection so running this twice will add the file twice.\n\u003e cargo run -- load-file example_docs/state_of_the_union.txt\n# To ask clippy about your files, use \"ask\"\n\u003e cargo run -- ask \"what does biden say about taxes?\"\n# To ask clippy without referring to it's memex and _ONLY_ relying on the knowledge\n# inside it's LLM, use \"qq\" / \"quick-question\"\n\u003e cargo run -- qq \"wget command to save a file to a directory\"\n# To clear clippy's memory\n\u003e cargo run -- forget\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspyglass-search%2Fmemex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspyglass-search%2Fmemex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspyglass-search%2Fmemex/lists"}