{"id":13593234,"url":"https://github.com/flepied/second-brain-agent","last_synced_at":"2025-10-10T05:39:32.577Z","repository":{"id":177417443,"uuid":"655253031","full_name":"flepied/second-brain-agent","owner":"flepied","description":"🧠 Second Brain AI agent","archived":false,"fork":false,"pushed_at":"2025-08-19T11:30:46.000Z","size":1436,"stargazers_count":237,"open_issues_count":9,"forks_count":20,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-08-19T12:23:23.843Z","etag":null,"topics":["artificial-intelligence","chatgpt","chatgpt-api","chatgpt-bot","langchain","langchain-python","personal-knowledge-management","pkm","second-brain"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flepied.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-06-18T10:58:37.000Z","updated_at":"2025-08-11T21:29:52.000Z","dependencies_parsed_at":"2023-09-22T05:44:38.750Z","dependency_job_id":"6e3d073f-41f6-43f4-b0f3-20d8e8533cef","html_url":"https://github.com/flepied/second-brain-agent","commit_stats":null,"previous_names":["flepied/second-brain-agent"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/flepied/second-brain-agent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flepied%2Fsecond-brain-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flepied%2Fsecond-brain-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flepied%2Fsecond-brain-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flepied%2Fsecond-brain-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flepied","download_url":"https://codeload.github.com/flepied/second-brain-agent/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flepied%2Fsecond-brain-agent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002889,"owners_count":26083468,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","chatgpt","chatgpt-api","chatgpt-bot","langchain","langchain-python","personal-knowledge-management","pkm","second-brain"],"created_at":"2024-08-01T16:01:18.201Z","updated_at":"2025-10-10T05:39:32.572Z","avatar_url":"https://github.com/flepied.png","language":"Python","funding_links":[],"categories":["Knowledge Management","开源项目","chatgpt-api","Python","Open Source Projects","Learning","Knowledge Manager","Agent Categories"],"sub_categories":["知识管理","Knowledge Management","Repositories","\u003ca name=\"Unclassified\"\u003e\u003c/a\u003eUnclassified"],"readme":"[![](https://img.shields.io/endpoint?style=plastic\u0026url=https%3A%2F%2Fosstrack.io%2Fjson%2Ffreshness%2Fgithub%2Fflepied%2Fsecond-brain-agent%2Fmain)](https://osstrack.io/freshness/github/flepied/second-brain-agent/main)\u003cbr/\u003e[![](https://img.shields.io/badge/Incubated_by-100.builders-9146ff?logo=gamejolt\u0026logoColor=white\u0026labelColor=464646\u0026style=for-the-badge)](https://app.100.builders/directory)[![](https://img.shields.io/badge/Official_Selection-Artizen_Season_3-1acc6c?logo=image/svg+xml;base64,PHN2ZyB4bWxucz0naHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmcnIHdpZHRoPSc2NCcgaGVpZ2h0PSc2NCcgZmlsbD0nI2ZmZicgdmlld0JveD0nMCAwIDE2MCAxMjgnPjxwYXRoIGQ9J00xNTkuODk2IDExOC45ODRDMTU5Ljg5NiAxMTYuODgxIDE1OC4wODggMTE1LjA3OCAxNTUuOTc5IDExNS4wNzhDMTQ0LjAyOCAxMTUuMDc4IDEzMi4xNzggMTEzLjU3NiAxMjEuMDMxIDEwOS43NzFDMTE1LjEwNiAxMDcuNzY5IDEwOS4yODEgMTA0Ljc2NSAxMDQuMzYgMTAwLjc1OUMxMTQuNzA0IDkxLjg0NzUgMTIzLjU0MSA4MS4zMzM1IDEyOS43NjggNjkuMTE3MkMxMzIuODgxIDYyLjkwOSAxMzUuMzkyIDU1Ljg5OTcgMTM0LjQ4OCA0OC43OTAyQzEzMy4wODIgMzguNDc2NSAxMjQuNTQ2IDMyLjQ2ODUgMTE0LjUwMyAzMS44Njc3Qzk4LjIzNCAzMC41NjYgODQuMDczOSAzOS45Nzg1IDc3LjE0NDQgNTQuMzk3N0M3My43Mjk5IDYxLjQwNyA3Mi4yMjM1IDY5LjMxNzUgNzIuNzI1NyA3Ny4xMjc5QzczLjEyNzQgODUuNjM5MiA3Ni4xNDAyIDkzLjY0OTkgODAuODYwMiAxMDAuNDU5QzY4LjcwODYgMTA4LjA2OSA1NC44NDk4IDExMi44NzUgNDAuNjg5NyAxMTIuOTc2QzM3LjE3NDcgMTEyLjk3NiAzMy41NTk0IDExMi43NzUgMzAuMTQ0OSAxMTIuMDc0QzI2LjkzMTIgMTExLjM3MyAyMy43MTc2IDExMC4wNzIgMjEuNDA3OCAxMDcuNzY5QzE2LjY4NzcgMTAyLjg2MiAxNy4zOTA3IDk1LjY1MjUgMTguOTk3NiA4OS4xNDM5QzIyLjkxNDIgNzUuMTI1MiAzMS43NTE3IDYyLjYwODYgNDEuNTkzNSA1MS44OTQzQzUxLjAzMzYgNDEuNzgwOSA2NC43OTIgMzAuNDY1OCA3OC45NTIxIDI2Ljk2MTJDODMuMzcwOSAyNS44NTk3IDg2LjU4NDUgMjIuMTU0OCA4Ny4yODc1IDE3LjY0ODhDODcuNDg4NCAxNS45NDY1IDg3LjY4OTIgMTQuMjQ0MiA4Ny42ODkyIDEyLjQ0MTlDODcuNjg5MiAxMC42Mzk1IDg3LjM4NzkgNy42MzU0NiA4Ny4wODY2IDUuNTMyNjZDODYuNDg0MSAyLjEyODEzIDgzLjM3MDkgLTAuMjc1MDY2IDc5Ljg1NTkgMC4wMjUzMzNDNzcuMDQ0IDAuMjI1NiA3NC4yMzIxIDAuNzI2MjY2IDcxLjUyMDYgMS40MjcyQzU0LjQ0ODEgNi4wMzMzMyA0MC4wODcxIDE4LjA0OTMgMjguNzM4OSAzMS4wNjY2QzE0LjM3NzkgNDcuNjg4NyAwLjgyMDM3MyA3MS4yMiAwLjAxNjk2MjMgOTMuNjQ5OUMtMC4zODQ3NDMgMTA4LjI2OSA2LjM0MzgyIDEyMC43ODYgMjAuNzA0OCAxMjUuNTkyQzI1LjcyNjEgMTI3LjI5NSAzMS4wNDg3IDEyNy44OTUgMzYuMjcwOSAxMjcuOTk2QzU1LjY1MzIgMTI4LjE5NiA3NC40MzI5IDEyMS41ODcgOTAuNTAxMSAxMTEuMTczQzkzLjgxNTIgMTEzLjk3NyA5Ny41MzEgMTE2LjM4IDEwMS40NDggMTE4LjI4M0MxMTcuMzE1IDEyNi4zOTMgMTM1LjM5MiAxMjcuOTk2IDE1Mi44NjYgMTI3Ljc5NUMxNTMuNzcgMTI3Ljc5NSAxNTQuODc0IDEyNy43OTUgMTU1Ljk3OSAxMjcuNzk1QzE1OC4xODkgMTI3Ljc5NSAxNjAuMDk3IDEyNS45OTMgMTU5Ljk5NiAxMjMuNzlWMTE4Ljk4NEgxNTkuODk2Wk05Mi4yMDg0IDgyLjEzNDZDODkuMzk2NSA2OS45MTgzIDk1LjYyMjkgNTUuMDk4NiAxMDguMTc2IDUxLjQ5MzhDMTEwLjI4NSA1MC44OTMgMTEyLjU5NSA1MC41OTI2IDExNC44MDQgNTAuNjkyN0MxMTUuMzA2IDUwLjY5MjcgMTE1LjgwOSA1MC43OTI5IDExNi4yMSA1MC44OTNDMTE2LjYxMiA1MC45OTMxIDExNi44MTMgNTEuMDkzMyAxMTcuMTE0IDUxLjM5MzdDMTE3LjgxNyA1Mi4wOTQ2IDExNy41MTYgNTMuNzk2OSAxMTcuNDE1IDU1LjA5ODZDMTE2LjkxMyA1OC41MDMxIDExNS41MDcgNjIuMzA4MiAxMTMuOCA2NS41MTI1QzEwOS4wOCA3NC4yMjQgMTAyLjU1MiA4Mi41MzUxIDk0LjkxOTkgODkuNTQ0NEM5My43MTQ4IDg3LjI0MTQgOTIuODExIDg0LjczOCA5Mi4yMDg0IDgyLjEzNDZaJy8+PC9zdmc+\u0026labelColor=464646\u0026style=for-the-badge)](https://vote.artizen.fund/index/second-brain-ai-agent-1692884508130x285177834121461760)\n\n# 🧠 Second Brain AI agent\n\n## Introducing the Second Brain AI Agent Project: Empowering Your Personal Knowledge Management\n\nAre you overwhelmed with the information you collect daily? Do you often find yourself lost in a sea of markdown files, videos, web pages, and PDFs? What if there's a way to seamlessly index, search, and even interact with all this content like never before? Welcome to the future of Personal Knowledge Management: The Second Brain AI Agent Project.\n\n### 📝 Inspired by Tiago Forte's Second Brain Concept\n\nTiago Forte's groundbreaking idea of the Second Brain has revolutionized the way we think about note-taking. It’s not just about jotting down ideas; it's about creating a powerful tool that enhances learning and creativity. Learn more about Building a Second Brain by Tiago Forte [here](https://fortelabs.com/blog/basboverview/).\n\n### 💼 What Can the Second Brain AI Agent Project Do for You?\n\n1. Automated Indexing: No more manually sorting through files! Automatically index the content of your markdown files along with contained links, such as PDF documents, YouTube videos, and web pages.\n\n2. Smart Search Engine: Ask questions about your content, and our AI will provide precise answers, using the robust OpenAI Large Language Model. It’s like having a personal assistant that knows your content inside out!\n\n3. Effortless Integration: Whether you follow the Second Brain method or have your own unique way of note-taking, our system seamlessly integrates with your style, helping you harness the true power of your information.\n\n4. Enhanced Productivity: Spend less time organizing and more time innovating. By accessing your information faster and more efficiently, you can focus on what truly matters.\n\n### ✅ Who Can Benefit?\n\n* Professionals: Streamline your workflow and find exactly what you need in seconds.\n* Students: Make study sessions more productive by quickly accessing and understanding your notes.\n* Researchers: Dive deep into your research without getting lost in information overload.\n* Creatives: Free your creativity by organizing your thoughts and ideas effortlessly.\n\n### 🚀 Get Started Today\n\nDon't let your notes and content overwhelm you. Make them your allies in growth, innovation, and productivity. Join us in transforming the way you manage your personal knowledge and take the leap into the future.\n\n## Details\n\nIf you take notes using markdown files like in the Second Brain method or using your own way, this project automatically indexes the content of the markdown files and the contained links (pdf documents, youtube video, web pages) and allows you to ask question about your content using the OpenAI Large Language Model.\n\nThe system is built on top of the [LangChain](https://python.langchain.com/) framework and the [ChromaDB](https://www.trychroma.com/) vector store.\n\nThe system takes as input a directory where you store your markdown notes. For example, I take my notes with [Obsidian](https://obsidian.md/). The system then processes any change in these files automatically with the following pipeline:\n\n```mermaid\ngraph TD\nA[Markdown files from your editor]--\u003eB[Text files from markdown and pointers]--\u003eC[Text Chunks]--\u003eD[Vector Database]--\u003eE[Second Brain AI Agent]\n```\n\nFrom a markdown file, [transform_md.py](transform_md.py) extracts the text from the markdown file, then from the links inside the markdown file, it extracts pdf, url, youtube video and transforms them into text. There is some support to extract history data from the markdown files: if there is an `## History` section or the file name contains `History`, the file is split in multiple parts according to `\u003cday\u003e \u003cmonth\u003e \u003cyear\u003e` sections like `### 10 Sep 2023`.\n\nFrom these text files, [transform_txt.py](transform_txt.py) breaks these text files into chunks, create a vector embeddings and then stores these vector embeddings into a vector database.\n\nThe second brain agent uses the vector database to get context for asking the question to the large language model. This process is called [Retrieval-augmented generation (RAG)](https://python.langchain.com/docs/use_cases/question_answering/).\n\nIn reality, the process is more complex than a standard RAG. It is analyzing the question and then using a different chain according to the intent:\n\n```mermaid\nflowchart TD\n    A[Question] --\u003e C[/Get Intent/]\n    C --\u003e E[Summary Request] --\u003e EA[/Extract all the chunks/] --\u003e EB[/Summarize chunks/]\n    C --\u003e F[pdf or URL Lookup] --\u003e FA[/Extract URL/]\n    C --\u003e D[Activity report]\n    C --\u003e G[Regular Question]\n    D --\u003e DA[/Get Period metadata/] --\u003e DB[/Get Subject metadata/] --\u003e DC[/Extract Question without time/] --\u003e H[/Extract nearest documents\\nfrom the vector database\\nfiltered by the metadata/]\n    G --\u003e GA[/Step back question/] --\u003e GB[/Extract nearest documents\\nfrom the vector database/]\n    H --\u003e I[/Use the documents as context\\nto ask the question to the LLM/]\n    GB --\u003e I\n```\n\nTo be able to manipulate dates for activity reports. The system relies on some naming conventions. The first one is filenames containing `History`, `Journal` or `StatusReport` are considered journals with entries in this format: `## 02 Dec 2024` for each date. Other files can have an `## History` section with entries in this format: `### 02 Dec 2024` for each date.\n\nTo classify documents, the second brain agent uses a concept of a domain per document. The domain metadata is computed for each document by removing numbers and these strings: `At`, `Journal`, `Project`, `Notes` and `History`. This is handy if you use a documents named like `WorkoutHistory202412.md` then the domain is `Workout`.\n\nTo know which domain to use to filter documents, the second brain agent uses a special document that can be described in the `.env` files in the `SBA_ORG_DOC` variable and is defaulting to `SecondBrainOrganization.md`. This document describes the mapping between domains and other concepts if you want for example to separate work and personal activities.\n\n## MCP Server\n\nThe Second Brain Agent now includes an MCP (Model Context Protocol) server that provides programmatic access to the vector database and document retrieval system. This allows other applications to integrate with your second brain without interfacing at the reasoning level.\n\n### MCP Server Features\n\n* **Query Vector Database**: Ask questions and get answers from your indexed content\n* **Search Documents**: Perform semantic search across your documents with metadata filtering\n* **Document Management**: Get document counts, metadata, and list available domains\n* **Domain-based Search**: Search within specific domains (work, personal, etc.)\n* **Recent Documents**: Retrieve recently accessed documents\n\n### Using the MCP Server\n\n1. **Install the MCP server**:\n\n   ```bash\n   poetry add fastmcp\n   ```\n\n2. **Run the MCP server**:\n\n   ```bash\n   poetry run python mcp_server.py\n   ```\n\n3. **Test the server**:\n\n   ```bash\n   poetry run python test_mcp_server.py\n   ```\n\n4. **Configure MCP clients** using the `mcp_config.json` file:\n\n   ```json\n   {\n     \"mcpServers\": {\n       \"second-brain-agent\": {\n         \"command\": \"/your/path/to/second-brain-agent/mcp-server.sh\"\n       }\n     }\n   }\n   ```\n\n### Available MCP Tools\n\n* `search_documents`: Search for documents using semantic similarity\n* `get_document_count`: Get the total number of documents\n* `get_domains`: List all available domains\n* `get_recent_documents`: Get recently accessed documents\n\n## Installation\n\nYou need a Python 3 interpreter, [`poetry`](https://github.com/python-poetry/install.python-poetry.org) and the `inotify-tools` installed. All this has been tested under Fedora Linux 42 on my laptop and Ubuntu latest in the CI workflows. Let me know if it works on your system.\n\nGet the source code:\n\n```ShellSession\n$ git clone https://github.com/flepied/second-brain-agent.git\n```\n\nCopy the example .env file and edit it to suit your settings:\n\n```ShellSession\n$ cp example.env .env\n```\n\nInstall the dependencies using [poetry](https://python-poetry.org/):\n\n```ShellSession\n$ poetry install\n```\n\nThere is a bug between poetry, torch and pypi, to workaround just do:\n\n```ShellSession\n$ poetry run pip install torch\n```\n\nThen to use the created virtualenv, do:\n\n```ShellSession\n$ poetry shell\n```\n\n### systemd services\n\nTo install systemd services to manage automatically the different scripts when the operating system starts, use the following command (need sudo access):\n\n```ShellSession\n$ ./install-systemd-services.sh\n```\n\nTo see the output of the md and txt services:\n\n```ShellSession\n$ journalctl --unit=sba-md.service --user\n$ journalctl --unit=sba-txt.service --user\n```\n\n### Doing a similarity search with the vector database\n\n```ShellSession\n$ ./similarity.py \"What is LangChain?\" type=notes\n```\n\n### Searching for new connections between notes\n\nUse the vector store to find new connections between notes:\n\n```ShellSession\n$ ./smart_connections.py\n```\n\n### Launching the web UI\n\nLaunch this command to access the web UI:\n\n```ShellSession\n$ streamlit run second_brain_agent.py\n  You can now view your Streamlit app in your browser.\n\n  Local URL: http://localhost:8502\n  Network URL: http://192.168.121.112:8502\n```\n\nHere is an example:\n\n![Screenshot](screenshot.png \"Screenshot\")\n\n## Development\n\nInstall the extra dependencies using [poetry](https://python-poetry.org/):\n\n```ShellSession\n$ poetry install --with test\n```\n\nAnd then run the tests, like this:\n\n```ShellSession\n# Run all tests (unit + integration)\n$ poetry run pytest\n\n# Run only unit tests (no external dependencies required)\n$ poetry run pytest -m \"not integration\"\n\n# Run only integration tests (requires vector database)\n$ poetry run pytest -m integration\n\n# Run only unit tests (same as above, more explicit)\n$ poetry run pytest -m unit\n```\n\n**Note**: Integration tests require a running vector database and are automatically excluded during pre-commit hooks. Unit tests run without external dependencies and are suitable for CI/CD pipelines.\n\n### Full Integration Testing\n\nFor comprehensive testing of the entire system including the vector database and MCP server:\n\n```ShellSession\n$ ./integration-test.sh\n```\n\nThis script:\n\n* Sets up a complete test environment with ChromaDB\n* Processes test documents through the system\n* Runs pytest integration tests to validate MCP server functionality\n* Tests document lifecycle (create, modify, delete)\n* Provides end-to-end validation of the system\n\n**Note**: This requires docker-compose/podman-compose and will create temporary test data.\n\n### pre-commit\n\nBefore submitting a PR, make sure to activate [pre-commit](https://pre-commit.com/):\n\n```ShellSession\npoetry run pre-commit install\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflepied%2Fsecond-brain-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflepied%2Fsecond-brain-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflepied%2Fsecond-brain-agent/lists"}