{"id":30692742,"url":"https://github.com/teabranch/simple-semantic-chunker","last_synced_at":"2025-09-02T05:02:30.315Z","repository":{"id":297852490,"uuid":"998092792","full_name":"teabranch/simple-semantic-chunker","owner":"teabranch","description":"Embeddings based semantic chunker","archived":false,"fork":false,"pushed_at":"2025-06-07T22:05:18.000Z","size":15,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-02T05:02:25.033Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/teabranch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-07T21:07:23.000Z","updated_at":"2025-08-08T12:26:31.000Z","dependencies_parsed_at":"2025-06-07T22:51:56.845Z","dependency_job_id":null,"html_url":"https://github.com/teabranch/simple-semantic-chunker","commit_stats":null,"previous_names":["teabranch/semantic-chunker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/teabranch/simple-semantic-chunker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teabranch%2Fsimple-semantic-chunker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teabranch%2Fsimple-semantic-chunker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teabranch%2Fsimple-semantic-chunker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teabranch%2Fsimple-semantic-chunker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/teabranch","download_url":"https://codeload.github.com/teabranch/simple-semantic-chunker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teabranch%2Fsimple-semantic-chunker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273233234,"owners_count":25068731,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-02T05:01:44.700Z","updated_at":"2025-09-02T05:02:30.301Z","avatar_url":"https://github.com/teabranch.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Simple Semantic Chunker\n\n`simple-semantic-chunker` is a Python library designed to split text documents into semantically coherent chunks. This is particularly useful for preparing text for indexing in vector databases or for other NLP tasks that benefit from contextually grouped text segments.\n\nThe library leverages OpenAI's embedding models to understand the semantic meaning of sentences and groups them based on a configurable similarity threshold.\n\n## Features\n\n- Splits text into sentences.\n- Generates embeddings for sentences using specified OpenAI models.\n- Compares semantic similarity between consecutive sentences.\n- Groups sentences into chunks based on a similarity threshold.\n- Asynchronous support for document processing.\n- Allows customization of OpenAI model, API key, and base URL.\n\n## Installation\n\nYou can install `simple-semantic-chunker` from PyPI:\n\n```bash\npip install simple-semantic-chunker\n```\n\n## Usage\n\nHere's a basic example of how to use the `DocumentChunker`:\n\n```python\nimport asyncio\nfrom simple_semantic_chunker.chunker import DocumentChunker\n\nasync def main():\n    # Initialize the chunker\n    # You can specify your OpenAI API key and a custom base URL if needed\n    # chunker = DocumentChunker(openai_api_key=\"YOUR_API_KEY\", openai_base_url=\"YOUR_CUSTOM_ENDPOINT\")\n    chunker = DocumentChunker(openai_model=\"text-embedding-ada-002\", similarity_threshold=0.5)\n\n    document_text = \"\"\"\n    The quick brown fox jumps over the lazy dog. This sentence is about an animal.\n    The weather is sunny today. The sky is clear and blue. This is about the weather.\n    AI is transforming many industries. Machine learning models are becoming more powerful.\n    \"\"\"\n\n    print(f\"Processing document with model: {chunker.openai_model}\")\n\n    # Process the document asynchronously\n    chunks = await chunker.process_document(document_text)\n\n    print(f\"\\nGenerated {len(chunks)} chunks:\")\n    for i, chunk in enumerate(chunks):\n        print(f\"--- Chunk {i+1} ---\")\n        # The 'content' of a chunk is a list of sentences\n        print(\"Sentences:\", \" \".join(chunk['content']))\n        # print(\"Embedding:\", chunk['embedding'][:5], \"...\") # Print first 5 elements of the embedding\n        print(f\"Number of sentences in chunk: {len(chunk['content'])}\")\n        print(\"---\")\n\n    # Synchronous processing is also available:\n    # chunks_sync = chunker.process_document_sync(document_text)\n    # print(f\"\\nGenerated {len(chunks_sync)} chunks (synchronously):\")\n    # for i, chunk in enumerate(chunks_sync):\n    #     print(f\"--- Chunk {i+1} (sync) ---\")\n    #     print(\"Sentences:\", \" \".join(chunk['content']))\n    #     print(\"---\")\n\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n### Configuration\n\nWhen initializing `DocumentChunker`, you can specify:\n\n- `openai_model`: The OpenAI embedding model to use (e.g., `\"text-embedding-ada-002\"`, `\"text-embedding-3-small\"`). Defaults to `\"text-embedding-ada-002\"`.\n- `similarity_threshold`: A float between 0 and 1. Sentences with similarity below this threshold will start a new chunk. Defaults to `0.45`.\n- `logger`: An optional custom logger instance.\n- `openai_api_key`: Your OpenAI API key. If not provided, the library will attempt to use the `OPENAI_API_KEY` environment variable.\n- `openai_base_url`: A custom base URL for the OpenAI API (e.g., for use with Azure OpenAI or other compatible endpoints). If not provided, the library will attempt to use the `OPENAI_BASE_URL` environment variable or the default OpenAI API URL.\n\n\n## How it Works\n\n1.  **Sentence Splitting**: The input document is first split into individual sentences.\n2.  **Embedding Generation**: Each sentence is converted into a numerical vector (embedding) using the specified OpenAI model.\n3.  **Similarity Comparison**: The cosine similarity between the embedding of the current sentence and the previous sentence (or the representative embedding of the current chunk) is calculated.\n4.  **Chunk Creation**:\n    *   If the similarity is above the `similarity_threshold`, the current sentence is added to the current chunk.\n    *   If the similarity is below the threshold, the current chunk is finalized (its overall embedding is calculated from its constituent sentences), and a new chunk begins with the current sentence.\n5.  **Final Output**: The process results in a list of chunks, where each chunk contains a list of sentences and the embedding for the entire chunk.\n\nThe core idea is that sentences that are semantically similar will be grouped together. The `similarity_threshold` controls how \"tightly\" related sentences must be to stay in the same chunk.\n\n## Development \u0026 Contributing\n\nThis project is managed by TeaBranch.\n\n### Setup for Development\n\n```bash\ngit clone https://github.com/TeaBranch/simple-semantic-chunker.git # Replace with your repo URL\ncd simple-semantic-chunker\npython -m venv venv\nsource venv/bin/activate # or venv\\Scripts\\activate on Windows\npip install -r requirements.txt # (You'll need to create this: pip freeze \u003e requirements.txt)\npip install -e . # Install in editable mode\n```\n\n### Running Tests\n(Test setup to be added)\n\n### Publishing to PyPI (Manual)\nThis project is configured with a GitHub Action to automatically publish to PyPI when changes are merged to the `main` branch. For manual publishing:\n\n1.  Ensure `setuptools`, `wheel`, and `twine` are installed: `pip install setuptools wheel twine`\n2.  Increment the version in `setup.py`.\n3.  Build the package: `python setup.py sdist bdist_wheel`\n4.  Upload to PyPI: `twine upload dist/*` (You will need a PyPI account and API token).\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteabranch%2Fsimple-semantic-chunker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fteabranch%2Fsimple-semantic-chunker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteabranch%2Fsimple-semantic-chunker/lists"}