{"id":27471099,"url":"https://github.com/swish78/chnkr","last_synced_at":"2026-01-24T16:01:37.157Z","repository":{"id":270397496,"uuid":"910259635","full_name":"Swish78/chNkr","owner":"Swish78","description":null,"archived":false,"fork":false,"pushed_at":"2025-01-27T02:53:27.000Z","size":12,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-27T03:25:33.534Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Cython","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Swish78.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-30T20:17:51.000Z","updated_at":"2025-01-27T02:53:30.000Z","dependencies_parsed_at":"2024-12-30T21:26:46.004Z","dependency_job_id":"253dd603-6209-4fa3-aeee-d5a31506069b","html_url":"https://github.com/Swish78/chNkr","commit_stats":null,"previous_names":["swish78/chnkr"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Swish78%2FchNkr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Swish78%2FchNkr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Swish78%2FchNkr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Swish78%2FchNkr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Swish78","download_url":"https://codeload.github.com/Swish78/chNkr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249182790,"owners_count":21226123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-16T01:39:03.520Z","updated_at":"2026-01-24T16:01:32.019Z","avatar_url":"https://github.com/Swish78.png","language":"Cython","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ChNkr\n\n**ChNkr** is a high-performance NLP library designed to enhance text chunking for large language models (LLMs) and retrieval-augmented generation (RAG) workflows. Built with speed and flexibility in mind, ChNkr supports multiple chunking styles, overlap options, and seamless integration with popular vector databases like Chroma, Pinecone, and Neo4j.\n\n## Features\n\n- **Flexible Chunking Styles**: Choose from fixed-token, semantic-aware, or custom chunking methods.\n- **Overlap Support**: Ensure context continuity between chunks with adjustable overlap settings.\n- **Vector Store Integration**: Direct support for Chroma, Pinecone, and Neo4j.\n- **Custom Models**: Support for user-provided semantic models for advanced chunking customization.\n- **Blazing Fast Performance**: Built using Cython/C++ for maximum speed.\n- **Ease of Use**: Python-friendly API with detailed documentation and examples.\n\n## Installation\n\n### Prerequisites\n- Python 3.8+\n- GCC/Clang (for building Cython/C++ components)\n- Pip or equivalent package manager\n\n### Install from PyPI\n```bash\npip install chnkr\n```\n\n### Build from Source\n```bash\ngit clone https://github.com/yourusername/chnkr.git\ncd chnkr\npip install .\n```\n\n## Quick Start\n\n### Basic Usage\n```python\nfrom chnkr import Chunker\n\n# Initialize the Chunker with your preferred style\nchunker = Chunker(style=\"semantic\", overlap=50)\n\n# Chunk a sample text\ntext = \"Your large text input here...\"\nchunks = chunker.chunk(text)\nprint(chunks)\n```\n\n### Integration with Chroma\n```python\nfrom chnkr import Chunker\nfrom chromadb.client import Client\n\n# Initialize ChNkr\nchunker = Chunker(style=\"fixed\", max_tokens=100, overlap=20)\n\n# Chunk your text\ntext = \"Your document text here...\"\nchunks = chunker.chunk(text)\n\n# Push to Chroma\ndb_client = Client()\ncollection = db_client.create_collection(\"my_collection\")\nfor chunk in chunks:\n    collection.add(document=chunk, metadata={\"source\": \"doc_1\"})\n```\n\n### Integration with Pinecone\n```python\nfrom chnkr import Chunker\nimport pinecone\n\n# Initialize Pinecone\npinecone.init(api_key=\"your-api-key\", environment=\"us-west1-gcp\")\nindex = pinecone.Index(\"my-index\")\n\n# Initialize ChNkr\nchunker = Chunker(style=\"semantic\", overlap=50)\n\n# Chunk your text\ntext = \"Your document text here...\"\nchunks = chunker.chunk(text)\n\n# Push to Pinecone\nfor chunk in chunks:\n    index.upsert([(chunk.id, chunk.vector)])\n```\n\n### Integration with Neo4j\n```python\nfrom chnkr import Chunker\nfrom neo4j import GraphDatabase\n\n# Initialize Neo4j driver\ndriver = GraphDatabase.driver(\"bolt://localhost:7687\", auth=(\"neo4j\", \"password\"))\n\n# Initialize ChNkr\nchunker = Chunker(style=\"fixed\", max_tokens=200)\n\n# Chunk your text\ntext = \"Your document text here...\"\nchunks = chunker.chunk(text)\n\n# Push to Neo4j\ndef add_chunk(tx, chunk):\n    tx.run(\"CREATE (c:Chunk {content: $content, metadata: $metadata})\", content=chunk.content, metadata=chunk.metadata)\n\nwith driver.session() as session:\n    for chunk in chunks:\n        session.write_transaction(add_chunk, chunk)\n```\n\n## Configuration Options\n\n| Parameter      | Description                                       | Default Value |\n|----------------|---------------------------------------------------|---------------|\n| `style`        | Chunking style (`fixed`, `semantic`, `custom`)    | `fixed`       |\n| `max_tokens`   | Maximum tokens per chunk (for `fixed` style)      | `100`         |\n| `overlap`      | Overlap size between chunks (in tokens)           | `0`           |\n| `embedding_dim`| Embedding size for semantic chunking              | `768`         |\n\n## Development\n\n### Setting Up for Development\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/yourusername/chnkr.git\n   cd chnkr\n   ```\n2. Install dependencies:\n   ```bash\n   pip install -r requirements.txt\n   ```\n3. Build the Cython/C++ components:\n   ```bash\n   python setup.py build_ext --inplace\n   ```\n\n### Running Tests\n```bash\npytest tests/\n```\n\n## Roadmap\n\n- Add support for more vector stores (e.g., Weaviate, Redis).\n- Implement advanced chunking styles (e.g., topic-based chunking, language-aware chunking).\n- Extend support for additional languages beyond English.\n- Add CLI support for quick operations from the terminal.\n\n## Contributing\n\nWe welcome contributions! Please read our [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to contribute.\n\n## License\n\nChNkr is licensed under the MIT License. See [LICENSE](LICENSE) for details.\n\n## Acknowledgements\n\n- [Faiss](https://github.com/facebookresearch/faiss) for semantic chunking.\n- [Pinecone](https://www.pinecone.io/) and [Chroma](https://www.trychroma.com/) for vector store integration.\n- The open-source NLP community for inspiration and support.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswish78%2Fchnkr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fswish78%2Fchnkr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswish78%2Fchnkr/lists"}