https://github.com/davidwrossiter/langchunk

Source code for chunking code in multiple different languages
https://github.com/davidwrossiter/langchunk

chunking chunking-algorithm embedding llm-context vectorization

Last synced: about 1 month ago
JSON representation

Source code for chunking code in multiple different languages

• Host: GitHub
• URL: https://github.com/davidwrossiter/langchunk
• Owner: davidwrossiter
• Created: 2024-06-17T17:26:06.000Z (7 months ago)
• Default Branch: main
• Last Pushed: 2024-06-18T11:19:19.000Z (7 months ago)
• Last Synced: 2024-06-18T20:00:38.489Z (7 months ago)
• Topics: chunking, chunking-algorithm, embedding, llm-context, vectorization
• Language: JavaScript
• Homepage:
• Size: 143 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Lang chunk

> I created langchunk to chunk entire codebases correctly.

RAG systems rely on data that has been properly chunked and embedded. Langchunk allows you to create high-quality chunks from any codebase including metadata about the file structure of the directory and the content of each file within it.

---

**Current capabilities**

- Allows for chunking of javascript source code.

- Chunks based on Function declaration, class declaration, variable declaration, and expression statements.

**Steps for future versions**

- Allow for different languages

- Generate embeddings for vector databases