https://github.com/jrazi/repo2file4gpt
This project aims to convert the content of GitHub repositories into a structured, machine-readable format, enabling AI models like ChatGPT to utilize them as a knowledge base.
https://github.com/jrazi/repo2file4gpt
collaborate github github-copilot gitkraken gpt information-retrieval knowledge-base llm
Last synced: 6 months ago
JSON representation
This project aims to convert the content of GitHub repositories into a structured, machine-readable format, enabling AI models like ChatGPT to utilize them as a knowledge base.
- Host: GitHub
- URL: https://github.com/jrazi/repo2file4gpt
- Owner: jrazi
- License: apache-2.0
- Created: 2023-12-17T15:56:27.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-05-13T07:21:42.000Z (about 2 years ago)
- Last Synced: 2024-05-13T08:33:07.788Z (about 2 years ago)
- Topics: collaborate, github, github-copilot, gitkraken, gpt, information-retrieval, knowledge-base, llm
- Language: Python
- Homepage:
- Size: 19.5 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# 📁 repo2file4gpt
## Expanding LLMs Knowledge through Public Codebases
repo2file4gpt facilitates the ingestion of open source GitHub repositories into AI systems, unlocking a vast set of technical knowledge for machine learning.
By structuring and aggregating content in an indexed markdown format, repo2file4gpt prepares the harvested open source intelligence for direct integration into large language models and Q&A systems. The full hierarchy from the original GitHub tree is preserved to retain useful context.
## Installation
You can install repo2file4gpt directly from PyPI:
```bash
pip install repo2file4gpt
```
## Quick Start
### Command Line Interface
After installing repo2file4gpt, you can use it from the command line as follows:
```bash
repo2file4gpt --token YOUR_GITHUB_TOKEN --repos user/repo1 user/repo2 --filetypes py js --output_dir ./outputs/
```
Replace `YOUR_GITHUB_TOKEN` with your actual GitHub token, and `user/repo1` and `user/repo2` with the actual repositories you want to process.
### Python Code
You can also use repo2file4gpt in your Python code:
```python
import repo2file4gpt
# Specify the GitHub token, list of repositories, file types, and output directory
token = "YOUR_GITHUB_TOKEN"
repos = ["user/repo1", "user/repo2"]
filetypes = ["py", "js"]
output_dir = "./outputs/"
# Create a RepositoryScraper instance
processor = repo2file4gpt.RepositoryScraper(token, filetypes, repo2file4gpt.LINE_LIMITS, output_dir)
# Process the repositories
processor.process_repositories(repos)
```
Again, replace `YOUR_GITHUB_TOKEN` with your actual GitHub token, and `user/repo1` and `user/repo2` with the actual repositories you want to process.
## TODO
- Add support for more file types.
- Improve error handling for robustness.
- Optimize performance for large repositories.
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.