https://github.com/mendableai/llmstxt-generator
https://github.com/mendableai/llmstxt-generator
Last synced: 8 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/mendableai/llmstxt-generator
- Owner: mendableai
- Created: 2024-11-19T20:55:56.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2025-03-31T19:34:44.000Z (3 months ago)
- Last Synced: 2025-06-12T17:57:39.406Z (8 days ago)
- Language: TypeScript
- Homepage: https://llmstxt.firecrawl.dev/
- Size: 284 KB
- Stars: 412
- Watchers: 2
- Forks: 42
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome - mendableai/llmstxt-generator - (TypeScript)
- StarryDivineSky - mendableai/llmstxt-generator - generator，它是一个使用大型语言模型（LLM）来生成文本的工具。该工具的核心特色是能够通过简单的提示词，快速生成高质量、多样化的文本内容。它基于transformers库和PyTorch，允许用户自定义模型和参数，以满足不同的文本生成需求。项目提供了易于使用的API和示例代码，方便开发者快速上手。其工作原理是利用预训练的LLM模型，根据用户输入的提示词，生成连贯且具有上下文的文本。此外，该项目还支持多种文本生成策略，例如贪婪搜索、束搜索等，以控制生成文本的多样性和质量。用户可以根据自己的需求选择合适的生成策略。总而言之，这个项目旨在简化LLM文本生成过程，让开发者能够更轻松地利用LLM的力量。 (A01_文本生成_文本对话 / 大语言对话模型及数据)
README
# llms.txt Generator 🚀
Generate consolidated text files from websites for LLM training and inference. Powered by [@firecrawl_dev](https://twitter.com/firecrawl_dev) for web crawling and GPT-4-mini for text processing.
## Features
- Crawls websites and combines content into a single text file
- Generates both standard (`llms.txt`) and full (`llms-full.txt`) versions
- Web interface and API access available
- No API key required for basic usage
## Usage
### Web Interface
Visit [llmstxt.firecrawl.dev](https://llmstxt.firecrawl.dev) to generate files through the browser.
### API Endpoint
```
GET https://llmstxt.firecrawl.dev/[YOUR_URL_HERE]
```
Note: Processing may take several minutes due to crawling and LLM operations.
## Local Development
### Prerequisites
Create a `.env` file with the following variables:
```
FIRECRAWL_API_KEY=
SUPABASE_URL=
SUPABASE_KEY=
OPENAI_API_KEY=
```
### Installation
```bash
npm install
npm run dev
```