{"id":18729257,"url":"https://github.com/postgresml/example-korvus-firecrawl","last_synced_at":"2025-04-10T09:46:05.035Z","repository":{"id":259987165,"uuid":"863741410","full_name":"postgresml/example-korvus-firecrawl","owner":"postgresml","description":null,"archived":false,"fork":false,"pushed_at":"2024-09-26T20:39:39.000Z","size":3,"stargazers_count":1,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-24T08:42:07.093Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/postgresml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-26T20:36:03.000Z","updated_at":"2024-12-05T08:31:12.000Z","dependencies_parsed_at":"2024-10-29T00:28:00.039Z","dependency_job_id":null,"html_url":"https://github.com/postgresml/example-korvus-firecrawl","commit_stats":null,"previous_names":["postgresml/example-korvus-firecrawl"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgresml%2Fexample-korvus-firecrawl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgresml%2Fexample-korvus-firecrawl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgresml%2Fexample-korvus-firecrawl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgresml%2Fexample-korvus-firecrawl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/postgresml","download_url":"https://codeload.github.com/postgresml/example-korvus-firecrawl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248196369,"owners_count":21063403,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T14:26:20.309Z","updated_at":"2025-04-10T09:46:05.007Z","avatar_url":"https://github.com/postgresml.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Korvus x Firecrawl Example\n\nThis example application demonstrates how to perform web crawling, semantic search, and Retrieval-Augmented Generation (RAG) using [Korvus](https://github.com/postgresml/korvus) and [Firecrawl](https://www.firecrawl.dev/).\n\n## Features\n\n- Web crawling using Firecrawl\n- Semantic search over crawled content\n- RAG (Retrieval-Augmented Generation) for question answering\n\n## Prerequisites\n\n- Python 3.7+\n- Firecrawl API key\n- PostgresML database URL\n\n## Installation\n\n1. Clone this repository:\n   ```\n   git clone https://github.com/postgresml/example-korvus-firecrawl\n   cd example-korvus-firecrawl\n   ```\n\n2. Install the required packages:\n   ```\n   pip install -r requirements.txt\n   ```\n\n3. Create a `.env` file in the project root and add your credentials:\n   ```\n   FIRECRAWL_API_KEY=your_firecrawl_api_key\n   KORVUS_DATABASE_URL=your_postgresml_database_url\n   CRAWL_URL=https://example.com\n   CRAWL_LIMIT=100\n   ```\n\n## Usage\n\nThe application supports three main actions: crawl, search, and rag.\n\n1. Crawl a website:\n   ```\n   python main.py crawl\n   ```\n\n2. Perform semantic search:\n   ```\n   python main.py search\n   ```\n\n3. Use RAG for question answering:\n   ```\n   python main.py rag\n   ```\n\nFor search and RAG, you'll be prompted to enter queries. Type 'q' to quit the input loop.\n\n## How it works\n\n1. The application uses Firecrawl to crawl the specified website and extract markdown content.\n2. Crawled data is processed and stored using Korvus.\n3. Semantic search allows you to find relevant documents based on your queries.\n4. RAG combines retrieved context with a language model to answer questions.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpostgresml%2Fexample-korvus-firecrawl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpostgresml%2Fexample-korvus-firecrawl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpostgresml%2Fexample-korvus-firecrawl/lists"}