https://github.com/discovai/discovai-crawl

🕷️ DiscovAI Crawl API(🚧 Work in Progress 🚧): A powerful web scraping solution for AI tools and vector databases. Extract clean HTML, generate LLM-friendly content, and create embeddings from any URL.
https://github.com/discovai/discovai-crawl

ai api crawler embedding vector-database web-scraping

Last synced: 4 days ago
JSON representation

🕷️ DiscovAI Crawl API(🚧 Work in Progress 🚧): A powerful web scraping solution for AI tools and vector databases. Extract clean HTML, generate LLM-friendly content, and create embeddings from any URL.

• Host: GitHub
• URL: https://github.com/discovai/discovai-crawl
• Owner: DiscovAI
• License: apache-2.0
• Created: 2024-08-01T04:28:43.000Z (4 months ago)
• Default Branch: main
• Last Pushed: 2024-08-05T11:45:54.000Z (3 months ago)
• Last Synced: 2024-08-05T13:32:45.210Z (3 months ago)
• Topics: ai, api, crawler, embedding, vector-database, web-scraping
• Language: TypeScript
• Homepage:
• Size: 391 KB
• Stars: 10
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# DiscovAI Crawl API 🕷️🔍

> One API to scrape everything you need from URLs for your AI tool and vector database.

🚧 **Work in Progress** 🚧

## 🌟 Features

Our API provides a comprehensive suite of data extraction and processing capabilities:

- 🧼 Clean HTML (JavaScript and CSS removed)

- 📝 LLM-friendly Markdown conversion

- 🚫 Ad-free, cookie banner-free, and dialog-free content

- 📸 Website screenshots (auto-saved to AWS S3 or Cloudflare R2)

- 🤖 LLM-generated SEO-friendly content

- 🔑 LLM-extracted key information (summary, features, FAQs, etc.)

- 🧠 Ready-to-use embeddings for vector database integration (auto-saved to db)

## 🔧 Installation

```bash

pnpm i

cd apps/api && pnpm exec playwright install

```

## 🚀 Usage

```bash

pnpm dev

open http://localhost:3000

```

## 📦 API Response Structure

```json

{

  "clean_html": "...",

  "LLM_friendly_markdown": "...",

  "clean_text": "...",

  "screenshot_url": "...",

  "llm_extracts_key_info": {

    "what": "...",

    "summary": "...",

    "features": ["...", "..."],

    "faqs": [{"q": "...", "a": "..."}]

  },

  "llm_summarized_detail": "...",

  "embeddings": [...]

}

```

## 📚 Documentation

TODO

## 🤝 Contributing

TODO