https://github.com/lumpinif/deepcrawl

100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy by yourself.
https://github.com/lumpinif/deepcrawl

ai-agent-tools ai-sdk better-auth cloudflare-workers crawling deepcrawl hono html-to-markdown links-extraction links-tree nextjs orpc typescript web-scraping

Last synced: 8 months ago
JSON representation

100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy by yourself.

• Host: GitHub
• URL: https://github.com/lumpinif/deepcrawl
• Owner: lumpinif
• License: mit
• Created: 2025-05-22T14:16:41.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2025-10-23T15:00:47.000Z (8 months ago)
• Last Synced: 2025-10-23T17:07:40.323Z (8 months ago)
• Topics: ai-agent-tools, ai-sdk, better-auth, cloudflare-workers, crawling, deepcrawl, hono, html-to-markdown, links-extraction, links-tree, nextjs, orpc, typescript, web-scraping
• Language: TypeScript
• Homepage: https://deepcrawl.dev
• Size: 5.77 MB
• Stars: 3
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Agents: AGENTS.md

Awesome Lists containing this project

README

          
# Deepcrawl

> WARNING: DO NOT USE DEEPCRAWL IN PRODUCTION RIGHT NOW AS IT IS SUBJECT TO CHANGE AND STILL UNDER RAPID DEVELOPMENT. USE WITH YOUR OWN RISK!

**100% free and open-source Firecrawl alternative with better performance and flexibility.**

![shots](./public/og.jpg)

Deepcrawl is an agents-oriented website data context extraction platform. It extracts cleaned markdown of page content, agent-favoured hierarchical links tree and metadata that LLMs can digest with minimal token cost to reduce context switching and hallucination.

> Full Platform (Nextjs Dashboard, API Workers, Auth Workers, and Database) is open and transparent.

## Documentation

Visit https://deepcrawl.dev/docs to view the documentation.

## Contributing

Please read the [contributing guide](./CONTRIBUTING.md).

## License

[![MIT License](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)

Open Source. Open Code - built with ❤️ by [@felixLu](https://x.com/felixlu1018).