{"id":32374665,"url":"https://github.com/lumpinif/deepcrawl","last_synced_at":"2025-10-24T22:41:49.951Z","repository":{"id":320404803,"uuid":"988419198","full_name":"lumpinif/deepcrawl","owner":"lumpinif","description":"100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy by yourself.","archived":false,"fork":false,"pushed_at":"2025-10-23T15:00:47.000Z","size":6046,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-23T17:07:40.323Z","etag":null,"topics":["ai-agent-tools","ai-sdk","better-auth","cloudflare-workers","crawling","deepcrawl","hono","html-to-markdown","links-extraction","links-tree","nextjs","orpc","typescript","web-scraping"],"latest_commit_sha":null,"homepage":"https://deepcrawl.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lumpinif.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-05-22T14:16:41.000Z","updated_at":"2025-10-23T15:00:51.000Z","dependencies_parsed_at":"2025-10-23T17:07:59.519Z","dependency_job_id":"309014f7-c1b3-4914-888d-90a2e47bb312","html_url":"https://github.com/lumpinif/deepcrawl","commit_stats":null,"previous_names":["lumpinif/deepcrawl"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/lumpinif/deepcrawl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lumpinif%2Fdeepcrawl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lumpinif%2Fdeepcrawl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lumpinif%2Fdeepcrawl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lumpinif%2Fdeepcrawl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lumpinif","download_url":"https://codeload.github.com/lumpinif/deepcrawl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lumpinif%2Fdeepcrawl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280878345,"owners_count":26406642,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-24T02:00:06.418Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agent-tools","ai-sdk","better-auth","cloudflare-workers","crawling","deepcrawl","hono","html-to-markdown","links-extraction","links-tree","nextjs","orpc","typescript","web-scraping"],"created_at":"2025-10-24T22:41:46.122Z","updated_at":"2025-10-24T22:41:49.943Z","avatar_url":"https://github.com/lumpinif.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deepcrawl\n\n\u003e WARNING: DO NOT USE DEEPCRAWL IN PRODUCTION RIGHT NOW AS IT IS SUBJECT TO CHANGE AND STILL UNDER RAPID DEVELOPMENT. USE WITH YOUR OWN RISK!\n\n**100% free and open-source Firecrawl alternative with better performance and flexibility.**\n\n![shots](./public/og.jpg)\n\nDeepcrawl is an agents-oriented website data context extraction platform. It extracts cleaned markdown of page content, agent-favoured hierarchical links tree and metadata that LLMs can digest with minimal token cost to reduce context switching and hallucination.\n\n\u003e Full Platform (Nextjs Dashboard, API Workers, Auth Workers, and Database) is open and transparent.\n\n## Documentation\n\nVisit https://deepcrawl.dev/docs to view the documentation.\n\n## Contributing\n\nPlease read the [contributing guide](./CONTRIBUTING.md).\n\n## License\n\n[![MIT License](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\nOpen Source. Open Code - built with ❤️ by [@felixLu](https://x.com/felixlu1018).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flumpinif%2Fdeepcrawl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flumpinif%2Fdeepcrawl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flumpinif%2Fdeepcrawl/lists"}