{"id":28635582,"url":"https://github.com/rowyio/llm-web-crawler","last_synced_at":"2025-07-15T06:43:35.189Z","repository":{"id":247910228,"uuid":"795625328","full_name":"rowyio/LLM-Web-Crawler","owner":"rowyio","description":"Web Scraper and Crawler for LLM Apps and AI Workflows with NoCode / LowCode. Plug and play with your own logic and customize it flexibly and scalably on BuildShip.","archived":false,"fork":false,"pushed_at":"2024-07-11T08:00:28.000Z","size":277,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-07-12T09:24:31.637Z","etag":null,"topics":["ai","automation","crawler","llm","lowcode","nocode","scraper","web","web-crawler","workflow"],"latest_commit_sha":null,"homepage":"https://llm-web-crawler.vercel.app","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rowyio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-03T17:17:23.000Z","updated_at":"2024-07-11T09:58:04.000Z","dependencies_parsed_at":"2024-07-11T09:18:04.222Z","dependency_job_id":"376b1165-14ad-4f5f-8754-082183f42f4d","html_url":"https://github.com/rowyio/LLM-Web-Crawler","commit_stats":null,"previous_names":["rowyio/llm-web-crawler"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rowyio/LLM-Web-Crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rowyio%2FLLM-Web-Crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rowyio%2FLLM-Web-Crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rowyio%2FLLM-Web-Crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rowyio%2FLLM-Web-Crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rowyio","download_url":"https://codeload.github.com/rowyio/LLM-Web-Crawler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rowyio%2FLLM-Web-Crawler/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259509474,"owners_count":22868837,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","automation","crawler","llm","lowcode","nocode","scraper","web","web-crawler","workflow"],"created_at":"2025-06-12T17:11:03.556Z","updated_at":"2025-06-12T17:11:04.353Z","avatar_url":"https://github.com/rowyio.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web Crawler for LLM Apps\n\nFlexible and Scalable low-code Web Crawler. Give it a try on the LIVE playground: https://llm-web-crawler.vercel.app/  \n  \nUses [BuildShip](https://buildship.com/?ref=llm-github) - a visual AI workflow builder to extract and gather data from your websites or sources. This data can then be used as a knowledge base to power your own LLM apps 🤖, or paired with BuildShip's [AI Assistant](https://docs.buildship.com/ai-assistant/assistant) to unlock powerful use cases and enhance your business or services.\n\n## Video Tutorial\n\n\u003ca href=\"https://www.youtube.com/watch?v=OJZzwcgjKW8\" target=\"_blank\"\u003e  \n\u003cimg width=\"600\" alt=\"yotube\" src=\"https://github.com/rowyio/LLM-Web-Crawler/assets/307298/8c2309ee-c753-4a57-b42c-eba2f0bbc809\"/\u003e\n\u003c/a\u003e\n\n## Features\n\n| Node           | Info                                                                                                                                                                                                                                                                                                                | Documentation                                                                | Template                                                                |\n| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------- |\n| Scrape         | Easy to get started with, scrape a given web URL and return the text content. Works great for less complex sites that don't rely on JavaScript to load.                                                                                                                                                             | [Read more](https://docs.buildship.com/utility-nodes/scrape-web-url)         | [Remix](https://buildship.app/remix?template=scrape-static-site)        |\n| Dynamic Scrape | Scrape a given web URL and return the text content. This method works well for more complex sites and allows for more interactive scraping by providing a set of steps to execute after loading the page. For example, loading an ecommerce site, searching for an item, and then scraping the search results info. | [Read more](https://docs.buildship.com/utility-nodes/scrape-web-url-dynamic) | [Remix](https://buildship.app/remix?template=scrape-dynamic-site)       |\n| Web Crawler    | Extract data from an entire website by crawling through and scraping all its pages. Perfect for aggregating data to create your own custom GPTs or \"Chat with Data\" apps.                                                                                                                                           | [Read more](https://docs.buildship.com/utility-nodes/crawler)                | [Remix](https://buildship.app/remix?template=gpt-crawler)               |\n| LLM Extraction | Extract structured data (just the data you care about) from any website. No need to scrape an entire webpage; simply specify the URL and the fields you want to extract. The LLM will handle the rest, delivering only the relevant data in a structured format.                                                    | [Read more](https://docs.buildship.com/utility-nodes/llm-extract)            | [Remix](https://buildship.app/remix?template=openai-extract-hackernews) |\n\n## How to use\n\n- First clone a template using most relevant to your using the following links.\n[LLM Extractor](https://buildship.app/remix?template=openai-extract-hackernews)  \n[Crawler](https://buildship.app/remix?template=gpt-crawler)  \n[Static Web Scraping](https://buildship.app/remix?template=scrape-static-site)  \n[Dynamic Web Scraping](https://buildship.app/remix?template=scrape-dynamic-site)  \n\n- Run or customize the template as per your usecase\n- Click Ship to deploy as an API or scheduled job\n\nRead full [documentation](https://docs.buildship.com/utility-nodes/llm-extract) to learn more\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frowyio%2Fllm-web-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frowyio%2Fllm-web-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frowyio%2Fllm-web-crawler/lists"}