https://github.com/kayyz1/llmscraper

LLM AI web scraper tool
https://github.com/kayyz1/llmscraper

openai puppeteer react remix typescript

Last synced: 14 days ago
JSON representation

LLM AI web scraper tool

• Host: GitHub
• URL: https://github.com/kayyz1/llmscraper
• Owner: kayYZ1
• Created: 2024-10-14T11:58:18.000Z (2 months ago)
• Default Branch: main
• Last Pushed: 2024-10-21T20:16:42.000Z (2 months ago)
• Last Synced: 2024-10-29T23:07:40.260Z (about 2 months ago)
• Topics: openai, puppeteer, react, remix, typescript
• Language: TypeScript
• Homepage:
• Size: 201 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        

# LLM Scraper

Project allows user to scrape dom content of the website and then allow LLM model to parse the content and return desired outcome.

## Features

- Scrape and parse website content (does not work for SPA)

- History (soon)

## Tech Stack

**Client:** Remix, React, Typescript, TailwindCSS, OpenAPI

**Server:** Remix (+NodeJS), Puppeteer

## Environment Variables

To run this project, you will need to add the following environment variables to your .env file

`OPEN_ROUTER`

or any other AI api provider in my case i used: https://www.openrouter.ai

## Run Locally

Clone the project

```bash

  git clone https://github.com/kayYZ1/LLMScraper.git

```

Go to the project directory

```bash

  cd LLMScraper

```

Install dependencies

```bash

  pnpm install

```

Start the server

```bash

  pnpm dev

```