Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/developerjosh/gogo-crawler
The tool kit for making an anime website with a database full of anime
https://github.com/developerjosh/gogo-crawler
crawler crawler-js gogoanime gogoanime-api gogoanime-scraper
Last synced: about 2 months ago
JSON representation
The tool kit for making an anime website with a database full of anime
- Host: GitHub
- URL: https://github.com/developerjosh/gogo-crawler
- Owner: DeveloperJosh
- Created: 2024-07-30T19:40:22.000Z (5 months ago)
- Default Branch: master
- Last Pushed: 2024-08-08T02:46:51.000Z (5 months ago)
- Last Synced: 2024-08-08T23:08:25.180Z (5 months ago)
- Topics: crawler, crawler-js, gogoanime, gogoanime-api, gogoanime-scraper
- Language: JavaScript
- Homepage:
- Size: 93.8 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Gogo Crawler
Gogo Crawler is a web crawling tool designed to crawl the web and store the data in a MongoDB database.
## Table of Contents
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Setup](#setup)
- [Usage](#usage)
- [Credits](#credits)
## Prerequisites
Before you begin, ensure you have met the following requirements:
- You have installed [Node.js](https://nodejs.org/en/download/) and [Bun](https://bun.sh/)
- You have a running instance of [MongoDB](https://www.mongodb.com/)
## Installation
1. Clone the repository:
```bash
git clone https://github.com/DeveloperJosh/gogo-crawler.git
cd gogo-crawler
```
2. Install the dependencies:
```bash
bun install
```
## Setup
1. Create a `.env` file in the root of the project and add your MongoDB connection details:
```env
MONGO_URL=your_mongo_url
```
2. Ensure your MongoDB instance is running and accessible.
## Usage
To start the Gogo Crawler, run the following command:
```bash
bun crawl
```
And that's it! The crawler will start crawling the web and storing the data in your MongoDB database.
## Credits
This project was inspired by the [anime-crawler](https://github.com/riimuru/anime-crawler) project by [riimuru](https://github.com/riimuru).