https://github.com/tharunoptimus/crawlerbot
Crawler Bot 🕷 to crawl websites for URLs and Image URLs and add it to MongoDB Database
https://github.com/tharunoptimus/crawlerbot
api cheerio expressjs mongodb node-fetch pug
Last synced: 8 months ago
JSON representation
Crawler Bot 🕷 to crawl websites for URLs and Image URLs and add it to MongoDB Database
- Host: GitHub
- URL: https://github.com/tharunoptimus/crawlerbot
- Owner: tharunoptimus
- Created: 2021-07-17T08:31:27.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2021-07-17T08:31:32.000Z (over 4 years ago)
- Last Synced: 2025-01-09T06:51:23.616Z (about 1 year ago)
- Topics: api, cheerio, expressjs, mongodb, node-fetch, pug
- Language: JavaScript
- Homepage: https://crawlbot.onrender.com
- Size: 97.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# Crawl websites for URLs and Images and add it to your database
## Config
Replace the database name, username and password with your own credentials in the database connection string in the database.js
## Install
`npm install` to install dependencies
## Run
`npm start` to start the application in dev server
## Usage
1. Access the '/' page or '/crawl' page to render the crawl page
2. Input the URL to crawl
3. Click on the crawl button
4. The crawl will start and the crawl status will be displayed
5. The 'Crawl Success' box will appear if the crawl is successful.