https://github.com/thecollinsbyte/webscribe

The application allows users to input a URL, scrape data from the specified webpage, and store the results in a MongoDB database. WebScribe features a simple and intuitive web interface, making web scraping accessible to users of all technical levels.
https://github.com/thecollinsbyte/webscribe

docker expressjs mongodb nodejs olostep typescript web webcrawler

Last synced: about 1 month ago
JSON representation

The application allows users to input a URL, scrape data from the specified webpage, and store the results in a MongoDB database. WebScribe features a simple and intuitive web interface, making web scraping accessible to users of all technical levels.

• Host: GitHub
• URL: https://github.com/thecollinsbyte/webscribe
• Owner: TheCollinsByte
• Created: 2024-08-17T20:01:41.000Z (5 months ago)
• Default Branch: main
• Last Pushed: 2024-08-18T19:37:04.000Z (5 months ago)
• Last Synced: 2024-09-08T20:26:20.439Z (4 months ago)
• Topics: docker, expressjs, mongodb, nodejs, olostep, typescript, web, webcrawler
• Language: TypeScript
• Homepage:
• Size: 104 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        


 

WebScribe: Olostep Track

 



 




 



 



 



 



 

 



### Description:

WebScribe is a full-stack web scraping application built using TypeScript, Node.js, Express, MongoDB, and Docker. The application allows users to input a URL, scrape data from the specified webpage, and store the results in a MongoDB database. WebScribe features a simple and intuitive web interface, making web scraping accessible to users of all technical levels.

### Key Features:

* Real-Time Scraping: Users can input any valid URL to scrape the title and meta description of the webpage.

* Persistent Data Storage: Scraped data is stored in a MongoDB database for easy retrieval and management.

* Full-Stack Implementation: Built using Node.js and Express on the backend with EJS templating for the frontend.

* Dockerized Environment: Easily deployable with Docker, ensuring consistent environments across development and production.

* CI/CD Pipeline: Automated deployment pipeline using GitHub Actions for seamless integration and delivery.

* TypeScript Support: Written in TypeScript for enhanced type safety and code maintainability.

### Tech Stack:

- Frontend: HTML, CSS, EJS

- Backend: Node.js, Express.js

- Database: MongoDB (via Mongoose)

- Containerization: Docker, Docker Compose

- CI/CD: GitHub Actions

- Language: TypeScript

### Usage:

- Clone the repository.

- Start the MongoDB service using Docker Compose.

- Run the application with TypeScript.

- Access the web interface to start scraping!

### Setup Instructions:

- Detailed setup instructions can be found in the README.md file.

### Contributing:

- Contributions are welcome! Feel free to fork the repository, make changes, and submit a pull request.