Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/aggie-coding-club/web-scraping-application

Web scraping project. We won awards.
https://github.com/aggie-coding-club/web-scraping-application

bootstrap bootstrap5 chromium mongodb node node-js nodejs puppeteer react reactjs vite vitejs web-scraping

Last synced: about 2 months ago
JSON representation

Web scraping project. We won awards.

Awesome Lists containing this project

README 

        
https://github.com/aggie-coding-club/Web-Scraping-Application/assets/59489624/988f8617-faaf-4a5a-be13-937861ca0099



# ACC Project: Web Scraping Application



## Introduction



The main goal of this project is to build an application designed to monitor website changes, such as price drops and new internship opportunities. Our vision is to create an application that allows users to set preferences and receive real-time notifications through various channels, maintaining a database of updated information for easy and future reference. This is the Result-Focused continuation of the past Learning-Focused project, the legacy repository can be found [here](https://github.com/Beketovian/wsapp).



## Teams and Responsibilities



If you are a member of the project, you have been assigned to one of the three teams listed:



### Frontend Team



The Frontend Team is tasked with creating a seamless and responsive user interface, ensuring an engaging and accessible experience across different devices.



### Backend Team



The Backend Team is in charge of the core application logic, data processing, and the management of the database, ensuring robust and efficient back-end support for the front-end interface.



### Web Scraping Team



The Web Scraping Team focuses on the logic behind how to extract data from any website. They are responsible for writing and maintaining scripts that gather data, ensuring the application has the latest content.



## Current TODOs:



- ~~Implementing a user authentication system.~~

- ~~Add scraping functionality and user-specific object~~

- ~~Implement a sophisticated table for the user's scraped sites dashboard~~

- Add more pages to enhance frontend UI/UX

- Develop a notification system and interval scraping options

- Deploy the application for public access.



## Contribution Guidelines



I welcome any member of this project to contribute to their respective team. Here is a general guideline on how to:



1. **Set Up Your Environment:**



   - Ensure Git is installed on your machine and your identity is configured.

   - Clone the repository to begin local development.

   - Watch the video posted in the discord on how to set up the database connection



2. **Make Changes:**



   - Branch off for new features or fixes.

   - Follow the project's coding standards and best practices.

   - Communicate in the discord with other collaborators and team managers



3. **Commit Changes:**



   - Use clear and descriptive commit messages.

   - Commit often with incremental changes.



4. **Push Changes:**



   - Push your commits to your branch.



5. **Open a Pull Request:**



   - Head to the repository on GitHub to open a pull request.

   - Provide a summary of your changes and submit the PR for review.



6. **Code Review:**

   - Engage with the feedback provided in the review process.

   - Once approved, a manager will merge your changes.



## Need Help?



If you're new to Git/GitHub or need a refresher, here are some resources to get you started:



- [Git Handbook](https://guides.github.com/introduction/git-handbook/)

- [GitHub Flow Guide](https://guides.github.com/introduction/flow/)

- [Creating a Pull Request](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request)



Here are some team-specific resources that may be helpful:



## Frontend:



- [Vite](https://vitejs.dev/guide)

- [React](https://react.dev/learn)

- [Bootstrap](https://react-bootstrap.netlify.app/docs/components/accordion)



## Backend:



- [Node JS](https://nodejs.org/en/learn)

- [Express JS](https://expressjs.com/)

- [MongoDB](https://www.mongodb.com/docs/manual/)



## Web Scraping:



- [Puppeteer](https://pptr.dev/)

- [Chromium & Puppeteer](https://medium.com/@devinred/introduction-to-scraping-and-automation-fed81d5944cf)



For additional support, please reach out to via the discord