Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/muhammedmagdyy/web-scraping-github
Learning and having fun with web scraping.
https://github.com/muhammedmagdyy/web-scraping-github
javascript
Last synced: 7 days ago
JSON representation
Learning and having fun with web scraping.
- Host: GitHub
- URL: https://github.com/muhammedmagdyy/web-scraping-github
- Owner: MuhammedMagdyy
- Created: 2024-05-24T16:57:58.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2024-06-27T18:38:26.000Z (6 months ago)
- Last Synced: 2024-06-28T14:11:53.058Z (6 months ago)
- Topics: javascript
- Language: JavaScript
- Homepage:
- Size: 72.3 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# GitHub Scraper
## Description 📄
This project involves developing a web scraper using Puppeteer to retrieve various features from a specified GitHub user profile, such as the number of followers, following, etc... The script is designed to navigate the user's GitHub profile page and systematically extract the desired information based on user-specified criteria.
## Features ✨
- Uses Puppeteer for web scraping and Inquirer for user input
- Fetches GitHub user profile information such as the number of followers, following, etc...
- Outputs the list of followers or following based on user choice
## Installation 🛠️
Please ensure you have [Node.js](https://nodejs.org/en/download/) installed on your machine.
1. Clone the repository:
```sh
git clone https://github.com/MuhammedMagdyy/web-scraping-github.git
cd web-scraping-github
```
2. Install the required dependencies:
```sh
npm install
```
## Usage 🚀
Run the script using the following command:
```sh
npm start
```
Then you will be prompted to enter the GitHub username you want to scrape, and then you will be asked to choose the information you want to retrieve (followers, following, etc...).
## Example output for a specific choice 📋
Assuming the user name is `example-user` and the user chooses to retrieve the list of followers, the output will be:
```
==============================================
GitHub Profile Information for example-user
==============================================
Username: example-user
Full Name: Example User
Bio: Example bio
Location: Cairo, Egypt
Company: Example Company
----------------------------------------------
Fetching followers from GitHub...
['follower1', 'follower2', 'follower3', 'follower4', 'follower5']
✅ Followers fetched successfully.
```
## Improvements and Future Work 🛠️
- Add a feature to save the output to a file 🔄
- Add a feature to fetch the list of repositories for a specified user 🔄
- Add a feature to fetch the list of starred repositories for a specified user 🔄
- Add a feature to fetch the list of organizations for a specified user 🔄
- Error handling for invalid usernames ✅
- Implement pagination to handle users with many followers 🔄
And more!
## Demo 🎥
https://github.com/MuhammedMagdyy/web-scraping-github/assets/60513866/5c09b2db-a055-47b0-aa46-4835c4784701
## Contributing 🤝
Contributions are welcome! Feel free to open a pull request for improvements or suggestions or contact me on [LinkedIn](https://www.linkedin.com/in/muhammedmagdyy/).
## Contact 📬
For any inquiries, don't hesitate to get in touch with me at my [email](mailto:[email protected]) or connect with me on [LinkedIn](https://www.linkedin.com/in/muhammedmagdyy/).
Happy coding! :rocket: