https://github.com/hetav21/youtube-scraper
A YouTube channel scraper that extracts key statistics such as rank, subscribers, views, and more using Axios, Cheerio, and Express. It also tracks performance trends with daily, weekly, and monthly averages.
https://github.com/hetav21/youtube-scraper
api-client cherrio scraper youtube
Last synced: about 2 months ago
JSON representation
A YouTube channel scraper that extracts key statistics such as rank, subscribers, views, and more using Axios, Cheerio, and Express. It also tracks performance trends with daily, weekly, and monthly averages.
- Host: GitHub
- URL: https://github.com/hetav21/youtube-scraper
- Owner: Hetav21
- License: gpl-3.0
- Created: 2024-11-30T16:14:31.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-12-01T14:29:38.000Z (10 months ago)
- Last Synced: 2025-03-27T00:29:59.940Z (7 months ago)
- Topics: api-client, cherrio, scraper, youtube
- Language: TypeScript
- Homepage:
- Size: 52.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# YouTube Channel Data Scraper
A YouTube channel scraper that extracts key statistics such as rank, subscribers, views, and more using Axios, Cheerio, and Express. It also tracks performance trends with daily, weekly, and monthly averages.
## Scraping Functions
- **rank**: Extracts the rank of the YouTube channel.
- **youtuber**: Extracts the name of the YouTube channel.
- **subscribers**: Extracts the number of subscribers.
- **video_views**: Extracts the total number of video views.
- **category**: Extracts the category of the YouTube channel.
- **title**: Extracts the title of the YouTube channel.
- **uploads**: Extracts the number of uploads.
- **abbreviation**: Extracts the country abbreviation.
- **country**: Extracts the country of origin.
- **channel_type**: Extracts the type of the channel.
- **user_created**: Extracts the creation date of the user.
- **subscribers_last_30_days**: Extracts the number of subscribers gained in the last 30 days.
- **video_views_last_30_days**: Extracts the number of video views in the last 30 days.
- **daily_averages**: Extracts daily averages for subscribers and views.
- **weekly_averages**: Extracts weekly averages for subscribers and views.
- **monthly_averages**: Extracts monthly averages for subscribers and views.
## Installation
1. Clone the repository:
```sh
git clone https://github.com/Hetav21/youtube-scraper.git
cd youtube-scraper
```
2. Install the necessary dependencies:
```sh
npm install
```
3. (OPTIONAL) To override default environment variables:
Create a `.env` file in the `src/env` directory based on the `.env.example` file:
```sh
cp src/env/.env.example src/env/.env
```
Set the `URL` and `PORT` variable in the `src/env/.env` file to the YouTube channel URL you want to scrape.
For example:
```env
URL=https://socialblade.com/youtube/user/
PORT=3000
```
4. To test the scraper:
```sh
npm run dev
```
5. To build and start the scraper, use the following command:
```sh
npm run build
npm run start
```
## Usage
To use the scraper, send a GET request to
```url
http://localhost:3000
```
with the following body parameters:
```url
{
"youtubeId": "@channel_id"
}
```
where `channel_id` is the id of the YouTube channel you want to scrape.
NOTE: Alternatively, the default PORT can be changed using the `.env` file
## Project Structure
- **src/index.ts**: The entry point of the application.
- **src/customError.ts**: Contains a class to create custom error objects.
- **src/env/config.ts**: To load and validate environment variables from `./.env` or set the value to defaults.
- **src/params/request.ts**: Contains the URL and headers for the request.
- **src/params/schemas/**: Directory containing schemas for input validation of the request.
- **src/components/request.ts**: Function to fetch data from the URL.
- **src/components/scrapper.ts**: Main scraping logic.
- **src/components/elements/**: Directory containing functions to extract specific elements from the page.