https://github.com/ndmen/scrapingbakery
This repository contains the implementation of a web scraping API designed to retrieve product information from a specified URL. The API is built using NestJS and employs asynchronous processing to handle requests efficiently.
https://github.com/ndmen/scrapingbakery
cache cheerio nestjs scraper
Last synced: about 1 month ago
JSON representation
This repository contains the implementation of a web scraping API designed to retrieve product information from a specified URL. The API is built using NestJS and employs asynchronous processing to handle requests efficiently.
- Host: GitHub
- URL: https://github.com/ndmen/scrapingbakery
- Owner: ndmen
- Created: 2024-01-25T11:10:43.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-01-26T19:29:16.000Z (over 2 years ago)
- Last Synced: 2025-08-02T01:24:15.521Z (11 months ago)
- Topics: cache, cheerio, nestjs, scraper
- Language: TypeScript
- Homepage:
- Size: 178 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
[circleci-image]: https://img.shields.io/circleci/build/github/nestjs/nest/master?token=abc123def456
[circleci-url]: https://circleci.com/gh/nestjs/nest
A progressive Node.js framework for building efficient and scalable server-side applications.
## Description
This repository contains the implementation of a web scraping API designed to retrieve product information from a specified URL. The API is built using NestJS and employs asynchronous processing to handle requests efficiently.
Features
- Receives requests containing a product ID and initiates asynchronous processing.
- Responds with an HTTP 200 status code and a unique process identifier upon request reception.
- Initiates the scraping process of the target URL and transforms the website data into a unified JSON format.
- Includes a 10-second timeout to simulate data processing.
- Responds with a "not ready" status if queried with the process identifier during the timeout period.
- Provides the final result via the same endpoint after the processing is complete.
Note
- This project uses NestJS and cache for processing purposes. In a real-world scenario, Redis would be used for processing, and PostgreSQL for storing results.
Data Retrieval Methods
To retrieve product information as per the requirements outlined in the task, the following methods were considered:
1. Open Graph in Meta Tags: Parsing meta tags with Open Graph protocol to extract product information.
2. Schema Parsing: Extracting product details from structured data using schema markup.
3. HTML Markup Parsing: Parsing HTML markup to identify and extract product information.
4. Script Tag Parsing: Extracting data from JavaScript scripts embedded within the HTML.
For the given task, the preferred method of data retrieval was Script Tag Parsing. This method was chosen because it provided the necessary information required by the task. Specifically, it allowed for the extraction of product identifiers and specifications required for further processing.
## Installation
```bash
$ npm install
```
## Running the app
```bash
# watch mode
$ npm run start:dev
```
## Using documentation
Open swagger http://localhost:3000/swagger/#/scraper/ScraperController_scrapeProduct and try to send post method with data:
```bash
{
"productId": "air-presto-mens-shoes-JlLlWz"
}
```
## Support
Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please [read more here](https://docs.nestjs.com/support).
## Stay in touch
- Author - [Kamil Myśliwiec](https://kamilmysliwiec.com)
- Website - [https://nestjs.com](https://nestjs.com/)
- Twitter - [@nestframework](https://twitter.com/nestframework)
## License
Nest is [MIT licensed](LICENSE).