https://github.com/leandro-bertoluzzi/scrap-to-api

API to get the content of web pages via scraping
https://github.com/leandro-bertoluzzi/scrap-to-api

docker docker-compose nodejs puppeteer scraper

Last synced: 3 months ago
JSON representation

API to get the content of web pages via scraping

• Host: GitHub
• URL: https://github.com/leandro-bertoluzzi/scrap-to-api
• Owner: Leandro-Bertoluzzi
• License: mit
• Created: 2022-04-21T23:06:11.000Z (about 4 years ago)
• Default Branch: master
• Last Pushed: 2022-09-25T01:45:02.000Z (almost 4 years ago)
• Last Synced: 2025-01-07T07:14:31.337Z (over 1 year ago)
• Topics: docker, docker-compose, nodejs, puppeteer, scraper
• Language: JavaScript
• Homepage:
• Size: 113 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

          
Scrap to API




  

  

  

  





	🚧 Scrap to API 🚀 Under construction...  🚧







  About   |  

  Features   |  

  Technologies   |  

  Requirements   |  

  Starting   |  

  Development   |  

  License   |  

  Author






## :dart: About

Proxy API to get info from MAL (My Anime List) website via web scraping.

## :sparkles: Features

:heavy_check_mark: REST API\

:heavy_check_mark: Scraping of websites via Puppeteer

## :rocket: Technologies

The following tools were used in this project:

-   [Node.js](https://nodejs.org/en/)

-   [Docker](https://www.docker.com/)

-   [Puppeteer](https://github.com/puppeteer/puppeteer)

## :white_check_mark: Requirements

Before starting :checkered_flag:, you need to have [Node](https://nodejs.org/en/) installed. You would also need [Docker](https://www.docker.com/) if you want to run the containers.

## :checkered_flag: Starting

```bash

# Clone this project

$ git clone https://github.com/Leandro-Bertoluzzi/scrap-to-api

# Access

$ cd scrap-to-api

# Set up environment variables

$ cp .env.example .env

# Edit .env and adjust values as needed

# Install dependencies

$ npm install

# Run the project (option 1: With Docker compose)

$ docker-compose up

# Run the project (option 2: Locally with Node.js)

$ npm run start:dev

# In any case, the API will initialize in 

```

### Environment configuration

The project uses environment variables for configuration. You can set these in a `.env` file at the root of the project. An example `.env.example` file is provided for reference.

The environment variables include:

| Variable        | Description                                             | Default Value             |

| --------------- | ------------------------------------------------------- | ------------------------- |

| `NODE_ENV`      | The environment mode (development, production, etc.)    | `development`             |

| `HOST`          | The host address for the API server                     | `0.0.0`                   |

| `PORT`          | The port number for the API server                      | `8000`                    |

| `MAL_BASE_URL`  | The base URL for MyAnimeList                            | `https://myanimelist.net` |

## Development

### Testing

The project includes unit and integration tests. To run the tests, use the following commands:

```bash

# Run unit tests

$ npm run test:unit

# Run integration tests

$ npm run test:integration

# Run all tests

$ npm test

# Run tests with coverage report

$ npm run test:coverage

```

## :memo: License

This project is under license from MIT. For more details, see the [LICENSE](LICENSE.md) file.

Made with :heart: by Leandro Bertoluzzi

Back to top