Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sesaquecruz/dev-challenge-scraper-service
The Scraper Service solution for a Dev Challenge.
https://github.com/sesaquecruz/dev-challenge-scraper-service
nodejs rabbitmq typescript web-scraper
Last synced: 23 days ago
JSON representation
The Scraper Service solution for a Dev Challenge.
- Host: GitHub
- URL: https://github.com/sesaquecruz/dev-challenge-scraper-service
- Owner: sesaquecruz
- License: mit
- Created: 2024-01-23T22:26:45.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2024-01-29T01:14:49.000Z (11 months ago)
- Last Synced: 2024-01-30T01:29:32.123Z (11 months ago)
- Topics: nodejs, rabbitmq, typescript, web-scraper
- Language: TypeScript
- Homepage:
- Size: 156 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Scraper Service - Dev Challenge
This project is part of a solution for a Dev Challenge that consists of creating a system to allow MEIs to receive their DAS directly by email.
The entire solution systems include:
- [**App Service**](https://github.com/sesaquecruz/dev-challenge-app-service): Provides a REST API to receive MEI and DAS information. The received data is then forwarded to the Scraper Service through RabbitMQ.
- **Scraper Service**: Retrieves the data from RabbitMQ, scrapes the DAS from the PGMEI site, and then sends it by email.
## Diagram
## Messaging
### Data Format
- From RabbitMQ queue `email.das.queue`:
```
{
"mei": {
"cnpj": "66.516.671/0001-18",
"email": "[email protected]"
},
"das": {
"year": 2024,
"month": 1
}
}
```
## Installation using Docker Image from GitHub
1. Clone this repository:
```
git clone https://github.com/sesaquecruz/dev-challenge-scraper-service
```
2. Enter the project directory:
```
cd dev-challenge-scraper-service
```
3. Run the docker compose to start the services:
```
docker compose -f docker-compose-demo.yml up -d
```
To stop all services, run: `docker compose -f docker-compose-demo.yml down`.
## Installation building Docker Image
1. Clone this repository:
```
git clone https://github.com/sesaquecruz/dev-challenge-scraper-service
```
2. Enter the project directory:
```
cd dev-challenge-scraper-service
```
3. Run the docker compose to start the infrastructure services:
```
docker compose --profile infra up -d
```
4. Run the docker compose to start the application service:
```
docker compose --profile prod up -d --build
```
To stop all services, run: `docker compose --profile infra --profile prod down`.
## Usage
1. Access the App Service to send the DAS information:
```
http://localhost:8080/api/v1/docs/
```
2. Access the Email Server to visualize emails received:
```
http://localhost:1080/
```
The RabbitMQ exchanges and queues can be accessed at `http://localhost:15672/`. Use `guest` as the username and password.
## Contributing
Contributions are welcome! If you find a bug or would like to suggest an enhancement, please make a fork, create a new branch with the bugfix or feature, and submit a pull request.
## License
This project is licensed under the MIT License. See [LICENSE](./LICENSE) file for more information.