Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/perebaj/newsletter
Why the software magicians guys don't have a newsletter?
https://github.com/perebaj/newsletter
concurrency golang mongodb scraper
Last synced: about 6 hours ago
JSON representation
Why the software magicians guys don't have a newsletter?
- Host: GitHub
- URL: https://github.com/perebaj/newsletter
- Owner: perebaj
- Created: 2024-01-03T16:56:07.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-01-24T14:00:19.000Z (10 months ago)
- Last Synced: 2024-06-19T18:01:00.396Z (5 months ago)
- Topics: concurrency, golang, mongodb, scraper
- Language: Go
- Homepage:
- Size: 16.8 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Newsletter
Some skilled engineers even have a blog site where they push some gold content, but they doesn't have yet, a way for their fan base to have recurrent access to this content. Newsletter try to circumvent it, scraping their pages and triggering e-mails for the guys who have an interest in those hidden gems.
## Architecture
## Environement Variables
The following environment variables are required to run the program:
- `LOG_LEVEL`: The level of the logs that will be printed. The values could be `DEBUG`, `INFO`, `WARNING` or `ERROR`.
- `LOG_TYPE`: The format of the logs that will be printed. The values could be `json` or `text`.
- `NL_MONGO_URI`: The URI of the MongoDB database that will be used to store the data.
- `NL_EMAIL_PASSWORD`: The password of the email that will be used to send the emails.
- `NL_EMAIL_USERNAME`: The user of the email that will be used to send the emails.
## Commands
`make help` - Show the available commands of this project. Using it, it's enoght to play around the project.
## Integration & Unit Tests
To run the integration tests, you need to have a MongoDB instance running in your machine. To do it, you can run the following command:
```bash
make dev/start
```
Access the dev container and run the tests:
```bash
make dev
make test
make integration-test
```
# Roadmap
This program aims to create the following features:
- Given a list of websites, that are located in a MongoDB collection, scrape the content of each website and save it in another MongoDB collection. ✅
- After the scraping, calculate the similarity between the new content and the previous content of each website, and update the MongoDB collection
with this information. ✅
- All the registered users will receive an email according to the URL that they have registered notifying them about news in their favorite engineers websites. ✅
- Create API routes to register new users and the websites that they want to follow. ⌛
Obs: All these flows will be trigerred by a cron job. ✅