{"id":30116329,"url":"https://github.com/edmolima/maracanaunews","last_synced_at":"2026-04-18T01:03:53.226Z","repository":{"id":308920350,"uuid":"1034540431","full_name":"edmolima/maracanaunews","owner":"edmolima","description":"Distributed, serverless news scraper from Maracanaú. AWS Lambda, SQS, DynamoDB, EventBridge, Terraform. Secure, open source.","archived":false,"fork":false,"pushed_at":"2025-08-08T17:44:34.000Z","size":17,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-08T18:24:05.265Z","etag":null,"topics":["aws","cloudwatch","dynamodb","eventbridge","lambda","nodejs","serverless","sqs","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/edmolima.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"edmolima","buy_me_a_coffee":"edmolima"}},"created_at":"2025-08-08T14:55:42.000Z","updated_at":"2025-08-08T17:44:38.000Z","dependencies_parsed_at":"2025-08-08T18:24:08.702Z","dependency_job_id":"8ee301d8-1e93-489e-a9a7-b888d25b20e9","html_url":"https://github.com/edmolima/maracanaunews","commit_stats":null,"previous_names":["edmolima/maracanaunews"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/edmolima/maracanaunews","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edmolima%2Fmaracanaunews","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edmolima%2Fmaracanaunews/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edmolima%2Fmaracanaunews/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edmolima%2Fmaracanaunews/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/edmolima","download_url":"https://codeload.github.com/edmolima/maracanaunews/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edmolima%2Fmaracanaunews/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31952208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"ssl_error","status_checked_at":"2026-04-18T00:39:20.671Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloudwatch","dynamodb","eventbridge","lambda","nodejs","serverless","sqs","terraform"],"created_at":"2025-08-10T09:10:25.965Z","updated_at":"2026-04-18T01:03:53.209Z","avatar_url":"https://github.com/edmolima.png","language":"HCL","funding_links":["https://github.com/sponsors/edmolima","https://buymeacoffee.com/edmolima","https://www.buymeacoffee.com/edmolima"],"categories":[],"sub_categories":[],"readme":"# Serverless Scraper AWS with Terraform\n\n\n[![Lambdas CI](https://github.com/edmolima/maracanaunews/actions/workflows/lambda.yml/badge.svg)](https://github.com/edmolima/maracanaunews/actions/workflows/lambda.yml)\n[![Infra CI](https://github.com/edmolima/maracanaunews/actions/workflows/infra.yml/badge.svg)](https://github.com/edmolima/maracanaunews/actions/workflows/infra.yml)\n[![Quality Gates](https://github.com/edmolima/maracanaunews/actions/workflows/quality.yml/badge.svg)](https://github.com/edmolima/maracanaunews/actions/workflows/quality.yml)\n[![Codecov Coverage](https://img.shields.io/codecov/c/github/edmolima/maracanaunews?label=coverage)](https://app.codecov.io/gh/edmolima/maracanaunews)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![Open in VS Code](https://img.shields.io/badge/Open%20in-VS%20Code-blue?logo=visualstudiocode)](https://open.vscode.dev/edmolima/maracanaunews)\n[![Last Commit](https://img.shields.io/github/last-commit/edmolima/maracanaunews)](https://github.com/edmolima/maracanaunews/commits/main)\n[![Issues](https://img.shields.io/github/issues/edmolima/maracanaunews)](https://github.com/edmolima/maracanaunews/issues)\n[![GitHub Sponsors](https://img.shields.io/github/sponsors/edmolima?label=sponsor\u0026logo=github)](https://github.com/sponsors/edmolima)\n[![Buy Me a Coffee](https://img.shields.io/badge/Buy%20Me%20a%20Coffee-donate-yellow?logo=buymeacoffee)](https://www.buymeacoffee.com/edmolima)\n\nThis repository contains infrastructure as code (Terraform) and Lambda code examples for a distributed news scraper from maracanet.com (Maracanaú).\n\n## Features\n- Modular, production-ready serverless architecture\n- Fully automated deployment with GitHub Actions (OIDC, least privilege)\n- Clean, idempotent scraping logic\n- Minimal AWS cost configuration\n- English documentation and code\n- Open source ready and secure by design\n\n## Security \u0026 Open Source Best Practices\n- **No secrets or credentials are versioned.** Only placeholders in `.env.example`.\n- **OIDC role for CI/CD** uses least privilege and is restricted to ARNs dos recursos do projeto.\n- **S3 state bucket is private** and never public.\n- **No public endpoints** are exposed by default.\n- **All IAM policies** are scoped to only the resources needed (Lambda, SQS, DynamoDB, CloudWatch, S3).\n- **Contributors must never commit real secrets.**\n- **Security reporting:** See [SECURITY.md](SECURITY.md).\n\n## Architecture\n- **EventBridge** schedules execution of the Lambda Scheduler (every 1 hour)\n- **Lambda Scheduler** fetches news URLs and sends them to the **SQS** queue\n- **SQS** stores and distributes URLs for processing\n- **Lambda Worker** scrapes, checks for duplicates in **DynamoDB**, and saves new news\n- **CloudWatch** receives detailed logs\n\n\n## Repository Structure\n- `infra/` — Terraform files for AWS provisioning\n- `packages/` — All Lambda function source code (Node.js, TypeScript)\n- `lambdas/` — Built and zipped Lambda artifacts for deployment (auto-generated by CI/CD)\n- `.github/workflows/` — All GitHub Actions workflow definitions (CI/CD, infra, Lambda)\n- `.github/actions/` — Custom composite actions (e.g., build-zip-lambdas)\n- `.github/` — Copilot instructions, funding, and community files\n\n## Quick Start\n\n### Prerequisites\n- [Terraform](https://www.terraform.io/downloads.html)\n- [AWS CLI](https://aws.amazon.com/cli/)\n- [Node.js](https://nodejs.org/)\n- AWS account and credentials\n\n\n### Setup\n1. Clone this repository\n2. Configure your AWS credentials (`aws configure`)\n3. Build and zip all Lambdas (locally or via CI/CD):\n   ```sh\n   pnpm install\n   pnpm -F @maracanaunews/lambda build\n   pnpm -F @maracanaunews/worker build\n   pnpm -F @maracanaunews/scheduler build\n   # Or use the composite action in .github/actions/build-zip-lambdas\n   ```\n4. Deploy infrastructure:\n   ```sh\n   cd infra\n   terraform init\n   terraform apply\n   ```\n5. Monitor logs in AWS CloudWatch\n\n\n## Usage\n- The scheduler Lambda runs every hour (EventBridge rule)\n- All logs are available in CloudWatch\n- Infrastructure is optimized for minimal AWS cost\n- All build/zip/deploy logic is DRY and managed via workflows in `.github/workflows/` and actions in `.github/actions/`\n\n## Flowchart\n```mermaid\nflowchart TD\n    A[EventBridge schedules Lambda Scheduler execution]\n    B[Lambda Scheduler fetches list of URLs]\n    C[Sends URLs to SQS]\n    D[SQS triggers Lambda Workers]\n    E[Worker scrapes, checks and saves in DynamoDB]\n    F[Logs in CloudWatch]\n    A --\u003e B --\u003e C --\u003e D --\u003e E --\u003e F\n```\n\n\n## Multi-source Pattern (Best Practice)\n- All scrapers share the same SQS queue and DynamoDB table.\n- Each message/item includes a `source` attribute (e.g., `maracanet`, `othersource`).\n- This enables easy scaling and management of multiple sources with minimal AWS resources.\n- To add a new source, add a new package in `packages/`, set a new `SOURCE` value in your Lambda environment and code, and update the build/zip logic if needed.\n\n## Managing Secrets (Best Practice)\n- Store all sensitive values (API keys, credentials) in a local `.env` file (never commit this file).\n- Use `.env.example` as a template for required variables—values should be random placeholders only.\n- Add `.env` to `.gitignore` to prevent accidental commits.\n- Never store real secrets in version control or public repositories.\n\n### Example: .env.example\n```\nAWS_REGION=us-east-1\nDYNAMODB_TABLE=table_example_123\nSQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/example-queue-abc\nSOURCE=example_source\nMY_API_KEY=exampleapikey1234567890\n```\n\n\n## Remote Terraform State (Production Setup)\n- This project uses an S3 backend for Terraform state. Edit `infra/main.tf` and set your S3 bucket name.\n- The S3 bucket is private and only accessible by the OIDC role with least privilege.\n- To migrate local state to S3, run:\n  ```sh\n  cd infra\n  terraform init\n  # Follow prompts to migrate state\n  ```\n- If you already have AWS resources, import them into state:\n  ```sh\n  terraform import module.dynamodb.aws_dynamodb_table.this news-maracanet\n  terraform import module.iam_worker.aws_iam_role.this scraper-worker-role\n  terraform import module.iam_scheduler.aws_iam_role.this scraper-scheduler-role\n  terraform import module.lambda_worker.aws_cloudwatch_log_group.this /aws/lambda/scraper-worker\n  terraform import module.lambda_scheduler.aws_cloudwatch_log_group.this /aws/lambda/scraper-scheduler\n  terraform import module.cloudwatch.aws_cloudwatch_log_group.this /aws/lambda/scraper-maracanet\n  ```\n- After migration/import, your CI/CD will only update Lambda code and never try to recreate existing infra.\n\n## Community \u0026 Discussions\n- Questions, ideas, and feedback are welcome in [GitHub Discussions](https://github.com/edmolima/maracanaunews/discussions)\n\n\n## Contributing\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. All contributions are welcome!\nAll workflow and action logic is in `.github/workflows/` and `.github/actions/`.\n\n## Code of Conduct\nSee [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).\n\n## Security\n- This project does **not** expose any public endpoints by default.\n- All Lambda triggers are internal (EventBridge, SQS).\n- SQS queue access is restricted to your AWS account and the worker Lambda only.\n- IAM roles follow least privilege and are scoped to project resources only.\n- For any future API or public endpoint, use authentication, throttling, and AWS WAF.\n- Monitor usage and set CloudWatch alarms for unusual activity.\n\nSee [SECURITY.md](SECURITY.md) for vulnerability reporting.\n\n## License\n[MIT](LICENSE)\n\n## Funding\nIf you find this project useful, consider supporting via [GitHub Sponsors](https://github.com/sponsors/edmolima) or [Buy Me a Coffee](https://www.buymeacoffee.com/edmolima).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedmolima%2Fmaracanaunews","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fedmolima%2Fmaracanaunews","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedmolima%2Fmaracanaunews/lists"}