{"id":35150751,"url":"https://github.com/frostyalce000/hackernewsalerts-backend","last_synced_at":"2026-04-10T11:01:18.143Z","repository":{"id":330319671,"uuid":"1122396079","full_name":"frostyalce000/hackernewsalerts-backend","owner":"frostyalce000","description":"The backend services for [hackernewsalerts.com](https://hackernewsalerts.com). A web application that sends email notifications when someone replies to one of your comments or posts on Hacker News.","archived":false,"fork":false,"pushed_at":"2025-12-24T16:32:00.000Z","size":54,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-26T07:30:54.119Z","etag":null,"topics":["aws-ses","caprover","django","django-ninja","django-q2","docker","gunicorn","postgresql","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/frostyalce000.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-24T16:29:15.000Z","updated_at":"2025-12-24T16:34:19.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/frostyalce000/hackernewsalerts-backend","commit_stats":null,"previous_names":["frostyalce000/hackernewsalerts-backend"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/frostyalce000/hackernewsalerts-backend","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frostyalce000%2Fhackernewsalerts-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frostyalce000%2Fhackernewsalerts-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frostyalce000%2Fhackernewsalerts-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frostyalce000%2Fhackernewsalerts-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/frostyalce000","download_url":"https://codeload.github.com/frostyalce000/hackernewsalerts-backend/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frostyalce000%2Fhackernewsalerts-backend/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31639524,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-10T07:40:12.752Z","status":"ssl_error","status_checked_at":"2026-04-10T07:40:11.664Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-ses","caprover","django","django-ninja","django-q2","docker","gunicorn","postgresql","python"],"created_at":"2025-12-28T15:35:23.251Z","updated_at":"2026-04-10T11:01:18.137Z","avatar_url":"https://github.com/frostyalce000.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# hackernewsalerts-backend\r\n\r\nThe backend services for [hackernewsalerts.com](https://hackernewsalerts.com). A web application that sends email notifications when someone replies to one of your comments or posts on Hacker News.\r\n\r\n[Link to the frontend](https://github.com/frostyalce000/hackernewsalerts-frontend).\r\n\r\n## Features\r\n\r\n- **Email Signup**: Users can sign up with their Hacker News username and email address\r\n- **Email Verification**: Secure email verification system using signed tokens\r\n- **Comment Replies Monitoring**: Automatically detects new replies to user's comments\r\n- **Post Comments Monitoring**: Tracks new comments on user's posts (within the last 14 days)\r\n- **Email Alerts**: Sends consolidated email notifications with all new activity\r\n- **Unsubscribe**: Easy one-click unsubscribe functionality\r\n- **Error Monitoring**: Admin notifications when processing errors occur\r\n\r\n## Architecture\r\n\r\nThe project consists of 2 Python Django services that share the same PostgreSQL database:\r\n\r\n### 1. REST API Service (Web)\r\n- **Framework**: [Django Ninja](https://github.com/vitalik/django-ninja) for building REST APIs\r\n- **Purpose**: Handles user signups, email verification, and unsubscribe requests\r\n- **Server**: Gunicorn WSGI server\r\n- **File**: [`alerts/views.py`](./alerts/views.py)\r\n\r\n### 2. Scheduled Worker Service (Tasks)\r\n- **Framework**: [Django Q2](https://django-q2.readthedocs.io/en/master/) for task scheduling\r\n- **Purpose**: Periodically checks for new Hacker News activity and sends email alerts\r\n- **File**: [`alerts/tasks.py`](./alerts/tasks.py)\r\n\r\nBoth services are deployed using [CapRover](https://caprover.com/) with separate Docker containers.\r\n\r\n## Tech Stack\r\n\r\n- **Python 3.12**\r\n- **Django 5.0.6** - Web framework\r\n- **Django Ninja 1.2.0** - REST API framework\r\n- **Django Q2 1.6.2** - Task queue and scheduling\r\n- **PostgreSQL** - Database (production) / SQLite (development)\r\n- **AWS SES (boto3)** - Email delivery service\r\n- **Gunicorn** - WSGI HTTP server\r\n- **Docker** - Containerization\r\n- **CapRover** - Deployment platform\r\n\r\n### Key Dependencies\r\n\r\n- `boto3` - AWS SDK for email sending via SES\r\n- `beautifulsoup4` - HTML parsing for comment content\r\n- `requests` - HTTP client for Hacker News RSS feeds\r\n- `pydantic` - Data validation\r\n- `django-cors-headers` - CORS handling\r\n- `whitenoise` - Static file serving\r\n\r\n## Project Structure\r\n\r\n```\r\nhackernewsalerts-backend/\r\n├── alerts/                    # Main Django app\r\n│   ├── models.py             # User model\r\n│   ├── views.py              # REST API endpoints\r\n│   ├── tasks.py              # Scheduled task for checking alerts\r\n│   ├── hn.py                 # Hacker News API integration\r\n│   ├── mail.py               # Email sending via AWS SES\r\n│   ├── utils.py              # Utility functions\r\n│   ├── urls.py               # URL routing\r\n│   ├── admin.py              # Django admin configuration\r\n│   └── migrations/           # Database migrations\r\n├── socialalerts/             # Django project settings\r\n│   ├── settings.py           # Django configuration\r\n│   ├── urls.py               # Root URL configuration\r\n│   ├── wsgi.py               # WSGI application\r\n│   └── asgi.py               # ASGI application\r\n├── Dockerfile-web            # Docker image for API service\r\n├── Dockerfile-tasks          # Docker image for worker service\r\n├── captain-definition-web.json    # CapRover config for web\r\n├── captain-definition-tasks.json  # CapRover config for tasks\r\n├── requirements.txt          # Python dependencies\r\n├── Makefile                  # Common commands\r\n├── run-web.sh               # Web service startup script\r\n└── manage.py                # Django management script\r\n```\r\n\r\n## Setup \u0026 Installation\r\n\r\n### Prerequisites\r\n\r\n- Python 3.12+\r\n- PostgreSQL (for production) or SQLite (for development)\r\n- AWS account with SES configured (for email sending)\r\n- Docker (optional, for containerized deployment)\r\n\r\n### Local Development Setup\r\n\r\n1. **Clone the repository**\r\n   ```bash\r\n   git clone https://github.com/frostyalce000/hackernewsalerts-backend.git\r\n   cd hackernewsalerts-backend\r\n   ```\r\n\r\n2. **Create a virtual environment**\r\n   ```bash\r\n   python -m venv venv\r\n   source venv/bin/activate  # On Windows: venv\\Scripts\\activate\r\n   ```\r\n\r\n3. **Install dependencies**\r\n   ```bash\r\n   pip install -r requirements.txt\r\n   ```\r\n\r\n4. **Create a `.env` file** (see [Configuration](#configuration) section)\r\n   ```bash\r\n   cp .env.example .env  # If you have an example file\r\n   # Or create .env manually with required variables\r\n   ```\r\n\r\n5. **Run database migrations**\r\n   ```bash\r\n   make migrate\r\n   # or\r\n   python manage.py migrate\r\n   ```\r\n\r\n6. **Create a superuser** (optional, for admin access)\r\n   ```bash\r\n   make create-superuser\r\n   # or\r\n   python manage.py createsuperuser\r\n   ```\r\n\r\n### Running the Services\r\n\r\n#### Run the Web API Service\r\n```bash\r\nmake run-web\r\n# or\r\npython manage.py runserver\r\n```\r\n\r\nThe API will be available at `http://localhost:8000`\r\n\r\n#### Run the Task Worker Service\r\n```bash\r\nmake run-tasks\r\n# or\r\npython manage.py qcluster\r\n```\r\n\r\n**Note**: The scheduled task (`check_for_alerts`) needs to be configured in Django Q2's admin interface or via management command. The task should be scheduled to run periodically (e.g., every hour).\r\n\r\n## Configuration\r\n\r\n### Environment Variables\r\n\r\nThe application uses environment variables for configuration. In local development, these are loaded from a `.env` file using `python-dotenv`.\r\n\r\n#### Required Variables\r\n\r\n- `SECRET_KEY` - Django secret key (required in production)\r\n- `DB_NAME` - PostgreSQL database name (production)\r\n- `DB_USER` - PostgreSQL database user (production)\r\n- `DB_PASSWORD` - PostgreSQL database password (production)\r\n- `DB_HOST` - PostgreSQL database host (production)\r\n- `DB_PORT` - PostgreSQL database port (production)\r\n- `UI_URL` - Frontend URL for email verification links (e.g., `https://hackernewsalerts.com`)\r\n- `API_URL` - Backend API URL for unsubscribe links (e.g., `https://api.hackernewsalerts.com`)\r\n\r\n#### AWS SES Configuration\r\n\r\nThe application uses AWS SES for sending emails. Configure AWS credentials using one of these methods:\r\n\r\n1. **AWS Credentials File** (`~/.aws/credentials`)\r\n   ```ini\r\n   [default]\r\n   aws_access_key_id = YOUR_ACCESS_KEY\r\n   aws_secret_access_key = YOUR_SECRET_KEY\r\n   ```\r\n\r\n2. **Environment Variables**\r\n   ```bash\r\n   export AWS_ACCESS_KEY_ID=your_access_key\r\n   export AWS_SECRET_ACCESS_KEY=your_secret_key\r\n   export AWS_DEFAULT_REGION=eu-west-2\r\n   ```\r\n\r\n3. **IAM Role** (if running on AWS infrastructure)\r\n\r\n**Note**: The sender email `alerts@hackernewsalerts.com` must be verified in AWS SES.\r\n\r\n### Django Settings\r\n\r\nKey settings in `socialalerts/settings.py`:\r\n\r\n- **Database**: Uses PostgreSQL in production, SQLite in development (when `SECRET_KEY` is not set)\r\n- **CORS**: Configured for frontend domains\r\n- **Django Q2**: Configured with ORM backend, 1 worker, 12000s timeout\r\n- **Logging**: Console logging with verbose format\r\n\r\n## API Endpoints\r\n\r\nAll API endpoints are prefixed with `/api/` and use Django Ninja.\r\n\r\n### POST `/api/signup`\r\nCreate a new alert subscription.\r\n\r\n**Request Body:**\r\n```json\r\n{\r\n  \"hn_username\": \"username\",\r\n  \"email\": \"user@example.com\"\r\n}\r\n```\r\n\r\n**Response:**\r\n- `201 Created` - User created, verification email sent\r\n- `400 Bad Request` - Username already exists\r\n\r\n### POST `/api/verify/{code}`\r\nVerify email address using verification code from email.\r\n\r\n**Response:**\r\n- `200 OK` - Email verified successfully\r\n- `400 Bad Request` - Invalid verification code\r\n- `404 Not Found` - User not found\r\n\r\n### GET `/api/unsubscribe/?token={token}`\r\nPreview unsubscribe page (confirmation form).\r\n\r\n**Response:**\r\n- HTML page with unsubscribe confirmation form\r\n- `400 Bad Request` - Invalid token\r\n\r\n### POST `/api/unsubscribe/confirm/?token={token}`\r\nConfirm and process unsubscribe request.\r\n\r\n**Response:**\r\n- `200 OK` - Successfully unsubscribed\r\n- `400 Bad Request` - Invalid token\r\n- `404 Not Found` - User not found\r\n\r\n## Database Models\r\n\r\n### User Model\r\n\r\nLocated in `alerts/models.py`:\r\n\r\n```python\r\nclass User(models.Model):\r\n    hn_username = models.CharField(max_length=100, db_index=True, unique=True)\r\n    email = models.EmailField(db_index=True)\r\n    created_at = models.DateTimeField(auto_now_add=True)\r\n    updated_at = models.DateTimeField(auto_now=True)\r\n    last_checked = models.DateTimeField(default=timezone.now)\r\n    is_verified = models.BooleanField(default=False)\r\n```\r\n\r\n**Fields:**\r\n- `hn_username` - Hacker News username (unique, indexed)\r\n- `email` - User's email address (indexed)\r\n- `created_at` - Account creation timestamp\r\n- `updated_at` - Last update timestamp\r\n- `last_checked` - Last time alerts were checked for this user\r\n- `is_verified` - Whether the email has been verified\r\n\r\n## How It Works\r\n\r\n### Signup Flow\r\n\r\n1. User submits signup form with HN username and email\r\n2. System creates a `User` record with `is_verified=False`\r\n3. Verification email is sent with a signed token\r\n4. User clicks verification link in email\r\n5. System verifies token and sets `is_verified=True`\r\n6. User is now eligible to receive alerts\r\n\r\n### Alert Checking Flow\r\n\r\nThe scheduled task (`check_for_alerts`) runs periodically:\r\n\r\n1. Fetches all verified users\r\n2. For each user:\r\n   - Calls `get_new_comment_replies()` to find replies to user's comments\r\n   - Calls `get_new_post_comments()` to find comments on user's posts (last 14 days)\r\n   - Filters out comments/replies by the user themselves\r\n   - Filters by `last_checked` timestamp to get only new activity\r\n3. If new activity found:\r\n   - Formats email content with all new comments/replies\r\n   - Adds unsubscribe link\r\n   - Sends email via AWS SES\r\n4. Updates `user.last_checked` timestamp\r\n5. If errors occur, sends alert email to admin\r\n\r\n### Hacker News Integration\r\n\r\nThe system uses [hnrss.org](https://hnrss.org/) RSS feeds:\r\n\r\n- **Comment Replies**: `https://hnrss.org/replies.jsonfeed?id={username}`\r\n- **User Posts**: `https://hnrss.org/submitted.jsonfeed?id={username}`\r\n- **Post Comments**: `https://hnrss.org/item.jsonfeed?id={post_id}`\r\n\r\nOnly posts from the last 14 days are checked for comments (posts older than 14 days are typically closed for discussion).\r\n\r\n## Development\r\n\r\n### Common Commands\r\n\r\nSee the [Makefile](./Makefile) for all available commands:\r\n\r\n```bash\r\n# Run web service\r\nmake run-web\r\n\r\n# Run task worker\r\nmake run-tasks\r\n\r\n# Database migrations\r\nmake makemigrations\r\nmake migrate\r\n\r\n# Create superuser\r\nmake create-superuser\r\n\r\n# Collect static files\r\nmake collect-static\r\n\r\n# Run tests\r\nmake test\r\n\r\n# Docker commands\r\nmake docker-build-web\r\nmake docker-run-web\r\n\r\n# Deployment (requires environment variables)\r\nmake deploy-web\r\nmake deploy-tasks\r\n```\r\n\r\n### Code Structure\r\n\r\n- **`alerts/hn.py`**: Functions to fetch data from Hacker News RSS feeds\r\n- **`alerts/mail.py`**: AWS SES email sending wrapper\r\n- **`alerts/utils.py`**: Utility functions (date formatting, HTML parsing, token signing)\r\n- **`alerts/tasks.py`**: Main task function that processes all users\r\n- **`alerts/views.py`**: REST API endpoint handlers\r\n\r\n## Testing\r\n\r\nRun tests with:\r\n```bash\r\nmake test\r\n# or\r\npython manage.py test\r\n```\r\n\r\n### Test Files\r\n\r\n- `alerts/tests.py` - General tests\r\n- `alerts/tests_hn.py` - Hacker News integration tests\r\n- `alerts/tests_tasks.py` - Task processing tests\r\n\r\n**Note**: Some tests require environment variables (e.g., `TEST_RUN_TASK=1`, `TEST_HN_USERNAME`) to run integration tests against real Hacker News data.\r\n\r\n## Deployment\r\n\r\n### Docker\r\n\r\nThe project includes two Dockerfiles:\r\n\r\n- **`Dockerfile-web`**: For the REST API service\r\n  - Runs migrations on startup\r\n  - Starts Gunicorn server on port 8000\r\n\r\n- **`Dockerfile-tasks`**: For the scheduled worker service\r\n  - Runs Django Q2 cluster (`qcluster`)\r\n\r\n### CapRover Deployment\r\n\r\nThe project is configured for CapRover deployment with two separate apps:\r\n\r\n1. **Web Service**: Deployed using `captain-definition-web.json`\r\n2. **Tasks Service**: Deployed using `captain-definition-tasks.json`\r\n\r\nDeploy using:\r\n```bash\r\nmake deploy-web    # Requires CAPROVER_APP_WEB, CAPROVER_NODE, BRANCH\r\nmake deploy-tasks  # Requires CAPROVER_APP_TASKS, CAPROVER_NODE, BRANCH\r\n```\r\n\r\n### Production Checklist\r\n\r\n- [ ] Set `SECRET_KEY` environment variable\r\n- [ ] Configure PostgreSQL database connection\r\n- [ ] Set up AWS SES and verify sender email\r\n- [ ] Configure `UI_URL` and `API_URL` environment variables\r\n- [ ] Set up Django Q2 scheduled task for `check_for_alerts`\r\n- [ ] Configure CORS allowed origins\r\n- [ ] Set up proper logging and monitoring\r\n- [ ] Create superuser for admin access\r\n\r\n## Notes\r\n\r\n### Known Issues\r\n\r\n- The scheduled worker sends email alerts to the admin user every time there's a failure. Sometimes these alerts occur due to missing data from the Hacker News RSS feeds, but the data usually becomes available after a couple of hours.\r\n\r\n### Security Considerations\r\n\r\n- Email verification uses Django's signing framework with secure tokens\r\n- Unsubscribe tokens are signed with a separate salt\r\n- CSRF protection is enabled for POST endpoints\r\n- CORS is configured to restrict origins\r\n- Database credentials should never be committed to version control\r\n\r\n### Performance\r\n\r\n- User queries are indexed on `hn_username` and `email`\r\n- Only posts from the last 14 days are checked for comments\r\n- The task processes users sequentially; consider parallelization for large user bases\r\n\r\n## License\r\n\r\nSee [LICENSE](./LICENSE) file for details.\r\n\r\n## Contributing\r\n\r\nContributions are welcome! Please feel free to submit a Pull Request.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrostyalce000%2Fhackernewsalerts-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffrostyalce000%2Fhackernewsalerts-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrostyalce000%2Fhackernewsalerts-backend/lists"}