{"id":50724566,"url":"https://github.com/ayushpramanik/mlh-production-engineering","last_synced_at":"2026-06-10T03:02:36.691Z","repository":{"id":348987016,"uuid":"1200619790","full_name":"AyushPramanik/MLH-Production-Engineering","owner":"AyushPramanik","description":"Production engineering and SRE-focused infrastructure project built during the MLH Production Engineering hackathon, covering scalable backend systems, observability, Docker, CI/CD, monitoring, load testing, and reliability engineering.","archived":false,"fork":false,"pushed_at":"2026-05-25T14:21:05.000Z","size":359,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-05-25T16:24:58.221Z","etag":null,"topics":["flask","grafana","k6","nginx","postgresql","prometheus","pytest","redis-cache"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AyushPramanik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-03T16:20:20.000Z","updated_at":"2026-05-25T14:21:09.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/AyushPramanik/MLH-Production-Engineering","commit_stats":null,"previous_names":["ayushpramanik/mlh-pde","ayushpramanik/mlh-production-engineering"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/AyushPramanik/MLH-Production-Engineering","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AyushPramanik%2FMLH-Production-Engineering","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AyushPramanik%2FMLH-Production-Engineering/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AyushPramanik%2FMLH-Production-Engineering/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AyushPramanik%2FMLH-Production-Engineering/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AyushPramanik","download_url":"https://codeload.github.com/AyushPramanik/MLH-Production-Engineering/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AyushPramanik%2FMLH-Production-Engineering/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34134634,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-10T02:00:07.152Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flask","grafana","k6","nginx","postgresql","prometheus","pytest","redis-cache"],"created_at":"2026-06-10T03:02:30.494Z","updated_at":"2026-06-10T03:02:36.684Z","avatar_url":"https://github.com/AyushPramanik.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MLH PE Hackathon — URL Shortener + Observability Platform\n\nA production-grade URL shortener with full observability: structured logging, Prometheus metrics, Grafana dashboards, Redis caching, and email alerting.\n\n**Stack:** Flask · Peewee ORM · PostgreSQL 16 · Redis 7 · Prometheus · Grafana · Docker Compose\n\n---\n\n## Architecture\n\n```\n                        ┌─────────────────────────────────────────┐\n                        │            Docker Compose Network        │\n                        │                                         │\n  Browser / Client ─────┤──► web (Flask :5000) ──► db (Postgres) │\n       :5001             │         │    │                         │\n                        │         │    └──────► redis (:6379)    │\n  Grafana :3000 ◄───────┤         │                              │\n                        │         ▼                               │\n  Prometheus :9090 ◄────┤──► /prometheus  (metrics scrape)       │\n                        │                                         │\n                        └─────────────────────────────────────────┘\n\n  Request flow:\n  Client → Flask before_request (timing start, DB connect)\n         → Route handler (cache check → DB query → cache write)\n         → Flask after_request (record metrics, log JSON)\n         → Response\n```\n\n---\n\n## Prerequisites\n\n- **Docker Desktop** (includes Docker Compose) — [install](https://docs.docker.com/get-docker/)\n- **uv** (for local development without Docker):\n  ```bash\n  # macOS / Linux\n  curl -LsSf https://astral.sh/uv/install.sh | sh\n  ```\n\n---\n\n## Quick Start (Docker — recommended)\n\n```bash\n# 1. Clone the repo\ngit clone \u003crepo-url\u003e \u0026\u0026 cd mlh-pe-hackathon\n\n# 2. Start everything (app + DB + Redis + Prometheus + Grafana)\ndocker compose up -d\n\n# 3. Wait ~15 seconds for services to be healthy, then verify\ncurl http://localhost:5001/health\n# → {\"status\": \"ok\", \"hostname\": \"...\"}\n\n# 4. (Optional) Seed with sample data\ndocker compose exec web uv run python seed/seed.py\n\n# 5. Open the monitoring dashboard\nopen http://localhost:5001/dashboard   # built-in live dashboard\nopen http://localhost:3000             # Grafana (admin / admin)\nopen http://localhost:9090             # Prometheus\n```\n\nThat's it — the entire stack runs with one command.\n\n---\n\n## Quick Start (Local Development)\n\n```bash\n# 1. Install dependencies\nuv sync\n\n# 2. Start only the backing services\ndocker compose up -d db redis\n\n# 3. Configure environment\ncp .env.example .env\n# Edit .env if your DB credentials differ from the defaults\n\n# 4. Run the Flask server\nuv run run.py\n\n# 5. Verify\ncurl http://localhost:5000/health\n```\n\n---\n\n## Running Tests\n\n```bash\nuv run pytest\n```\n\nTests mock Redis and run against a real PostgreSQL database. Coverage must stay above 50%.\n\n---\n\n## Project Structure\n\n```\nmlh-pe-hackathon/\n├── app/\n│   ├── __init__.py          # App factory, middleware, system routes\n│   ├── database.py          # Peewee ORM setup and connection hooks\n│   ├── cache.py             # Redis client and get/set/delete helpers\n│   ├── alerting.py          # Background email alert manager\n│   ├── prometheus_metrics.py # Prometheus counter/histogram/gauge definitions\n│   ├── metrics_store.py     # Thread-safe sliding-window error tracker\n│   ├── logging_config.py    # Structured JSON logging setup\n│   ├── models/              # Peewee data models (User, URL, Event)\n│   ├── routes/              # Blueprint route handlers\n│   │   ├── users.py         # User CRUD + bulk load\n│   │   ├── url.py           # URL shortener CRUD + redirect\n│   │   └── events.py        # Audit log queries\n│   └── templates/\n│       └── dashboard.html   # Self-contained live monitoring dashboard\n├── docs/\n│   ├── RUNBOOK.md           # Incident response playbooks\n│   ├── DEPLOY.md            # Deployment and rollback guide\n│   ├── TROUBLESHOOTING.md   # Common problems and fixes\n│   ├── DECISIONS.md         # Architecture decision log\n│   └── CAPACITY.md          # Capacity planning and limits\n├── monitoring/\n│   ├── prometheus.yml       # Scrape config\n│   └── grafana/provisioning # Auto-provisioned datasource + dashboard\n├── scaling/\n│   ├── nginx/nginx.conf     # Load balancer config (multi-replica)\n│   └── load_test/           # k6 and Python load test scripts\n├── seed/\n│   ├── seed.py              # Database seeding script\n│   └── *.csv                # Sample data files\n├── tests/                   # pytest test suite\n├── docker-compose.yml       # Full stack orchestration\n├── Dockerfile               # App container (python:3.13-slim + uv)\n├── pyproject.toml           # Dependencies and pytest config\n├── .env.example             # All environment variables documented\n└── fire_drill.py            # Alert testing tool (safe, no real incidents needed)\n```\n\n---\n\n## API Reference\n\nAll endpoints return JSON. Errors follow `{\"error\": \"\u003cmessage\u003e\"}`.\n\n### System\n\n| Method | Path | Description |\n|--------|------|-------------|\n| GET | `/health` | Health check — returns `{\"status\":\"ok\",\"hostname\":\"...\"}` |\n| GET | `/metrics` | JSON system metrics — CPU%, memory MB, request counts |\n| GET | `/prometheus` | Prometheus scrape endpoint (text/plain exposition format) |\n| GET | `/logs?limit=100` | Recent application log entries as JSON array |\n| GET | `/alert-status` | Current alert states (service_down, high_error_rate) |\n| GET | `/dashboard` | Live monitoring dashboard (HTML) |\n\n### Users\n\n| Method | Path | Description |\n|--------|------|-------------|\n| GET | `/users?page=1\u0026per_page=20` | List users (paginated, cached) |\n| POST | `/users` | Create user — body: `{\"username\":\"…\",\"email\":\"…\"}` |\n| GET | `/users/\u003cid\u003e` | Get user by ID (cached) |\n| PUT | `/users/\u003cid\u003e` | Update username/email |\n| DELETE | `/users/\u003cid\u003e` | Hard delete user |\n| GET | `/users/\u003cid\u003e/urls` | All URLs belonging to a user |\n| POST | `/users/bulk` | Bulk import from CSV — body: `{\"file\":\"users.csv\"}` |\n\n### URLs\n\n| Method | Path | Description |\n|--------|------|-------------|\n| GET | `/urls?user_id=\u0026is_active=\u0026page=1\u0026per_page=20` | List URLs (filtered, cached) |\n| POST | `/urls` | Create short URL — body: `{\"original_url\":\"…\",\"title\":\"…\",\"user_id\":1}` |\n| POST | `/shorten` | Alias for POST /urls — body: `{\"url\":\"…\"}` |\n| GET | `/urls/\u003cid\u003e` | Get URL by ID (cached) |\n| PUT/PATCH | `/urls/\u003cid\u003e` | Update URL fields |\n| DELETE | `/urls/\u003cid\u003e` | Soft delete (sets `is_active=false`) |\n| GET | `/urls/\u003cid\u003e/events` | Audit log for a URL |\n| POST | `/urls/bulk` | Bulk import from CSV — body: `{\"file\":\"urls.csv\"}` |\n| GET | `/\u003cshort_code\u003e` | Redirect to original URL (302) |\n\n### Events\n\n| Method | Path | Description |\n|--------|------|-------------|\n| GET | `/events?url_id=\u0026user_id=\u0026event_type=` | List events (filtered) |\n| POST | `/events` | Create event — body: `{\"url_id\":1,\"event_type\":\"created\"}` |\n| GET | `/events/\u003cid\u003e` | Get event by ID |\n\n### Response shapes\n\n**User object:**\n```json\n{\"id\": 1, \"username\": \"alice\", \"email\": \"alice@example.com\", \"created_at\": \"2024-01-01T12:00:00\"}\n```\n\n**URL object:**\n```json\n{\n  \"id\": 1, \"user_id\": 1, \"short_code\": \"abc123\",\n  \"original_url\": \"https://example.com\", \"title\": \"Example\",\n  \"is_active\": true, \"created_at\": \"…\", \"updated_at\": \"…\"\n}\n```\n\n**Event object:**\n```json\n{\"id\": 1, \"url_id\": 1, \"user_id\": 1, \"event_type\": \"created\", \"timestamp\": \"…\", \"details\": \"…\"}\n```\n\n---\n\n## Monitoring URLs\n\n| Service | URL | Credentials |\n|---------|-----|-------------|\n| App | http://localhost:5001 | — |\n| Live Dashboard | http://localhost:5001/dashboard | — |\n| Grafana | http://localhost:3000 | admin / admin |\n| Prometheus | http://localhost:9090 | — |\n\n---\n\n## Alerting (Email)\n\nCopy `.env.example` to `.env` and fill in the SMTP section to enable email alerts:\n\n```bash\nSMTP_USER=you@gmail.com\nSMTP_PASSWORD=your-app-password   # Gmail App Password (not your login password)\nALERT_EMAIL_TO=oncall@yourteam.com\n```\n\nTwo alerts fire automatically:\n- **Service Down** — PostgreSQL unreachable\n- **High Error Rate** — \u003e10% of requests return 5xx in a 2-minute window\n\nSee [`docs/RUNBOOK.md`](docs/RUNBOOK.md) for response playbooks.\n\n---\n\n## Error Handling\n\n| Status | Meaning |\n|--------|---------|\n| 200 / 201 | Success |\n| 302 | Redirect (short URL) |\n| 400 | Bad request — validation failed (see `\"error\"` field) |\n| 404 | Resource not found or inactive |\n| 405 | Wrong HTTP method |\n| 409 | Conflict — duplicate username or email |\n| 500 | Unexpected server error (never an HTML stack trace) |\n\n---\n\n## Further Reading\n\n- [`docs/DEPLOY.md`](docs/DEPLOY.md) — how to deploy and rollback\n- [`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md) — common problems and fixes\n- [`docs/DECISIONS.md`](docs/DECISIONS.md) — why we chose each technology\n- [`docs/CAPACITY.md`](docs/CAPACITY.md) — how many users can we handle\n- [`docs/RUNBOOK.md`](docs/RUNBOOK.md) — incident response playbooks\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fayushpramanik%2Fmlh-production-engineering","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fayushpramanik%2Fmlh-production-engineering","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fayushpramanik%2Fmlh-production-engineering/lists"}