{"id":51417463,"url":"https://github.com/cemroot/my-site","last_synced_at":"2026-07-04T21:01:42.447Z","repository":{"id":349622444,"uuid":"1073916929","full_name":"CemRoot/My-Site","owner":"CemRoot","description":"My Platform enjoy! ","archived":false,"fork":false,"pushed_at":"2026-07-01T16:07:11.000Z","size":7952,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-01T18:09:30.267Z","etag":null,"topics":["ai","ai-agents","chatbot","news","portfolio-website","tech-news"],"latest_commit_sha":null,"homepage":"https://cemkoyluoglu.codes/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CemRoot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-10T20:30:43.000Z","updated_at":"2026-07-01T16:07:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/CemRoot/My-Site","commit_stats":null,"previous_names":["cemroot/my-site"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CemRoot/My-Site","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CemRoot%2FMy-Site","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CemRoot%2FMy-Site/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CemRoot%2FMy-Site/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CemRoot%2FMy-Site/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CemRoot","download_url":"https://codeload.github.com/CemRoot/My-Site/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CemRoot%2FMy-Site/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35135342,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-04T02:00:05.987Z","response_time":113,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-agents","chatbot","news","portfolio-website","tech-news"],"created_at":"2026-07-04T21:01:41.680Z","updated_at":"2026-07-04T21:01:42.437Z","avatar_url":"https://github.com/CemRoot.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# 🌐 Tech News Automation Platform\n\n### AI-Powered News Aggregation \u0026 Distribution System\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)\n[![Node Version](https://img.shields.io/badge/node-%3E%3D20.0.0-brightgreen)](https://nodejs.org/)\n[![TypeScript](https://img.shields.io/badge/TypeScript-5.9-blue)](https://www.typescriptlang.org/)\n[![React](https://img.shields.io/badge/React-18.3-61dafb)](https://reactjs.org/)\n[![Vercel](https://img.shields.io/badge/Deployed%20on-Vercel-000000?logo=vercel)](https://vercel.com)\n[![Sentry](https://img.shields.io/badge/Monitored%20by-Sentry-362d59?logo=sentry)](https://sentry.io)\n\n[Live Demo](https://cemkoyluoglu.codes) · [Report Bug](https://github.com/CemRoot/My-Site/issues) · [Request Feature](https://github.com/CemRoot/My-Site/issues)\n\n\u003c/div\u003e\n\n---\n\n## 📋 Table of Contents\n\n- [Overview](#-overview)\n- [Key Features](#-key-features)\n- [Architecture](#-architecture)\n- [Daily News Agent](#-daily-news-agent)\n- [Tech Stack](#-tech-stack)\n- [Quick Start](#-quick-start)\n- [Telegram Bot](#-telegram-bot-control-center)\n- [Automation](#-automation-workflows)\n- [Scrape Tech News Architecture](#-scrape-tech-news-architecture-and-data-flow)\n- [Configuration](#-configuration)\n- [Deployment](#-deployment)\n- [Monitoring](#-monitoring--observability)\n- [Contributing](#-contributing)\n- [Security](#-security)\n- [License](#-license)\n\n---\n\n## 🎯 Overview\n\n**Tech News Platform** is an enterprise-grade, fully automated news aggregation and distribution system that combines cutting-edge AI, intelligent scraping, and seamless automation to deliver tech news in real-time.\n\n### 🌟 What Makes This Special?\n\n- **🤖 AI-Powered**: Multi-AI system (Groq Llama 3.3 + Google Gemini 2.0 Flash)\n- **💬 AI Chatbot**: Interactive portfolio chatbot with n8n fallback\n- **📱 Telegram Control Center**: Full system control from your phone\n- **📧 Newsletter System**: Email subscription and newsletter management\n- **🔄 100% Automated**: GitHub Actions + n8n + Vercel integration\n- **📊 Production-Ready**: Sentry monitoring, health checks, deployment tracking\n- **⚡ Lightning Fast**: Supabase backend, optimized queries\n- **🔒 Enterprise Security**: Rate limiting, API secrets, fail-safe mechanisms\n\n---\n\n## ✨ Key Features\n\n### 🗞️ News Aggregation System\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\"\u003e\n\n#### Intelligent Scraping\n- Deterministic daily agent across 6 business categories\n- Smart rate limiting (respects API limits)\n- Duplicate detection \u0026 prevention\n- Source attribution \u0026 tracking\n- Original article link preservation\n- Replayable run artifacts for rejected/failed/deleted batches\n- **Manual article scraper via Telegram**\n\n\u003c/td\u003e\n\u003ctd width=\"50%\"\u003e\n\n#### AI Translation \u0026 Processing\n- Turkish → English translation (Groq AI)\n- **Google Gemini 2.0 Flash** for content optimization\n- Context-aware processing\n- Social media embed preservation\n- Markdown formatting retention\n- **Smart content quality validation**\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n### 💬 AI Chatbot System\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\"\u003e\n\n#### Portfolio Chatbot\n- **Groq AI (Llama 3.3 70B)** primary backend\n- **n8n fallback** for high availability\n- Chat history persistence (Supabase)\n- Session management\n- Rate limiting protection\n\n\u003c/td\u003e\n\u003ctd width=\"50%\"\u003e\n\n#### Smart Features\n- Context-aware responses\n- Page context integration\n- Multi-language support\n- Real-time conversation tracking\n- Automatic session cleanup\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n### 🤖 Telegram Bot Control Center\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\"\u003e\n\n#### Interactive Menu System\n- 📰 Manual scraping trigger\n- ➕ Manual article addition\n- 🔧 System Management\n  - 🤖 n8n trial tracking\n  - 🔄 Webhook reset\n  - 🏥 Health checks\n- 📊 Real-time statistics\n- 💾 Database management\n- 📱 LinkedIn digest management\n\n\u003c/td\u003e\n\u003ctd width=\"50%\"\u003e\n\n#### Automated Notifications\n- ✅ Success reports\n- ❌ Error alerts with details\n- 📈 Daily health summaries\n- 🚀 Deployment notifications\n- 🚨 Vercel status alerts\n- 📱 Real-time updates\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n### 📧 Newsletter System\n\n- Email subscription management\n- Subscriber data storage (Supabase + JSON backup)\n- Rate limiting protection\n- Input validation\n\n### 🔄 Full Automation\n\n- **Scheduled Scraping**: 3x daily on weekdays (07:00, 13:00, 15:00 UTC)\n- **Manual Article Scraper**: On-demand article processing via Telegram\n- **LinkedIn Digest**: Daily automated post generation (via n8n)\n- **Vercel Status Monitor**: 30-min interval platform health checks\n- **Auto Deployment**: Vercel CI/CD with post-build hooks\n\n---\n\n## 🏗️ Architecture\n\n\u003e **Enterprise-Grade System Design** following Google Cloud Architecture best practices\n\n### 📊 Component Architecture\n\n| Layer | Components | Technology | Purpose |\n|-------|-----------|------------|---------|\n| **Edge** | CDN, Cache | Vercel Edge Network | Global content delivery, \u003c100ms latency |\n| **Frontend** | React SPA, UI Components | React 18, TypeScript, Vite | User interface, real-time updates |\n| **API Gateway** | Serverless Functions | Vercel Edge Functions | Request routing, authentication |\n| **Orchestration** | Workflows, Schedulers | GitHub Actions, n8n | Automation, scheduled tasks |\n| **AI/ML** | Translation, Chat, Generation | Groq AI, Google Gemini, Firecrawl | Content processing, intelligence |\n| **Observability** | Monitoring, Logging | Sentry, Custom Health Checks | Error tracking, performance monitoring |\n| **Data** | Database, Storage | Supabase PostgreSQL | Persistent storage, real-time sync |\n| **Communication** | Messaging, Notifications | Telegram Bot, LinkedIn API | User interaction, distribution |\n\n### 🗂️ System Components\n\n| Layer | Component | Technology | Purpose |\n|-------|-----------|------------|---------|\n| **Frontend** | React SPA | React 18.3 + TypeScript 5.9 | Modern, responsive UI |\n| | Build Tool | Vite 6.4 | Fast build \u0026 HMR |\n| | Styling | Tailwind CSS 4 (CSS-first) | Utility-first styling |\n| **Backend** | API Gateway | Vercel Edge Functions | Serverless endpoints |\n| | Webhooks | Telegram, Deployment | Event handling |\n| **Database** | Primary DB | Supabase PostgreSQL | Structured data storage |\n| | Chat History | Supabase | Conversation persistence |\n| **AI/ML** | Translation | Groq AI (Llama 3.1 8B primary, 70B last resort) | Multi-language support |\n| | Content Gen | Google Gemini 2.0 Flash | Article processing |\n| | Chat Widget | Groq AI + n8n fallback | User interaction |\n| | Web Scraping | Firecrawl API | Article extraction |\n| **Observability** | Error Tracking | Sentry 10.x | Frontend \u0026 backend |\n| | Health Checks | Custom Scripts | System monitoring |\n| **Communication** | Bot Platform | Telegram Bot API | Interactive control |\n| | Social Media | LinkedIn API (n8n OAuth) | Content distribution |\n| | Email | Newsletter System | Subscriber management |\n\n---\n\n## 🛠️ Tech Stack\n\n### Frontend\n| Technology | Version | Purpose |\n|-----------|---------|---------|\n| **React** | 19.2 | UI Framework |\n| **TypeScript** | 6.0 | Type Safety |\n| **Vite** | 8.0 | Build Tool |\n| **Tailwind CSS** | 4.x (CSS-first) | Styling |\n| **Radix UI** | Selective | Dialog, Checkbox, Separator |\n| **React Router** | 7.18 | Navigation |\n| **React Markdown** | 10.1 | Content Rendering |\n\n### Backend \u0026 APIs\n| Service | Purpose | Notes |\n|---------|---------|-------|\n| **Supabase** | PostgreSQL Database | Free/Pro tier |\n| **Groq AI** | Translation \u0026 Chat | Chat: Llama 3.3 70B · Translation: Llama 3.1 8B (70B fallback) |\n| **Google Gemini** | Content Generation | 2.0 Flash |\n| **Firecrawl** | Web Scraping | 500/mo free |\n| **n8n** | Workflow Automation | Self-hosted/Cloud |\n| **Telegram Bot API** | Notifications \u0026 Control | Unlimited |\n| **GitHub Actions** | CI/CD \u0026 Automation | 2000 min/mo free |\n| **Sentry** | Error Tracking | 5K errors/mo |\n\n### Infrastructure\n| Platform | Purpose | Cost |\n|----------|---------|------|\n| **Vercel** | Hosting \u0026 Edge Functions | Free/Pro |\n| **GitHub** | Version Control \u0026 Actions | Free |\n\n---\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n```bash\nnode \u003e= 20.0.0\nnpm \u003e= 10.0.0\ngit \u003e= 2.40.0\n```\n\n### Installation\n\n```bash\n# 1. Clone the repository\ngit clone https://github.com/CemRoot/My-Site.git\ncd My-Site\n\n# 2. Install dependencies\nnpm install\n\n# 3. Setup environment variables\ncp .env.example .env\n\n# 4. Add your API keys to .env\n# See Configuration section below\n\n# 5. Start development server\nnpm run dev\n```\n\n### First-Time Setup\n\n```bash\n# Setup Telegram bot menu\nnpm run telegram:setup-menu\n\n# Run initial health check\nnpm run health:check\n```\n\n---\n\n## 📱 Telegram Bot Control Center\n\n### Features\n\nThe Telegram bot provides complete system control from your mobile device:\n\n```\n┌─────────────────────────────────┐\n│     🤖 TECH NEWS BOT MENU       │\n├─────────────────────────────────┤\n│  📰 Haberleri Çek               │\n│  📱 LinkedIn Posts              │\n│  🏥 Sağlık Kontrolü             │\n│  📊 Sistem Durumu               │\n│  📈 İstatistikler               │\n│  🔧 GitHub Actions              │\n│  💾 Veritabanı                  │\n│  🔄 Menüyü Yenile               │\n│  ℹ️ Yardım                      │\n└─────────────────────────────────┘\n```\n\n### Available Commands\n\n| Command | Description |\n|---------|-------------|\n| `/start` | Initialize bot and show menu |\n| `/menu` | Display main control menu |\n| `/linkedin` | LinkedIn digest management |\n| `/status` | Quick system status |\n| `/scrape` | Trigger news scraping |\n| `/health` | Run health check |\n| `/help` | Show help and commands |\n\n### Setup\n\n```bash\n# 1. Create Telegram bot with @BotFather\n# 2. Get bot token and chat ID\n# 3. Add to environment variables\nTELEGRAM_BOT_TOKEN=your_token_here\nTELEGRAM_CHAT_ID=your_chat_id_here\n\n# 4. Setup bot menu\nnpm run telegram:setup-menu\n\n# 5. Test in Telegram\n# Send: /start\n```\n\n---\n\n## 🔄 Automation Workflows\n\n### Scheduled Jobs\n\n| Workflow | Schedule | Purpose |\n|----------|----------|---------|\n| **Scrape Tech News** | 07:00, 13:00, 15:00 UTC (M-F) | Deterministic today-only ingestion across 6 categories |\n| **Manual Article Scraper** | On-demand via Telegram | Process single articles |\n| **System Health Check** | 08:00 UTC (Daily) | Monitor system health |\n| **Vercel Status Monitor** | Every 30 minutes | Monitor Vercel platform status |\n| **LinkedIn Digest** | Daily via n8n | Generate \u0026 post digest |\n\n### Manual Triggers\n\nAll workflows can be triggered manually:\n\n```bash\n# Via Telegram Bot\n/menu      # Open interactive menu\n/scrape    # Trigger news scraping\n/health    # Run health check\n/linkedin  # Manage LinkedIn digests\n\n# Via npm scripts\nnpm run scrape:news          # Full scrape (no preflight — unlike scheduled CI)\nnpm run scrape:news:parity   # Preflight then scrape only if gate says proceed (matches scheduled Actions)\nnpm run scrape:news:replay -- --replay-file artifacts/tech-news-runs/\u003crun\u003e.json\nnpm run cleanup:duplicates:dry\nnpm run cleanup:duplicates\nnpm run health:check         # Health check\nnpm run vercel:status        # Check Vercel status\n```\n\n#### Manual / Forced URL Ingest\n\nWhen specific articles are missed by scheduled discovery, trigger `workflow_dispatch` and fill in the **force_urls** input:\n\n```\nforce_urls:  https://nuvemmag.com/article-1/,https://nuvemmag.com/article-2/\n```\n\nThe pipeline will bypass category discovery and process those URLs directly through the full pipeline (detail scrape → quality → translation → persistence). Normal duplicate and quality checks remain active.\n\nLocally:\n\n```bash\nnpm run scrape:news -- --force-urls \"https://nuvemmag.com/some-article/\"\n# or via env var\nTECH_NEWS_FORCE_URLS=\"https://nuvemmag.com/a/,https://nuvemmag.com/b/\" npm run scrape:news\n```\n\n---\n\n## 📐 Scrape Tech News — Architecture \u0026 Data Flow\n\nEnd-to-end view of the **Scrape Tech News** GitHub Actions workflow (`.github/workflows/scrape-tech-news.yml`) and the Node orchestrator (`scripts/lib/scraper/ScrapeOrchestrator.js`, CLI entry: `scripts/news-scraper.js`). Diagrams use [Mermaid](https://mermaid.js.org/) and render on GitHub; for local editing, use a preview that supports fenced `mermaid` blocks.\n\n### Workflow (CI/CD)\n\n```mermaid\nflowchart TB\n  subgraph triggers[\"Triggers\"]\n    CRON[\"Cron — weekdays 07:00 / 13:00 / 15:00 UTC\"]\n    WD[\"workflow_dispatch — manual\"]\n  end\n\n  subgraph job[\"Job: scrape-and-translate · ubuntu-latest\"]\n    S1[\"Checkout repository\"]\n    S2[\"Setup Node.js 22 + npm cache\"]\n    PF[\"Preflight gate — may skip npm ci + scrape on schedule\"]\n    S3[\"npm ci\"]\n    S4[\"Export run env + npm run scrape:news\"]\n    S5[\"Upload artifacts\"]\n    S6[\"Success banner\"]\n    S7[\"Telegram success via CI helper script\"]\n    S8[\"Telegram failure alert\"]\n    S1 --\u003e S2 --\u003e PF --\u003e S3 --\u003e S4 --\u003e S5\n    S4 --\u003e S6 --\u003e S7\n    S4 -.-\u003e|failure| S8\n  end\n\n  subgraph secrets[\"Secrets injected into step 4\"]\n    SEC[\"GROQ_API_KEY · GROQ_PARSER_API_KEY · FIRECRAWL_API_KEY\\nNEXT_PUBLIC_SUPABASE_* · SUPABASE_SERVICE_ROLE_KEY\\nTELEGRAM_BOT_TOKEN · TELEGRAM_CHAT_ID · OLLAMA_API_KEY\"]\n  end\n\n  triggers --\u003e job\n  secrets -.-\u003e S4\n```\n\n**Concurrency:** `group: tech-news-daily-agent-${{ github.ref }}` with `cancel-in-progress: false` so overlapping runs are not cancelled mid-flight.\n\n**Scheduled CI vs local `npm run scrape:news`:** On a schedule, Actions runs a **preflight** step (cheap headline URLs vs Supabase) and may skip `npm ci` and the full scraper when nothing new is expected. Locally, `npm run scrape:news` always runs the full pipeline. Use `npm run scrape:news:parity` to mirror scheduled behavior, or trigger **workflow_dispatch** in GitHub.\n\n**`workflow_dispatch` inputs:**\n- `skip_preflight` (boolean) — bypass the preflight headline gate and always run the full scraper.\n- `force_urls` (string) — comma-separated source URLs to force-ingest directly, skipping discovery (useful when specific articles are missed by the scheduled run).\n\n### Agent Architecture\n\nThe pipeline in `scripts/lib/scraper/ScrapeOrchestrator.js` consists of **6 agents**:\n\n| # | Agent | Responsibility |\n|---|-------|----------------|\n| 1 | **DiscoveryAgent** | Fetches category pages via `ScraperRouter` (Firecrawl → Cheerio fallback), runs AI list parsing + regex fallback, deduplicates candidates across categories. |\n| 2 | **DetailExtractionAgent** | Fetches the full article page, extracts title/description/content/date/embeds. |\n| 3 | **TranslationAgent** | Translates Turkish content to English via Groq, with placeholder swap to protect embed tokens. |\n| 4 | **EnhancementAgent** | Quality gate: rejects TL;DR-only content, validates English completeness. |\n| 5 | **QualityGateAgent** | Date integrity checks, garbage content detection, save disposition routing. |\n| 6 | **PersistenceAgent** | Duplicate detection (source URL, slug prefix, content hash) and Supabase insert. |\n\n### LLM Models\n\n| Role | Model | Provider |\n|------|-------|----------|\n| Translation (primary) | `llama-3.1-8b-instant` | Groq |\n| Translation (fallback) | `openai/gpt-oss-20b` | Groq |\n| Translation (last resort) | `llama-3.3-70b-versatile` | Groq |\n| List extraction / parser | `llama-3.1-8b-instant` | Groq |\n| Enhancement checks | `llama-3.1-8b-instant` | Groq |\n| Optional (content translation) | `gemini-3-flash-preview:cloud` | Ollama cloud |\n\n\u003e **Model tiering rationale:** the lightweight, high-throughput `llama-3.1-8b-instant`\n\u003e is the primary translation model so a full run does not exhaust the daily token\n\u003e budget (TPD) on the heavy 70B model. `llama-3.3-70b-versatile` is kept only as a\n\u003e last-resort quality fallback. Both Groq clients are configured with `maxRetries`\n\u003e and `timeout`, so transient connection drops (e.g. \"Premature close\") are retried\n\u003e automatically before the model cascade falls back.\n\nRequired secrets: `GROQ_API_KEY`, `GROQ_PARSER_API_KEY`, `OLLAMA_API_KEY` (optional fallback).\n\n### Slug Generation\n\nAll new articles use an **English-safe ASCII slug** generated from the translated English title:\n\n- Turkish and Latin-extended characters are transliterated before slug generation (e.g. `ğ→g`, `ü→u`, `ş→s`, `ı→i`, `ö→o`, `ç→c`).\n- Slugs are derived from the post-translation English title, not the Turkish source URL.\n- Existing Turkish slugs can be migrated using the guidance in `scripts/migrate/update-turkish-slugs.sql`.\n\n### Orchestration pipeline (application)\n\n```mermaid\nflowchart TB\n  subgraph entry[\"Entry\"]\n    A[\"scrapeNews() — ScraperRouter, run report, run label\"]\n    A --\u003e B[\"Telegram: run started\"]\n    B --\u003e C[\"getArticleCount()\"]\n  end\n\n  subgraph list[\"Category discovery — 6 configured feeds\"]\n    D[\"scrapeAllCategories()\"]\n    D --\u003e D1[\"scrapeArticleList per category\\nFirecrawl page · Groq list parse · regex merge\"]\n    D1 --\u003e D2[\"mergeArticleCandidates — dedupe URLs across categories\"]\n    D2 --\u003e E[\"Unique candidates + metrics\"]\n  end\n\n  C --\u003e D\n\n  subgraph dates[\"Date partitioning\"]\n    F[\"partitionCandidatesByDate\"]\n    F --\u003e P1[\"today\"]\n    F --\u003e P2[\"recent stale window\"]\n    F --\u003e P3[\"unknown\"]\n    F --\u003e P4[\"stale\"]\n    F --\u003e P5[\"future — rejected\"]\n  end\n\n  E --\u003e F\n\n  subgraph unk[\"Unknown candidates\"]\n    VU[\"verifyUnknownCandidates — optional detail scrape\"]\n  end\n\n  P3 --\u003e VU\n\n  subgraph db[\"Database gate\"]\n    G[\"actionable = today + recent + unknown\"]\n    G --\u003e H[\"getExistingArticles()\"]\n    H --\u003e I[\"missing only — cap at MAX_ARTICLES_PER_RUN\"]\n  end\n\n  P1 --\u003e G\n  P2 --\u003e G\n  VU --\u003e G\n\n  subgraph queue[\"processArticleQueue\"]\n    J[\"Per article: scrapeArticleDetails\"]\n    J --\u003e K[\"Clean content · embed tokens · garbage checks\"]\n    K --\u003e L{\"Valid for save?\"}\n    L --\u003e|no| M[\"skip / reject / defer\"]\n    L --\u003e|yes| N[\"translateArticle — Groq\"]\n    N --\u003e O[\"saveArticle — Supabase\"]\n    O --\u003e P[\"Circuit breaker on consecutive failures\"]\n  end\n\n  I --\u003e J\n\n  subgraph out[\"Outputs\"]\n    R1[\"JSON artifact under artifacts/tech-news-runs/\"]\n    R2[\"Telegram summary\"]\n    R1 --\u003e R2\n  end\n\n  J --\u003e R1\n\n  subgraph ext[\"External APIs\"]\n    FC[\"Firecrawl\"]\n    GQ[\"Groq\"]\n    SB[\"Supabase\"]\n    TG[\"Telegram\"]\n  end\n\n  D1 -.-\u003e FC\n  D1 -.-\u003e GQ\n  J -.-\u003e FC\n  N -.-\u003e GQ\n  O -.-\u003e SB\n  B -.-\u003e TG\n  R2 -.-\u003e TG\n```\n\n### Pipeline reference\n\n| Stage | Responsibility |\n|--------|----------------|\n| **Triggers** | Weekday cron (3×) or manual `workflow_dispatch`. |\n| **Environment** | `TECH_NEWS_RUN_DATE` (Europe/Istanbul) and `TECH_NEWS_RUN_LABEL` tie logs, artifacts, and Telegram to a single run. |\n| **Discovery** | `ScraperRouter` → Firecrawl for HTML/markdown (up to 2 archive pages per category, 25 articles/page); Groq extracts structured article rows; regex supplements; URL merge removes cross-category duplicates. Legacy `/post/` URL 404s trigger an automatic canonical-URL retry. |\n| **Dates** | Candidates classified; future dates rejected; mismatches between list and detail dates can **defer** work. |\n| **Deduplication** | Bulk Supabase lookup before expensive translation; `source_url` / slug rules apply. |\n| **Processing** | Detail scrape → cleaning \u0026 embed preservation → Groq translation with quality gates → `tech_news_articles` insert. |\n| **Artifacts** | Each run writes a replayable JSON report (uploaded even if a later step fails, `if: always()`). |\n| **Notifications** | In-run Telegram messages from the script; workflow success step formats the final summary via `scripts/ci/telegram-tech-news-success-message.cjs`. |\n\n---\n\n## 🧠 Daily News Agent\n\nThe tech news pipeline now behaves as a deterministic daily agent, not an open-ended crawler.\n\n### Target Categories\n\n- `yapay-zeka`\n- `teknoloji`\n- `yapay-zeka-uygulamalari`\n- `gundem`\n- `surdurulebilirlik`\n- `bilim-ve-dunya`\n\n### Decision Flow\n\n1. Discover candidates from the six configured category endpoints, scanning up to 2 archive pages per category (25 articles per page, newest-first).\n2. Normalize each candidate date in Turkey time and classify it as `today`, `unknown`, `stale`, or `future`.\n3. `stale` articles within the 5-day recency window (`MAX_RECENT_PUBLISH_DAYS`) are still eligible — this safety net catches articles missed in a previous run due to the per-page discovery cap.\n4. Bulk-check normalized `source_url` values in Supabase before expensive work.\n5. Skip candidates that already exist in the database.\n6. Verify `unknown` candidates with detail metadata before translation or save.\n7. Translate, validate, and save only valid missing articles.\n8. Write a replayable JSON artifact for every run under `artifacts/tech-news-runs/`.\n\n### Operational Rules\n\n- `scripts/news-scraper.js` is CLI entrypoint only.\n- `scripts/lib/scraper/ScrapeOrchestrator.js` is the active orchestration and decision engine.\n- Scrapers discover candidates; orchestration decides whether they move forward.\n- Unknown dates must never silently become \"today\".\n- **Slugs are always generated from the translated English title** (via `generateSlug` in `scripts/lib/scraper/slugUtils.js`). The Turkish source URL slug is intentionally ignored for new records.\n- Future or inconsistent dates are deferred or rejected, not normalized into production data.\n- Replay only helps after conflicting bad rows are deleted or repaired.\n\n### Run Output\n\nEach run records business-facing metrics including:\n\n- `rawFound`\n- `todayCandidates`\n- `unknownCandidates`\n- `alreadyInDb`\n- `verifiedUnknown`\n- `staleSkipped`\n- `futureRejected`\n- `saved`\n- `failed`\n- `deferred`\n\nThis makes it easy to answer \"what happened in this run?\" without reading raw logs.\n\n---\n\n## ⚙️ Configuration\n\n### Environment Variables\n\n#### Required for Development\n\n```env\n# AI Services\nGROQ_API_KEY=gsk_your_key_here                    # Groq AI for translation \u0026 chat\nGROQ_PARSER_API_KEY=gsk_your_parser_key_here     # Groq parser model for list extraction\nFIRECRAWL_API_KEY=fc-your_key_here                # Firecrawl for scraping\nGEMINI_API_KEY=your_gemini_key_here               # Google Gemini for content gen\nOLLAMA_API_KEY=your_ollama_cloud_key_here         # Ollama cloud API key for content translation\n\n# Database\nNEXT_PUBLIC_SUPABASE_URL=https://...              # Supabase project URL\nNEXT_PUBLIC_SUPABASE_ANON_KEY=eyJ...              # Public anon key\nSUPABASE_SERVICE_ROLE_KEY=eyJ...                  # Service role key (admin)\n\n# Telegram Bot\nTELEGRAM_BOT_TOKEN=123456:ABC...                  # Bot token from @BotFather\nTELEGRAM_CHAT_ID=123456789                        # Your chat ID\n```\n\n#### Optional (Production)\n\n```env\n# Security\nTELEGRAM_CONTROL_API_SECRET=random_key            # API endpoint security\nDEPLOYMENT_WEBHOOK_SECRET=random_hex              # Deployment webhook security\n\n# Sentry Error Tracking\nVITE_SENTRY_DSN=https://key@org.ingest.sentry.io/project\nSENTRY_DSN=https://key@org.ingest.sentry.io/project\n\n# GitHub Integration\nGITHUB_TOKEN=ghp_...                              # For workflow triggers\nGITHUB_REPOSITORY=username/repo                   # Repository name\n\n# n8n Integration\nN8N_LINKEDIN_WORKFLOW_WEBHOOK=https://...         # LinkedIn digest workflow\nN8N_CHATBOT_WEBHOOK=https://...                   # Chatbot fallback webhook\n```\n\n### GitHub Secrets\n\nAdd these in: `Repository Settings → Secrets and variables → Actions`\n\n```\nGROQ_API_KEY\nGROQ_PARSER_API_KEY\nFIRECRAWL_API_KEY\nGEMINI_API_KEY\nOLLAMA_API_KEY\nNEXT_PUBLIC_SUPABASE_URL\nNEXT_PUBLIC_SUPABASE_ANON_KEY\nSUPABASE_SERVICE_ROLE_KEY\nTELEGRAM_BOT_TOKEN\nTELEGRAM_CHAT_ID\n```\n\n---\n\n## 🚢 Deployment\n\n### Vercel (Recommended)\n\n#### Quick Deploy\n\n[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https://github.com/CemRoot/My-Site)\n\n#### Manual Deploy\n\n```bash\n# 1. Install Vercel CLI\nnpm i -g vercel\n\n# 2. Login to Vercel\nvercel login\n\n# 3. Deploy\nvercel\n\n# 4. Deploy to production\nvercel --prod\n```\n\n### Post-Deployment\n\n```bash\n# Automatic (via postbuild hook)\n# - Bot commands configured\n# - Telegram notification sent\n# - Menu updated\n\n# Manual setup (if needed)\nnpm run telegram:setup-menu\n```\n\n---\n\n## 📊 Monitoring \u0026 Observability\n\n### 🔍 Error Tracking (Sentry)\n\n**Real-time error monitoring across frontend and backend:**\n\n- Automatic error capture (React Error Boundaries)\n- Performance monitoring (Core Web Vitals)\n- Session replay\n- Source maps for readable stack traces\n- Release tracking per deployment\n\n### 🏥 Health Checks\n\n```bash\n# Comprehensive system health check\nnpm run health:check\n\n# Automated (daily at 08:00 UTC)\n✅ Supabase database connectivity\n✅ Firecrawl API availability  \n✅ Groq AI API status\n✅ Telegram Bot responsiveness\n✅ Vercel platform health\n✅ Recent article metrics\n```\n\n### 📱 Telegram Notifications\n\n| Event | Notification Type | Frequency |\n|-------|------------------|-----------|\n| ✅ **Scraping Success** | Success report + stats | Per workflow run |\n| ❌ **Scraping Errors** | Error details + logs | Immediate |\n| 🏥 **Health Reports** | System status summary | Daily |\n| 🚀 **Deploy Success** | Deployment confirmed | Per deployment |\n| 🚨 **Vercel Incidents** | Platform status alerts | Real-time |\n\n---\n\n## 📁 Project Structure\n\n```\nMy-Site/\n├── api/                          # Vercel Serverless Functions\n│   ├── lib/                      # Shared API modules\n│   │   ├── supabaseAdmin.js      # Shared Supabase admin client\n│   │   ├── telegram.js           # Shared Telegram utilities\n│   │   ├── chatHelpers.js        # Chat endpoint helpers\n│   │   └── chatSystemPrompt.js   # AI system prompt\n│   ├── chat.js                   # AI Chatbot endpoint\n│   ├── tech-news.js              # News API (Edge Runtime)\n│   ├── telegram-webhook.js       # Telegram bot webhook\n│   ├── telegram-control.js       # Bot control endpoint\n│   ├── newsletter.js             # Newsletter subscription\n│   ├── og-meta.js                # Dynamic Open Graph meta\n│   ├── deployment-webhook.js     # Deploy notifications\n│   ├── frontend-health-monitor.js# Error monitoring\n│   ├── conversation-state.js     # Telegram state management\n│   └── revalidate-news.js        # News cache revalidation\n├── lib/                          # Server-side shared libraries\n│   ├── rate-limit.js             # Rate limiting\n│   ├── sentry-server.js          # Sentry integration\n│   ├── supabase.js               # Database client\n│   └── conversation-state.js     # Conversation state logic\n├── scripts/                      # Automation \u0026 CI scripts\n│   ├── lib/                      # Shared script modules\n│   │   ├── config.js             # Centralized env config\n│   │   ├── supabaseAdmin.js      # Shared Supabase client\n│   │   ├── telegram.js           # Shared Telegram utilities\n│   │   ├── scraper/              # News scraper sub-modules\n│   │   │   ├── config.js         # Scraper configuration\n│   │   │   ├── database.js       # Article storage \u0026 dedup\n│   │   │   ├── dateUtils.js      # Date parsing utilities\n│   │   │   └── translator.js     # AI translation pipeline\n│   │   └── menu/                 # Telegram bot menu modules\n│   │       └── keyboards.js      # Keyboard layouts\n│   ├── news-scraper.js           # News scraping CLI entrypoint\n│   ├── telegram-menu-handler.js  # Telegram bot menu handler\n│   ├── manual-article-scraper.js # Manual article processing\n│   ├── system-health-check.js    # Health monitoring\n│   ├── validation/               # Content validation\n│   └── translate/                # Translation prompts\n├── src/                          # React Frontend\n│   ├── components/               # UI Components\n│   │   ├── ui/                   # shadcn/radix primitives\n│   │   ├── chat/                 # Chat widget sub-components\n│   │   ├── embeds/               # Social media embeds\n│   │   └── markdown/             # Markdown rendering\n│   ├── pages/                    # Page components\n│   ├── lib/                      # Frontend shared modules\n│   │   ├── constants/            # Centralized constants\n│   │   ├── types/                # Shared TypeScript types\n│   │   ├── hooks/                # Custom React hooks\n│   │   ├── utils/                # Utility functions\n│   │   └── context/              # React context providers\n│   └── styles/                   # CSS files\n└── public/                       # Static assets\n```\n\n---\n\n## 📜 Scripts Reference\n\n### Development\n```bash\nnpm run dev                     # Start dev server (port 3000)\nnpm run build                   # Production build\n```\n\n### Tech News System\n```bash\nnpm run scrape:news             # Full scrape (no preflight)\nnpm run scrape:news:parity      # Preflight + conditional scrape (like scheduled CI)\nnpm run scrape:news:replay -- --replay-file artifacts/tech-news-runs/\u003crun\u003e.json\nnpm run cleanup:duplicates:dry  # Preview cleanup of bad legacy rows\nnpm run cleanup:duplicates      # Remove bad legacy rows and snapshot them\nnpm run cleanup:db              # Clean up old/invalid articles\n```\n\n### LinkedIn Automation\n```bash\nnpm run linkedin:analyze        # Analyze content for posting\nnpm run linkedin:post           # Post to LinkedIn\nnpm run linkedin:test           # Test LinkedIn workflow\nnpm run linkedin:groups         # Daily LinkedIn groups digest\nnpm run linkedin:groups-weekly  # Weekly LinkedIn groups digest\n```\n\n### Telegram Bot\n```bash\nnpm run telegram:setup-menu     # Setup bot menu\nnpm run telegram:webhook-setup  # Configure webhook\nnpm run telegram:webhook-remove # Remove webhook\nnpm run telegram:reset          # Reset webhook \u0026 clear queue\nnpm run telegram:check          # Check webhook status\n```\n\n### n8n Trial Management\n```bash\nnpm run n8n:status              # Check trial status\nnpm run n8n:check               # Check \u0026 send notification\nnpm run n8n:reset               # Reset trial period\n```\n\n### Monitoring\n```bash\nnpm run health:check            # System health check\nnpm run vercel:status           # Check Vercel status\n```\n\n---\n\n## 🔒 Security\n\n### Reporting Vulnerabilities\n\nPlease report security vulnerabilities to: **cemkoyluoglu@icloud.com**\n\n### Security Features\n\n- ✅ **API Rate Limiting** (10 requests/minute per user)\n- ✅ **UUID Validation** (RFC 4122 compliant)\n- ✅ **Chat ID Authorization** (Telegram webhook)\n- ✅ **Environment Variable Encryption**\n- ✅ **Webhook Signature Verification**\n- ✅ **SQL Injection Prevention** (Parameterized queries)\n- ✅ **XSS Protection** (HTML sanitization)\n- ✅ **CORS Configuration** (Origin whitelisting)\n\n---\n\n## 🎯 Roadmap\n\n### ✅ Completed\n- [x] Multi-category news scraping\n- [x] AI-powered translation (Groq)\n- [x] Google Gemini content generation\n- [x] AI Chatbot with n8n fallback\n- [x] Telegram bot control system\n- [x] GitHub Actions automation\n- [x] Supabase migration\n- [x] Health monitoring system\n- [x] LinkedIn digest automation\n- [x] Newsletter system\n- [x] Manual article scraper\n- [x] Chat history persistence\n- [x] Smart content validation\n\n### 🚧 In Progress\n- [ ] RSS feed generation\n- [ ] Article search functionality\n- [ ] Advanced analytics dashboard\n\n### 📋 Planned\n- [ ] Multi-language support (Spanish, French)\n- [ ] Mobile app (React Native)\n- [ ] AI-powered article summarization\n- [ ] Newsletter email delivery\n\n---\n\n## 🤝 Contributing\n\nWe welcome contributions! \n\n### Development Workflow\n\n```bash\n# 1. Fork the repository\n# 2. Create your feature branch\ngit checkout -b feature/AmazingFeature\n\n# 3. Commit your changes\ngit commit -m 'Add some AmazingFeature'\n\n# 4. Push to the branch\ngit push origin feature/AmazingFeature\n\n# 5. Open a Pull Request\n```\n\n---\n\n## 🏆 Acknowledgments\n\n- [Groq AI](https://groq.com/) - Lightning-fast AI inference\n- [Google Gemini](https://ai.google.dev/) - Advanced content generation\n- [Firecrawl](https://firecrawl.dev/) - Reliable web scraping\n- [Supabase](https://supabase.com/) - Open-source Firebase alternative\n- [Vercel](https://vercel.com/) - Seamless deployment platform\n- [n8n](https://n8n.io/) - Workflow automation\n- [Telegram](https://telegram.org/) - Secure messaging platform\n\n---\n\n## 📞 Contact \u0026 Support\n\n\u003cdiv align=\"center\"\u003e\n\n### Dr. Cem Koyluoglu (CK)\n\n[![Email](https://img.shields.io/badge/Email-cemkoyluoglu%40icloud.com-red?style=for-the-badge\u0026logo=gmail)](mailto:cemkoyluoglu@icloud.com)\n[![LinkedIn](https://img.shields.io/badge/LinkedIn-Cem%20Koyluoglu-blue?style=for-the-badge\u0026logo=linkedin)](https://www.linkedin.com/in/cem-koyluoglu/)\n[![GitHub](https://img.shields.io/badge/GitHub-CemRoot-black?style=for-the-badge\u0026logo=github)](https://github.com/CemRoot)\n[![WhatsApp](https://img.shields.io/badge/WhatsApp-%2B353%2087%20344%205918-green?style=for-the-badge\u0026logo=whatsapp)](https://wa.me/353873445918)\n\n📍 **Dublin, Ireland** | 🌍 **Available Worldwide**\n\n\u003c/div\u003e\n\n---\n\n## 📄 License\n\nThis project is licensed under the **MIT License** - see the [LICENSE](LICENSE) file for details.\n\n```\nMIT License\n\nCopyright (c) 2025 Cem Koyluoglu\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n```\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n### ⭐ If you find this project useful, please consider giving it a star!\n\n**Made with ❤️ in Dublin, Ireland**\n\n[⬆ Back to Top](#-tech-news-automation-platform)\n\n---\n\n**Last Updated**: March 30, 2026 | **Version**: 3.0.0\n\n[![Built with Love](https://img.shields.io/badge/Built%20with-❤️-red?style=flat-square)](https://github.com/CemRoot/My-Site)\n[![Maintained](https://img.shields.io/badge/Maintained-Yes-green?style=flat-square)](https://github.com/CemRoot/My-Site)\n[![PRs Welcome](https://img.shields.io/badge/PRs-Welcome-brightgreen?style=flat-square)](http://makeapullrequest.com)\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcemroot%2Fmy-site","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcemroot%2Fmy-site","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcemroot%2Fmy-site/lists"}