{"id":49443103,"url":"https://github.com/asrat77/adsensie","last_synced_at":"2026-04-29T21:02:32.398Z","repository":{"id":325146268,"uuid":"1100021110","full_name":"Asrat77/AdSensie","owner":"Asrat77","description":"Smart Telegram Analytics For Ad Placement.","archived":false,"fork":false,"pushed_at":"2025-11-19T18:57:36.000Z","size":378,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-11-19T20:20:25.747Z","etag":null,"topics":["clickhouse","postgresql","redis","ruby","ruby-on-rails","sidekiq"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Asrat77.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-19T18:29:25.000Z","updated_at":"2025-11-19T19:11:48.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Asrat77/AdSensie","commit_stats":null,"previous_names":["asrat77/adsensie"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/Asrat77/AdSensie","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asrat77%2FAdSensie","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asrat77%2FAdSensie/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asrat77%2FAdSensie/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asrat77%2FAdSensie/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Asrat77","download_url":"https://codeload.github.com/Asrat77/AdSensie/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asrat77%2FAdSensie/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32443576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T20:22:27.477Z","status":"ssl_error","status_checked_at":"2026-04-29T20:22:26.507Z","response_time":110,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","postgresql","redis","ruby","ruby-on-rails","sidekiq"],"created_at":"2026-04-29T21:02:31.572Z","updated_at":"2026-04-29T21:02:32.393Z","avatar_url":"https://github.com/Asrat77.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg width=\"100\" height=\"100\" alt=\"AdSensie\" src=\"https://github.com/user-attachments/assets/43b1b5d1-a62c-4334-90a5-44fd89f4f718\" /\u003e\n\u003c/div\u003e\n\n\n\nA Rails/Postgres/ClickHouse/Redis/Sidekiq analytics platform to help advertisers identify the best Telegram channels for ad placement through data-driven metrics.\n\n## 🎯 Features\n- **Channel Discovery**: Browse and search channels\n- **Advanced Filtering**: Filter by engagement rate, subscriber count, growth rate\n- **Smart Sorting**: Sort by engagement, subscribers, growth, or activity\n- **Collections**: Organize channels into custom lists for campaigns\n- **Real-time Data**: Fetch live data from Telegram API\n- **Analytics Dashboard**: View key metrics and trending channels with interactive charts\n- **ClickHouse Integration**: Lightning-fast analytics queries on large datasets\n- **Performance Monitoring**: Compare PostgreSQL vs ClickHouse query performance\n- **Automatic Data Sync**: Scheduled syncing between PostgreSQL and ClickHouse\n\n## 🚀 Quick Start\n### Prerequisites\n- Ruby 3.4.1+\n- PostgreSQL\n- Redis (for Sidekiq background jobs)\n- Docker (for ClickHouse)\n- Python 3.7+ (for Telegram data fetching)\n- Telegram API credentials\n\n### Installation\n1. **Clone and setup**\n   ```bash\n   git clone \u003cyour-repo-url\u003e\n   cd ad_sensie\n   bundle install\n   ```\n\n2. **Configure environment variables**\n  \n   Create a `.env` file (DO NOT commit this file):\n   ```env\n   # Telegram API Credentials\n   # Get these from https://my.telegram.org/apps\n   TELEGRAM_API_ID=your_api_id_here\n   TELEGRAM_API_HASH=your_api_hash_here\n  \n   # Database Configuration\n   DATABASE_USERNAME=postgres\n   DATABASE_PASSWORD=your_secure_password\n  \n   # Redis URL (optional, defaults to localhost)\n   REDIS_URL=redis://localhost:6379/1\n   ```\n\n3. **Start Services via Docker**\n  \n   If you don't have these services installed locally, run them with Docker:\n   **PostgreSQL** (Main Database):\n   ```bash\n   docker run -d \\\n     --name postgres \\\n     -e POSTGRES_USER=postgres \\\n     -e POSTGRES_PASSWORD=password \\\n     -p 5432:5432 \\\n     postgres:16\n   ```\n   **Redis** (Caching \u0026 Background Jobs):\n   ```bash\n   docker run -d \\\n     --name redis \\\n     -p 6379:6379 \\\n     redis:7-alpine\n   ```\n   **ClickHouse** (Analytics):\n   ```bash\n   docker run -d \\\n     --name clickhouse-server \\\n     -p 8123:8123 -p 9000:9000 \\\n     --ulimit nofile=262144:262144 \\\n     clickhouse/clickhouse-server\n   ```\n\n4. **Configure database**\n   ```bash\n   bin/rails db:create db:migrate\n   ```\n\n5. **Load seed data** (optional)\n   ```bash\n   bin/rails db:seed\n   ```\n\n6. **Initialize ClickHouse**\n   ```bash\n   docker exec -i clickhouse-server clickhouse-client \u003c db/clickhouse/schema.sql\n   bin/rails clickhouse:sync\n   ```\n\n7. **Start the application**\n   ```bash\n   bin/dev\n   ```\n  \n   This starts:\n   - Rails server (port 3000)\n   - Sidekiq (background jobs)\n   - Tailwind CSS watcher\n\n8. **Access the application**\n   - Open http://localhost:3000\n   - Login with: `email` / `password`\n\n## 📡 Fetching Real Telegram Data\n### Setup Telegram API\n1. **Get API credentials**:\n   - Visit https://my.telegram.org/apps\n   - Create a new application\n   - Copy your `api_id` and `api_hash`\n   - Add them to your `.env` file\n\n2. **Edit the channel list** in `lib/telegram/fetch_channels.py`:\n   ```python\n   channels = [\n       '@EthioTechNews',\n       '@DevEthiopia',\n       '@EthioJobsTech',\n       # Add more channels...\n   ]\n   ```\n\n3. **Run the fetcher** (first time will require phone authentication):\n   ```bash\n   python3 lib/telegram/fetch_channels.py\n   ```\n  \n   You'll be prompted to enter your phone number and verification code.\n\n4. **Import data into Rails**:\n   ```bash\n   bin/rails telegram:import\n   ```\n\n5. **Or do both in one command**:\n   ```bash\n   bin/rails telegram:fetch_and_import\n   ```\n\n## 📊 Database Architecture\n### PostgreSQL (Source of Truth)\n- **Users**: Authentication with Devise\n- **Channels**: Telegram channel information and metrics\n- **Posts**: Individual channel posts with engagement data\n- **Collections**: User-created channel lists\n- **CollectionChannels**: Join table for collections\n\n### ClickHouse (Analytics Engine)\n- **channels_analytics**: Optimized for fast aggregations\n- **posts_analytics**: Time-series partitioned by month\n- **engagement_metrics_mv**: Materialized view for pre-aggregated metrics\n\n### Data Flow\n```\nTelegram API → PostgreSQL → ClickHouse (every 10 min) → Dashboard\n```\n\n## 🎨 Key Pages\n- `/` - Dashboard with overview and trending channels\n- `/channels` - Browse all channels with search and filters\n- `/channels/:id` - Channel detail page\n- `/channels/compare` - Compare multiple channels side-by-side\n- `/collections` - Manage your channel collections\n- `/performance` - PostgreSQL vs ClickHouse performance comparison\n\n## 🔧 Development\n### Run tests\n```bash\n# Tests not yet implemented\n```\n\n### Rails console\n```bash\nbin/rails console\n```\n\n### Check routes\n```bash\nbin/rails routes\n```\n\n### ClickHouse queries\n```bash\n# Via Docker\ndocker exec clickhouse-server clickhouse-client --database=adsensie_analytics\n# Example query\ndocker exec clickhouse-server clickhouse-client --query=\"\n  SELECT toDate(posted_at) as date, COUNT(*) as posts\n  FROM adsensie_analytics.posts_analytics\n  WHERE posted_at \u003e= now() - INTERVAL 7 DAY\n  GROUP BY date\n  ORDER BY date\n\"\n```\n\n## 📝 Available Rake Tasks\n```bash\n# Telegram data\nbin/rails telegram:import # Import from JSON\nbin/rails telegram:fetch_and_import # Fetch and import\n# ClickHouse\nbin/rails clickhouse:sync # Sync all data to ClickHouse\nbin/rails clickhouse:stats # Show ClickHouse statistics\n# Database\nbin/rails db:reset # Reset and reload seed data\n```\n\n## 🗂️ Project Structure\n```\nad_sensie/\n├── app/\n│ ├── controllers/ # Dashboard, Channels, Collections, Performance\n│ ├── models/ # User, Channel, Post, Collection\n│ │ └── clickhouse/ # ClickHouse models\n│ ├── services/ # AnalyticsService, ClickhouseSyncService\n│ ├── jobs/ # TelegramSyncJob, ClickhouseSyncJob\n│ ├── views/ # ERB templates with Tailwind CSS\n│ └── helpers/ # View helpers\n├── lib/\n│ ├── telegram/ # Python scripts for Telegram API\n│ └── tasks/ # Rake tasks (telegram, clickhouse)\n├── db/\n│ ├── migrate/ # PostgreSQL migrations\n│ ├── clickhouse/ # ClickHouse schema\n│ └── seeds.rb # Mock data generator\n└── config/\n    ├── routes.rb # Application routes\n    ├── database.yml # PostgreSQL configuration\n    ├── clickhouse.yml # ClickHouse configuration\n    └── schedule.rb # Cron jobs (whenever gem)\n```\n\n## 📚 Tech Stack\n- **Backend**: Ruby on Rails 8.0\n- **Databases**:\n  - PostgreSQL (transactional data)\n  - ClickHouse (analytics)\n  - Redis (caching, Sidekiq)\n- **Frontend**: Tailwind CSS, Turbo, Stimulus\n- **Charts**: Chartkick + Chart.js\n- **Authentication**: Devise\n- **Background Jobs**: Sidekiq\n- **Scheduling**: Whenever (cron)\n- **Telegram API**: Telethon (Python)\n\n## ⚡ Performance\n### ClickHouse Benefits\n- **10-100x faster** for analytical queries\n- **10x better compression** than PostgreSQL\n- **Handles billions of rows** with sub-second queries\n- **Columnar storage** optimized for aggregations\n\n### When to Use Each Database\n- **PostgreSQL**: User accounts, collections, real-time updates\n- **ClickHouse**: Analytics, dashboards, reports, time-series data\n\n## 🎯 Roadmap\n### Completed ✅\n- [x] Channel discovery and filtering\n- [x] Collections management\n- [x] Analytics dashboard with charts\n- [x] ClickHouse integration\n- [x] Performance monitoring\n- [x] Automatic data syncing\n\n### In Progress 🚧\n- [ ] Real-time sync feedback UI\n- [ ] Advanced search with multiple criteria\n- [ ] Date range filters\n\n### Planned 📋\n- [ ] Email reports\n- [ ] CSV/PDF export\n- [ ] Advanced analytics (cohort, retention)\n- [ ] API endpoints\n- [ ] Mobile responsive improvements\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasrat77%2Fadsensie","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasrat77%2Fadsensie","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasrat77%2Fadsensie/lists"}