{"id":48661038,"url":"https://github.com/askb/ai-n8domata","last_synced_at":"2026-04-10T10:09:36.600Z","repository":{"id":312993034,"uuid":"1049364007","full_name":"askb/ai-n8domata","owner":"askb","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-11T03:51:54.000Z","size":183,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-09-11T07:03:23.382Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/askb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-02T21:42:56.000Z","updated_at":"2025-09-11T03:51:58.000Z","dependencies_parsed_at":"2025-09-03T09:28:21.317Z","dependency_job_id":"17dbf2a5-c5c1-4a67-bf95-8a6b580a24c7","html_url":"https://github.com/askb/ai-n8domata","commit_stats":null,"previous_names":["askb/ai-n8domata"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/askb/ai-n8domata","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/askb%2Fai-n8domata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/askb%2Fai-n8domata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/askb%2Fai-n8domata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/askb%2Fai-n8domata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/askb","download_url":"https://codeload.github.com/askb/ai-n8domata/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/askb%2Fai-n8domata/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31637771,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-10T07:40:12.752Z","status":"ssl_error","status_checked_at":"2026-04-10T07:40:11.664Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-10T10:09:36.020Z","updated_at":"2026-04-10T10:09:36.585Z","avatar_url":"https://github.com/askb.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# N8N AI Services Platform\n\nA comprehensive Docker Compose stack for running N8N workflow automation with integrated AI services, auto-scaling, and monitoring capabilities.\n\n\u003e **Latest Update:** Now includes Cloudflare tunnel support, refactored auto-scaling, and comprehensive AI service integration.\n\n## 🏗️ System Architecture\n\nThis platform combines N8N workflow automation with a suite of AI services, providing a complete automation and AI processing solution:\n\n- **N8N Workflow Engine** - Main automation platform with queue-based processing\n- **Dynamic Auto-Scaling** - Intelligent worker scaling based on queue metrics  \n- **AI Services** - Video generation, text-to-speech, image processing, and more\n- **Real-time Monitoring** - Queue metrics and system observability\n- **Reverse Proxy** - Traefik with SSL termination and routing\n- **Data Persistence** - PostgreSQL database and Redis queue management\n\n### Architecture Diagram\n\n```mermaid\ngraph TB\n    subgraph \"External Access\"\n        CF[Cloudflare Tunnel]\n        TR[Traefik Reverse Proxy]\n    end\n    \n    subgraph \"N8N Core\"\n        N8N[N8N Main Instance]\n        NW[N8N Workers - Auto-scaled]\n        NH[N8N Webhook Handler]\n    end\n    \n    subgraph \"Data Layer\"\n        PG[PostgreSQL Database]\n        RD[Redis Queue]\n    end\n    \n    subgraph \"AI Services\"\n        SVM[Short Video Maker]\n        AIA[AI Agents API]\n        SD[Stable Diffusion]\n        TTS[Text-to-Speech]\n        CR[Intelligent Cropper]\n    end\n    \n    subgraph \"Platform Services\"\n        QM[Queue Metrics Monitor]\n        DS[Dynamic Scaler]\n        BK[Backup Service]\n        BR[Baserow Database]\n        MC[MinIO Object Storage]\n    end\n    \n    CF --\u003e TR\n    TR --\u003e N8N\n    TR --\u003e NH\n    TR --\u003e SVM\n    TR --\u003e AIA\n    TR --\u003e SD\n    \n    N8N --\u003e PG\n    N8N --\u003e RD\n    NW --\u003e PG\n    NW --\u003e RD\n    NH --\u003e PG\n    \n    QM --\u003e RD\n    DS --\u003e RD\n    DS --\u003e NW\n    \n    BK --\u003e PG\n    BK --\u003e N8N\n```\n\n## ✨ Features\n\n- **🔄 Dynamic Auto-Scaling** - Intelligent worker scaling based on queue depth\n- **🤖 Integrated AI Services** - Video generation, image processing, TTS, and more\n- **📊 Real-time Monitoring** - Queue metrics with structured logging\n- **🔒 Security First** - Cloudflare tunnels, non-root containers, encrypted data\n- **🚀 Production Ready** - Health checks, backups, and graceful shutdowns\n- **📦 Easy Deployment** - Single Docker Compose command\n- **🖥️ CPU Processing** - All AI services run on CPU (GPU support planned for future versions)\n- **🔧 Highly Configurable** - Environment-based configuration\n\n## 🚀 Quick Start Guide\n\n### Prerequisites\n\n- **Docker \u0026 Docker Compose v2+** - Container orchestration\n- **System Requirements:**\n  - 16GB RAM minimum (32GB+ recommended for AI services)\n  - 100GB+ free disk space\n  - Linux host (tested on Ubuntu/Fedora)\n- **Note:** Current version runs on CPU only (GPU acceleration coming in future versions)\n- **Cloudflare Account** - For secure tunneling (recommended)\n\n### Step 1: Repository Setup\n\n```bash\n# Clone the repository\ngit clone https://github.com/askb/ai-n8domata.git\ncd ai-n8domata\n\n# Initialize submodules for AI services\ngit submodule update --init --recursive\n\n# Create Docker network (for external integrations)\ndocker network create shark\n```\n\n### Step 2: Environment Configuration\n\n```bash\n# Copy the example environment file\ncp .env.example .env\n\n# Edit configuration (IMPORTANT: Set your own passwords!)\nnano .env\n```\n\n**🔑 Critical Environment Variables to Configure:**\n\n```bash\n# Cloudflare Tunnel (get from cloudflare.com)\nCLOUDFLARE_TUNNEL_TOKEN=your-tunnel-token-here\n\n# Security (GENERATE NEW VALUES!)\nN8N_ENCRYPTION_KEY=your-32-character-encryption-key\nPOSTGRES_PASSWORD=your-secure-database-password\n\n# Domains (configure your subdomains)\nN8N_HOST=n8n.yourdomain.com\nN8N_WEBHOOK=webhook.yourdomain.com\nTRAEFIK_HOST=traefik.yourdomain.com\n\n# AI Service APIs (optional but recommended)\nPEXELS_API_KEY=your-pexels-api-key\nOPENAI_API_KEY=your-openai-api-key\n```\n\n### Step 3: Start the Platform\n\n**Option A: Core Services Only (Recommended for First Run)**\n```bash\ndocker compose --profile core up -d\n```\n\n**Option B: Core + GPU AI Services**\n```bash\ndocker compose --profile core --profile anim up -d\n```\n\n**Option C: All Services (Resource Intensive)**\n```bash\ndocker compose --profile core --profile anim --profile disabled up -d\n```\n\n### Step 4: Verify Deployment\n\n```bash\n# Check service status\ndocker compose ps\n\n# Monitor startup logs\ndocker compose logs -f n8n-main n8n-queue-metrics n8n-dynamic-scaler\n\n# Check health status\ndocker compose ps --format \"table {{.Name}}\\t{{.Status}}\\t{{.Health}}\"\n```\n\n### Step 5: Access Your Platform\n\n- **🔧 N8N Workflow Editor:** `https://n8n.yourdomain.com`\n- **📊 Traefik Dashboard:** `https://traefik.yourdomain.com`\n- **🤖 AI Agents API:** `https://ai-agent.yourdomain.com/docs`\n- **🎥 Video Maker:** `https://video-maker.yourdomain.com`\n- **🎨 Stable Diffusion:** `https://stable-diffusion.yourdomain.com`\n\n## ⚙️ Service Profiles Explained\n\nThe platform uses Docker Compose profiles to manage different service groups efficiently:\n\n### 🚀 Core Profile (`--profile core`)\n**Essential services for N8N operation:**\n- N8N main instance, workers, and webhook handler\n- PostgreSQL database and Redis queue\n- Queue metrics monitoring and dynamic auto-scaling\n- Traefik reverse proxy and Cloudflare tunnel\n- Automated backup service\n- Basic AI services (CPU-based video maker, AI agents)\n- MinIO object storage and Baserow database\n\n**Resource Usage:** ~8GB RAM, 4 CPU cores\n\n### 🎨 Animation Profile (`--profile anim`)\n**GPU-accelerated AI services for advanced content generation:**\n- Stable Diffusion WebUI (image generation)\n- Stable Video Diffusion via ComfyUI\n- WAN2.1 Video Generation\n- CogVideo (experimental video AI)\n- CPU-optimized processing\n\n**Resource Usage:** +16GB RAM, GPU required\n\n### 🧪 Disabled Profile (`--profile disabled`)\n**Experimental and alternative services:**\n- ROCm-enabled alternatives to CPU services\n- Development/testing versions\n- Experimental AI models\n\n**Resource Usage:** Variable\n\n## 🔧 Configuration Deep Dive\n\n### N8N Core Configuration\n\n```bash\n# Queue-based execution (required for auto-scaling)\nEXECUTIONS_MODE=queue\nQUEUE_BULL_REDIS_HOST=redis\nQUEUE_HEALTH_CHECK_ACTIVE=true\n\n# Performance tuning\nN8N_CONCURRENCY_PRODUCTION_LIMIT=10          # Tasks per worker\nN8N_QUEUE_BULL_GRACEFULSHUTDOWNTIMEOUT=300   # Graceful shutdown time\nN8N_GRACEFUL_SHUTDOWN_TIMEOUT=300            # Total shutdown timeout\n\n# Webhook configuration\nN8N_WEBHOOK_URL=https://webhook.yourdomain.com\nWEBHOOK_URL=https://webhook.yourdomain.com\n\n# Security\nN8N_SECURE_COOKIE=true\nN8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true\n```\n\n### Auto-Scaling Configuration\n\nThe dynamic scaler intelligently manages N8N workers based on real-time queue metrics:\n\n```bash\n# Scaling boundaries\nMIN_REPLICAS=1                    # Always keep at least 1 worker\nMAX_REPLICAS=5                    # Maximum concurrent workers\n \n# Scaling triggers\nSCALE_UP_QUEUE_THRESHOLD=5        # Scale up when queue \u003e 5 jobs\nSCALE_DOWN_QUEUE_THRESHOLD=0      # Scale down when queue empty\n\n# Timing controls\nPOLLING_INTERVAL_SECONDS=30       # Check queue every 30 seconds\nCOOLDOWN_PERIOD_SECONDS=120       # Wait 2 minutes between scaling actions\n\n# Queue configuration\nQUEUE_NAME_PREFIX=bull            # BullMQ prefix\nQUEUE_NAME=jobs                   # Queue name\n```\n\n**🧠 Scaling Logic:**\n1. Monitor Redis queue depth every 30 seconds\n2. **Scale Up:** When queue length \u003e threshold AND current replicas \u003c max\n3. **Scale Down:** When queue length ≤ threshold AND current replicas \u003e min  \n4. **Cooldown:** Prevent rapid scaling with configurable wait periods\n5. **Gradual:** Scale one worker at a time for stability\n\n### AI Services Configuration\n\n**🎥 Short Video Maker:**\n```bash\nPEXELS_API_KEY=your-key           # For stock footage\nVIDEO_CACHE_SIZE_IN_BYTES=1GB     # Local cache size\nWHISPER_MODEL=medium.en           # Speech recognition model\n```\n\n**🤖 AI Agents:**\n```bash\nOPENAI_API_KEY=your-key           # For GPT integration\nLOG_LEVEL=debug                   # Logging verbosity\n```\n\n**🎨 Stable Diffusion:**\n```bash\n# CPU processing configuration\nWEBUI_PORT=7860                   # Web interface port\nCPU_THREADS=4                     # CPU thread utilization\n```\n\n## 📊 Monitoring and Observability\n\n### Queue Metrics Monitor\nReal-time Redis queue monitoring with structured logging:\n\n```bash\n# View queue metrics\ndocker compose logs -f n8n-queue-metrics\n\n# Example output:\n# 2024-01-01T12:00:00 [INFO] Queue metrics queue=bull:jobs waiting_jobs=3\n# 2024-01-01T12:00:00 [INFO] Detailed queue stats wait=3 active=2 completed=45 failed=0\n```\n\n**Features:**\n- ✅ Multi-pattern BullMQ support (v3+, v4+, legacy)\n- ✅ Automatic Redis reconnection with exponential backoff\n- ✅ Structured JSON logging with timestamps\n- ✅ Health checks and error recovery\n\n### Dynamic Scaler\nIntelligent auto-scaling with decision logging:\n\n```bash\n# Monitor scaling decisions\ndocker compose logs -f n8n-dynamic-scaler\n\n# Example output:\n# 2024-01-01T12:00:00 [INFO] Current metrics queue_length=7 current_replicas=2\n# 2024-01-01T12:00:00 [INFO] Scaling decision made target_replicas=3 reason=\"Queue length 7 \u003e threshold 5\"\n# 2024-01-01T12:00:00 [INFO] Scaling completed successfully new_replicas=3\n```\n\n**Features:**\n- ✅ Predictive scaling based on queue trends  \n- ✅ Docker Compose integration for seamless scaling\n- ✅ Configuration validation and setup checks\n- ✅ Graceful error handling and recovery\n\n### Health Monitoring\n\n```bash\n# Check all service health\ndocker compose ps --format \"table {{.Name}}\\t{{.Status}}\\t{{.Health}}\"\n\n# View service resource usage\ndocker stats --format \"table {{.Container}}\\t{{.CPUPerc}}\\t{{.MemUsage}}\"\n\n# Monitor specific service logs\ndocker compose logs --tail=100 -f service-name\n```\n\n## 💾 Data Management and Backups\n\n### Automated Backup System\n\n**Daily Backups Include:**\n- PostgreSQL database dumps\n- N8N workflows and credentials  \n- Binary data and configurations\n- Retention: 7 days automatic cleanup\n\n```bash\n# Manual backup\ndocker compose exec n8n-backup sh -c \"\n  BACKUP_DATE=\\$(date +%Y%m%d-%H%M%S)\n  pg_dump -h postgres -U postgres -d n8n_db \u003e /backups/manual-\\$BACKUP_DATE.sql\n  tar -czf /backups/manual-backup-\\$BACKUP_DATE.tar.gz -C /source .\n\"\n\n# View available backups\nls -la ./daily-backups/\n\n# Restore from backup\ndocker compose down\ndocker compose run --rm postgres psql -h postgres -U postgres -d n8n_db \u003c backup.sql\ntar -xzf backup.tar.gz -C ./backups/\ndocker compose --profile core up -d\n```\n\n### Data Persistence\n\n**Volume Mappings:**\n```\n./backups/n8n-data          → N8N application data\n./backups/n8n-workflows     → Workflow definitions\n./backups/n8n-credentials   → Encrypted credentials\n./backups/n8n-binary        → Binary data storage\n./postgres-data/pgdata      → Database files\n./videos                    → Generated video content\n./data/minio                → Object storage\n```\n\n**🔗 Symbolic Links for Data Continuity:**\nThe system uses symbolic links to maintain data continuity when moving between directories, ensuring your workflows and data remain intact during deployments.\n\n## 🛠️ Advanced Usage\n\n### Custom N8N Nodes\n\n```bash\n# Add custom nodes\nmkdir -p ./services/custom-nodes\n# Place your node packages here - they'll be auto-mounted\n```\n\n### Adding New AI Services\n\n1. **Define in docker-compose.yml:**\n```yaml\nmy-ai-service:\n  image: my-ai-service:latest  \n  container_name: n8n-my-ai-service\n  restart: unless-stopped\n  profiles:\n    - core\n  networks:\n    - n8n-network\n  environment:\n    - API_KEY=${MY_AI_API_KEY}\n  labels:\n    - \"traefik.enable=true\"\n    - \"traefik.http.routers.my-ai.rule=Host(`${MY_AI_HOST}`)\"\n    - \"traefik.http.routers.my-ai.entrypoints=web\"\n```\n\n2. **Add to .env:**\n```bash\nMY_AI_HOST=my-ai.yourdomain.com\nMY_AI_API_KEY=your-api-key\n```\n\n3. **Update .env.example** with placeholders\n\n### Performance Optimization\n\n**High-Volume Workflows:**\n```bash\n# Increase worker capacity\nMAX_REPLICAS=10\nN8N_CONCURRENCY_PRODUCTION_LIMIT=20\n\n# Database optimization\nPOSTGRES_SHARED_BUFFERS=256MB\nPOSTGRES_MAX_CONNECTIONS=200\n\n# Redis optimization  \nREDIS_MAXMEMORY=2gb\nREDIS_MAXMEMORY_POLICY=allkeys-lru\n```\n\n**CPU Processing:**\n```bash\n# Verify CPU resources\nnproc  # Check available CPU cores\nfree -h  # Check available memory\n```\n\n## 🔍 Troubleshooting Guide\n\n### Common Issues and Solutions\n\n**❌ Services won't start:**\n```bash\n# Check configuration syntax\ndocker compose config\n\n# Validate environment variables\ndocker compose config | grep -A5 -B5 ERROR\n\n# Check resource availability\ndocker system df\nfree -h\n```\n\n**❌ N8N can't connect to database:**\n```bash\n# Verify PostgreSQL health\ndocker compose exec postgres pg_isready -U postgres\n\n# Check credentials consistency\ngrep -E \"POSTGRES_(USER|PASSWORD|DB)\" .env\n\n# View database logs\ndocker compose logs postgres\n```\n\n**❌ Auto-scaling not working:**\n```bash\n# Check queue metrics service\ndocker compose logs n8n-queue-metrics\n\n# Verify scaler decisions\ndocker compose logs n8n-dynamic-scaler  \n\n# Test Docker socket access\ndocker compose exec n8n-dynamic-scaler docker ps\n\n# Manual queue inspection\ndocker compose exec redis redis-cli LLEN bull:jobs:wait\n```\n\n**❌ AI services failing:**\n```bash\n# Check CPU and memory resources\nnproc\nfree -h\n\n# Memory issues\ndmesg | grep -i \"killed process\"\ndocker stats\n\n# Check CPU usage\ntop -p $(docker inspect --format='{{.State.Pid}}' container-name)\n```\n\n**❌ Cloudflare tunnel issues:**\n```bash\n# Verify tunnel token\ndocker compose logs n8n-cloudflared\n\n# Test connectivity\ncurl -H \"CF-Connecting-IP: 1.2.3.4\" http://localhost/health\n```\n\n### Log Analysis\n\n**View comprehensive logs:**\n```bash\n# All services\ndocker compose logs --tail=100 -f\n\n# Filter by profile\ndocker compose logs -f $(docker compose ps --services --filter profile=core)\n\n# Error hunting\ndocker compose logs 2\u003e\u00261 | grep -i -C3 error\n\n# Performance monitoring\ndocker compose logs | grep -E \"(memory|cpu|performance)\"\n```\n\n### Emergency Recovery\n\n**Complete system reset:**\n```bash\n# Stop all services\ndocker compose down\n\n# Clean containers and volumes (⚠️ DATA LOSS)\ndocker compose down -v\ndocker system prune -af\n\n# Restore from backup\ntar -xzf latest-backup.tar.gz -C ./\ndocker compose --profile core up -d\n```\n\n**Partial service restart:**\n```bash\n# Restart specific services\ndocker compose restart n8n-main n8n-worker\n\n# Rebuild and restart\ndocker compose up -d --force-recreate n8n-main\n```\n\n## 📚 Advanced Workflows\n\n### Webhook URL Configuration\n\n**Format:** `https://webhook.yourdomain.com/webhook/{workflow-id}`\n\n**Example N8N Workflow:**\n1. **Webhook Trigger** → Set to production URL\n2. **Queue Job** → Process with available workers  \n3. **AI Service Call** → Use internal service URLs\n4. **Response** → Return via webhook\n\n### Monitoring Dashboards\n\nSet up comprehensive monitoring:\n```bash\n# Add Prometheus + Grafana (optional)\n# Monitor queue metrics, system resources, and AI service performance\n# Custom dashboards for workflow success rates\n```\n\n## 🤝 Contributing\n\nWe welcome contributions! Here's how to get started:\n\n1. **Fork the repository**\n2. **Create a feature branch:** `git checkout -b feature/amazing-feature`\n3. **Make your changes** with proper documentation\n4. **Test thoroughly:** `docker compose config \u0026\u0026 docker compose --profile core up -d`\n5. **Submit a pull request**\n\n**Development Guidelines:**\n- Follow existing code structure and naming conventions\n- Update documentation for any new features\n- Add appropriate health checks for new services\n- Test with multiple profiles and configurations\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## 🙏 Acknowledgments\n\n- **N8N Community** - For the amazing workflow automation platform\n- **Docker Team** - For containerization technology\n- **AI/ML Communities** - For open-source models and tools\n- **Contributors** - For continuous improvements and bug fixes\n\n---\n\n**📊 Platform Status:** Production Ready ✅  \n**🔄 Auto-scaling:** Active ✅  \n**🤖 AI Services:** Integrated ✅  \n**📚 Documentation:** Comprehensive ✅\n\n---\n\n*Generated with [Claude Code](https://claude.ai/code)*\n\n*Co-Authored-By: Claude \u003cnoreply@anthropic.com\u003e*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faskb%2Fai-n8domata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faskb%2Fai-n8domata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faskb%2Fai-n8domata/lists"}