{"id":29959142,"url":"https://github.com/sin4ch/qgjob","last_synced_at":"2025-10-13T18:33:41.243Z","repository":{"id":306096709,"uuid":"1024981876","full_name":"sin4ch/qgjob","owner":"sin4ch","description":"A production-ready job orchestration system for managing and executing AppWright automated tests on BrowserStack. Provides complete job queuing, processing, and monitoring for web and mobile app testing.","archived":false,"fork":false,"pushed_at":"2025-07-23T22:31:00.000Z","size":74,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-03T21:45:19.294Z","etag":null,"topics":["cli","click-cli","docker","fastapi-sqlalchemy","qa-automation","redis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sin4ch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-23T14:37:19.000Z","updated_at":"2025-07-24T06:15:26.000Z","dependencies_parsed_at":"2025-07-23T17:39:45.347Z","dependency_job_id":null,"html_url":"https://github.com/sin4ch/qgjob","commit_stats":null,"previous_names":["sin4ch/qgjob"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sin4ch/qgjob","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sin4ch%2Fqgjob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sin4ch%2Fqgjob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sin4ch%2Fqgjob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sin4ch%2Fqgjob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sin4ch","download_url":"https://codeload.github.com/sin4ch/qgjob/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sin4ch%2Fqgjob/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279016594,"owners_count":26085852,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","click-cli","docker","fastapi-sqlalchemy","qa-automation","redis"],"created_at":"2025-08-03T21:42:29.582Z","updated_at":"2025-10-13T18:33:41.235Z","avatar_url":"https://github.com/sin4ch.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# QualGent Job Orchestrator\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![CI/CD](https://github.com/sin4ch/qgjob/actions/workflows/ci-cd.yml/badge.svg)](https://github.com/sin4ch/qgjob/actions/workflows/ci-cd.yml)\n\nA production-ready job orchestration system for managing and executing AppWright automated tests on BrowserStack. Provides complete job queuing, processing, and monitoring for web and mobile app testing.\n\n## Features\n\n- **Job Queue Management**: Redis-based queuing with priority support\n- **BrowserStack Integration**: Web and mobile testing on real devices\n- **REST API \u0026 CLI**: Complete interfaces for job management\n- **Worker Processes**: Background test execution with retry logic\n- **Database Persistence**: PostgreSQL job tracking and results\n- **Production Ready**: Comprehensive logging, monitoring, and CI/CD integration\n\n## Architecture\n\n```\nCLI/API → FastAPI Server → Redis Queue → Workers → BrowserStack\n                ↓\n          PostgreSQL Database\n```\n\n## Quick Start\n\n### How It Works\n\n1. **Deploy Apps**: Place mobile apps in `/tmp/apps/{version}.apk` or configure web URLs in tests\n2. **Submit Jobs**: Use CLI or API to queue AppWright tests for execution\n3. **Automatic Execution**: Workers run tests on BrowserStack with real devices/browsers\n4. **Monitor Results**: Get execution videos, pass/fail status, and detailed logs\n\n### Application Types\n\n**Web Applications:**\n```bash\n# Test file specifies URL: await device.goto('https://myapp.com')\nqgjob submit --org-id \"myorg\" --app-version-id \"v1.0\" \\\n  --test \"web-test.spec.js\" --target browserstack\n```\n\n**Mobile Applications:**\n```bash\n# 1. Deploy app with version-based naming\ncp builds/myapp.apk /tmp/apps/v1.0.apk\n\n# 2. Submit test (system finds and uploads app automatically)\nqgjob submit --org-id \"myorg\" --app-version-id \"v1.0\" \\\n  --test \"mobile-test.spec.js\" --target device\n```\n\n## Prerequisites\n\n- Python 3.8+\n- PostgreSQL 12+ (running and accessible)\n- Redis 6+ (running and accessible)\n- BrowserStack Account with Automate plan\n\n**Important:** This is a production application that requires all dependencies. The application will fail to start if any component is missing.\n\n## Installation \u0026 Setup\n\n### 1. Basic Setup\n```bash\ngit clone \u003crepository-url\u003e\ncd qgjob\npython -m venv venv\nsource venv/bin/activate  # On Windows: venv\\Scripts\\activate\npip install -r requirements.txt\n```\n\n### 2. Infrastructure Setup\n\n**PostgreSQL:**\n```bash\n# Install (Ubuntu/Debian)\nsudo apt-get install postgresql postgresql-contrib\n\n# Create database\ncreatedb qgjob\ncreateuser qgjob_user\n```\n\n**Redis:**\n```bash\n# Install and start\nsudo apt-get install redis-server\nredis-server\n```\n\n**BrowserStack:**\n1. Create account at https://www.browserstack.com/\n2. Get credentials from Account → Settings → Automate\n\n### 3. Environment Configuration\n\nCreate `.env` file:\n```env\n# Database (Required)\nDATABASE_URL=postgresql://qgjob_user:secure_password@localhost:5432/qgjob\n\n# Redis (Required)\nREDIS_URL=redis://localhost:6379\n\n# BrowserStack Credentials (Required)\nBROWSERSTACK_USERNAME=your_actual_username\nBROWSERSTACK_ACCESS_KEY=your_actual_access_key\n\n# Application Settings\nAPP_STORAGE_DIR=/tmp/apps\nLOG_LEVEL=INFO\n```\n\n### 4. Initialize \u0026 Run\n\n```bash\n# Initialize database\npython -c \"from src.qgjob.database import create_tables; create_tables()\"\n\n# Start services (both required)\npython -m src.qgjob.main \u0026      # API Server\npython -m src.qgjob.worker \u0026    # Worker Process\n```\n\n## Mobile App Testing\n\n### App Deployment Model\n\nMobile apps use a **pre-deployment model**: apps must be placed in the storage directory before submitting tests.\n\n**App Storage Convention:**\n```bash\n# Naming: APP_STORAGE_DIR/{app_version_id}.apk\n/tmp/apps/v1.2.3.apk          # For --app-version-id \"v1.2.3\"\n/tmp/apps/release-2024.apk    # For --app-version-id \"release-2024\"\n```\n\n**Workflow:**\n```bash\n# 1. Deploy app with version-based naming\nmkdir -p /tmp/apps\ncp builds/myapp-v1.2.3.apk /tmp/apps/v1.2.3.apk\n\n# 2. Submit test (system finds and uploads app automatically)\nqgjob submit --org-id \"myorg\" --app-version-id \"v1.2.3\" \\\n  --test \"mobile-test.spec.js\" --target device\n\n# 3. System automatically:\n# - Finds /tmp/apps/v1.2.3.apk\n# - Uploads to BrowserStack (cached for future tests)\n# - Creates mobile session with app installed\n# - Runs AppWright test on real device\n```\n\n**Benefits:** Performance (cached uploads), CI/CD friendly, secure, clear version management.\n\n## CI/CD Integration\n\nThe project includes comprehensive GitHub Actions workflow with automated testing.\n\n**Setup:**\n1. Configure GitHub Secrets:\n   ```\n   BROWSERSTACK_USERNAME: your_browserstack_username\n   BROWSERSTACK_ACCESS_KEY: your_browserstack_access_key\n   ```\n\n2. Pipeline runs on: push to `main`/`develop`, pull requests, manual dispatch\n\n**CI/CD Example:**\n```yaml\n- name: Run AppWright Tests\n  run: |\n    qgjob submit --org-id \"$ORG\" --app-version-id \"$VERSION\" \\\n      --test \"regression.spec.js\" --priority 1 --target browserstack \u003e job_output.txt\n\n    JOB_ID=$(grep \"Job ID:\" job_output.txt | awk '{print $3}')\n    qgjob wait --job-id \"$JOB_ID\" --timeout 300\n    qgjob status --job-id \"$JOB_ID\" --verbose\n```\n\n## Usage\n\n### CLI Commands\n\n```bash\n# Submit jobs\nqgjob submit --org-id \"my-org\" --app-version-id \"v1.0.0\" --test \"test.spec.js\" --target browserstack\nqgjob submit --org-id \"my-org\" --app-version-id \"v1.0.0\" --test \"mobile.spec.js\" --target device\n\n# Monitor jobs\nqgjob status --job-id \"job-uuid\" [--verbose]\nqgjob list --org-id \"my-org\" [--status failed]\nqgjob wait --job-id \"job-uuid\" --timeout 300\n\n# Manage jobs\nqgjob retry --job-id \"job-uuid\"\nqgjob cancel --job-id \"job-uuid\"\nqgjob metrics\n```\n\n### REST API\n\n```bash\n# Submit job\ncurl -X POST http://localhost:8000/jobs \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"org_id\": \"my-org\", \"app_version_id\": \"v1.0.0\", \"test_path\": \"test.spec.js\", \"priority\": 1, \"target\": \"browserstack\"}'\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n### Supported Targets\n\n- **browserstack**: Web testing on BrowserStack\n- **device**: Mobile app testing on real devices via BrowserStack\n- **emulator**: Mobile app testing on emulators via BrowserStack\n\n## Example AppWright Test\n\nSample test file (`tests/wikipedia.spec.js`):\n\n```javascript\nimport { test, expect } from \"appwright\";\n\ntest(\"Open Playwright on Wikipedia and verify Microsoft is visible\", async ({ device }) =\u003e {\n  await device.getByText(\"Skip\").tap();\n\n  const searchInput = device.getByText(\"Search Wikipedia\", { exact: true });\n  await searchInput.tap();\n  await searchInput.fill(\"playwright\");\n\n  await device.getByText(\"Playwright (software)\").tap();\n  await expect(device.getByText(\"Microsoft\")).toBeVisible();\n});\n```\n\n**Key Features:** AppWright framework, device context for mobile-first testing, touch interactions (`.tap()`, `.fill()`), content-based element selection, visual assertions.\n\n**Execution Results:** Creates BrowserStack session, records video, provides detailed results, integrates with CI/CD.\n\n## Production Deployment\n\n### Docker Deployment\n\n```yaml\nversion: '3.8'\nservices:\n  postgres:\n    image: postgres:13\n    environment:\n      POSTGRES_DB: qgjob\n      POSTGRES_USER: qgjob_user\n      POSTGRES_PASSWORD: ${DB_PASSWORD}\n    volumes:\n      - postgres_data:/var/lib/postgresql/data\n\n  redis:\n    image: redis:alpine\n    command: redis-server --appendonly yes\n    volumes:\n      - redis_data:/data\n\n  api:\n    build: .\n    ports:\n      - \"8000:8000\"\n    environment:\n      - DATABASE_URL=postgresql://qgjob_user:${DB_PASSWORD}@postgres:5432/qgjob\n      - REDIS_URL=redis://redis:6379\n      - BROWSERSTACK_USERNAME=${BS_USERNAME}\n      - BROWSERSTACK_ACCESS_KEY=${BS_ACCESS_KEY}\n    depends_on:\n      - postgres\n      - redis\n\n  worker:\n    build: .\n    command: python -m src.qgjob.worker\n    environment:\n      - DATABASE_URL=postgresql://qgjob_user:${DB_PASSWORD}@postgres:5432/qgjob\n      - REDIS_URL=redis://redis:6379\n      - BROWSERSTACK_USERNAME=${BS_USERNAME}\n      - BROWSERSTACK_ACCESS_KEY=${BS_ACCESS_KEY}\n    depends_on:\n      - postgres\n      - redis\n\nvolumes:\n  postgres_data:\n  redis_data:\n```\n\n### Production Considerations\n\n- **Scalability**: Multiple workers, load balancing, database optimization\n- **Security**: Secure credential management, network security, API authentication\n- **Monitoring**: Application metrics, infrastructure monitoring, alerting\n- **Backup**: Database backups, configuration version control, disaster recovery\n\n## Troubleshooting\n\n### Common Issues\n\n**Database Connection Failed:**\n```\nRuntimeError: Database connection failed: connection to server at \"localhost\" (127.0.0.1), port 5432 failed\n```\n- Verify PostgreSQL is running: `pg_isready`\n- Check DATABASE_URL format: `postgresql://user:password@host:port/database`\n\n**Redis Connection Failed:**\n```\nRuntimeError: Redis connection failed: Error 111 connecting to localhost:6379. Connection refused.\n```\n- Verify Redis is running: `redis-cli ping`\n- Check REDIS_URL format: `redis://localhost:6379`\n\n**BrowserStack Credentials Required:**\n```\nRuntimeError: BrowserStack credentials required: BrowserStack credentials not found\n```\n- Set BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY\n- Verify credentials are not placeholder values\n\n**App File Not Found:**\n```\nError: App file not found for v1.2.3 at path: /tmp/apps/v1.2.3.apk\n```\n- Check naming: `--app-version-id \"v1.2.3\"` requires `/tmp/apps/v1.2.3.apk`\n- Verify file exists: `ls -la /tmp/apps/`\n- Check permissions: `chmod 644 /tmp/apps/*.apk`\n\n**Worker Not Processing Jobs:**\n- Check worker logs: `qgjob-worker.log`\n- Verify environment variables are set\n- Ensure BrowserStack account has available sessions\n\n### Monitoring\n\n**Log Files:** `qgjob.log` (API), `qgjob-worker.log` (Worker)\n**Health Checks:** `GET /health`, `GET /metrics`\n\n## API Reference\n\n### Job Object\n```json\n{\n  \"id\": \"uuid\",\n  \"org_id\": \"string\",\n  \"app_version_id\": \"string\", \n  \"test_path\": \"string\",\n  \"priority\": 1,\n  \"target\": \"browserstack|device|emulator\",\n  \"status\": \"queued|processing|completed|failed\",\n  \"created_at\": \"2023-01-01T00:00:00Z\",\n  \"updated_at\": \"2023-01-01T00:00:00Z\",\n  \"result\": \"string\",\n  \"error_message\": \"string\"\n}\n```\n\n### Endpoints\n- `POST /jobs` - Submit new job\n- `GET /jobs/{job_id}` - Get job details\n- `GET /jobs` - List jobs with filters (supports org_id, status, app_version_id, limit, offset)\n- `DELETE /jobs/{job_id}` - Cancel job\n- `GET /jobs/{job_id}/retry` - Retry failed job\n- `GET /health` - Health check\n- `GET /metrics` - System metrics\n\n## License\n\nThis project is licensed under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsin4ch%2Fqgjob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsin4ch%2Fqgjob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsin4ch%2Fqgjob/lists"}