{"id":35416056,"url":"https://github.com/full-chaos/dev-health-ops","last_synced_at":"2026-01-31T03:00:37.425Z","repository":{"id":289881175,"uuid":"972703482","full_name":"full-chaos/dev-health-ops","owner":"full-chaos","description":"Developer Teams and Individual SPACE-like metrics","archived":false,"fork":false,"pushed_at":"2026-01-28T23:45:45.000Z","size":3573,"stargazers_count":1,"open_issues_count":12,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-29T07:44:35.320Z","etag":null,"topics":["developer-experience","developer-heal","developer-productivity","development-health","mergestat","space-metrics"],"latest_commit_sha":null,"homepage":"https://full-chaos.github.io/dev-health-ops/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/full-chaos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-04-25T14:13:51.000Z","updated_at":"2026-01-28T20:24:05.000Z","dependencies_parsed_at":"2026-01-28T19:00:46.482Z","dependency_job_id":null,"html_url":"https://github.com/full-chaos/dev-health-ops","commit_stats":null,"previous_names":["chrisgeo/mergestat-syncs","chrisgeo/dev-health-ops","full-chaos/dev-health-ops"],"tags_count":22,"template":false,"template_full_name":null,"purl":"pkg:github/full-chaos/dev-health-ops","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/full-chaos%2Fdev-health-ops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/full-chaos%2Fdev-health-ops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/full-chaos%2Fdev-health-ops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/full-chaos%2Fdev-health-ops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/full-chaos","download_url":"https://codeload.github.com/full-chaos/dev-health-ops/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/full-chaos%2Fdev-health-ops/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28927762,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-31T02:59:34.861Z","status":"ssl_error","status_checked_at":"2026-01-31T02:59:05.369Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["developer-experience","developer-heal","developer-productivity","development-health","mergestat","space-metrics"],"created_at":"2026-01-02T15:24:00.160Z","updated_at":"2026-01-31T03:00:37.314Z","avatar_url":"https://github.com/full-chaos.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dev-health-ops\n\nDevelopment team and developers' operational help should be available for all.\n\nThis project's goal is to provide tools and quick-win implementations by integrating with a majority of popular tooling.\n\n## Why this exists\n\nDeveloper health tooling drifted into expensive, opaque “scoring” systems that are easy to misuse. This project is intentionally different.\n\n### Principles\n\n- **Accessibility over extraction**: derived from data teams already own; should be cheap to run and never gated behind per-seat pricing.\n- **Learning, not judgment**: metrics are **signals** about system behavior (WIP, churn, cycle time, blocked work), not performance rankings.\n- **Trends \u003e absolutes**: compare change over time and distributions, not “who’s best”.\n- **Inspectable by default**: open schemas, explicit definitions, and reproducible computation.\n\n### Non-goals\n\n- Individual leaderboards and “scores”\n- HR/performance-management tooling\n- Executive theater dashboards that hide context\n\n## Installation\n\nIf you are not developing on this project and just want to use the tools, you can install the package directly:\n\n```bash\npip install dev-health-ops\n```\n\nThis provides the `dev-hops` command in your terminal.\n\n```bash\ndev-hops --help\n```\n\n_Note: In the documentation below, you can replace `dev-hops` with `dev-hops` if you have installed the package._\n\n## Private Repository Support ✅\n\n**Both GitHub and GitLab connectors fully support private repositories!** When provided with tokens that have appropriate permissions, you can access and sync data from private repositories just as easily as public ones.\n\n- **GitHub**: Requires `repo` scope on your personal access token\n- **GitLab**: Requires `read_api` and `read_repository` scopes on your private token\n\nSee [`PRIVATE_REPO_TESTING.md`](./PRIVATE_REPO_TESTING.md) for detailed instructions on setting up and testing private repository access, or [`VERIFICATION_SUMMARY.md`](./VERIFICATION_SUMMARY.md) for a comprehensive overview.\n\n## Batch Repository Processing ✅\n\nThe GitHub connector supports batch processing of repositories with:\n\n- **Pattern matching** - Filter repositories using fnmatch-style patterns (e.g., `chrisgeo/*`, `*/api-*`)\n- **Configurable batch size** - Process repositories in batches to manage memory and API usage\n- **Rate limiting** - Delay between batches plus shared backoff across workers (avoids stampedes; honors server reset/`Retry-After` when available)\n- **Async processing** - Process multiple repositories concurrently for better performance\n- **Callbacks** - Get notified as each repository is processed\n\n### Example Usage\n\n```python\nfrom connectors import GitHubConnector\n\nconnector = GitHubConnector(token=\"your_token\")\n\n# List repos with pattern matching (integrated into list_repositories)\nrepos = connector.list_repositories(\n    org_name=\"myorg\",\n    pattern=\"myorg/api-*\",      # Filter repos matching this pattern\n    max_repos=50,\n)\n\n# Get all repos matching a pattern with stats\nresults = connector.get_repos_with_stats(\n    org_name=\"myorg\",\n    pattern=\"myorg/api-*\",      # Filter repos matching this pattern\n    batch_size=10,              # Process 10 repos at a time\n    max_concurrent=4,           # Use 4 concurrent workers\n    rate_limit_delay=1.0,       # Wait 1 second between batches\n    max_commits_per_repo=100,   # Limit commits analyzed per repo\n    max_repos=50,               # Maximum repos to process\n)\n\nfor result in results:\n    if result.success:\n        print(f\"{result.repository.full_name}: {result.stats.total_commits} commits\")\n```\n\n### Async Processing\n\nFor even better performance, use the async version:\n\n```python\nimport asyncio\nfrom connectors import GitHubConnector\n\nasync def main():\n    connector = GitHubConnector(token=\"your_token\")\n\n    results = await connector.get_repos_with_stats_async(\n        org_name=\"myorg\",\n        pattern=\"myorg/*\",\n        batch_size=10,\n        max_concurrent=4,\n    )\n\n    for result in results:\n        if result.success:\n            print(f\"{result.repository.full_name}: {result.stats.total_commits} commits\")\n\nasyncio.run(main())\n```\n\n### Pattern Matching Examples\n\n| Pattern       | Matches                                      |\n| ------------- | -------------------------------------------- |\n| `chrisgeo/m*` | `chrisgeo/dev-health-ops`, `chrisgeo/my-app` |\n| `*/api-*`     | `anyorg/api-service`, `myuser/api-gateway`   |\n| `org/repo`    | Exactly `org/repo`                           |\n| `chrisgeo/*`  | All repositories owned by `chrisgeo`         |\n| `*sync*`      | Any repository with `sync` in the name       |\n\n## Developer Health Metrics (Work + Git) + Grafana ✅\n\n\u003e **DEPRECATION NOTICE**: The Grafana panels and plugin integration have been moved to a separate repository (`dev-health-panels`) and are being superseded by the full-stack web application [`dev-health-web`](../dev-health-web).\n\nThis repo can compute daily “developer health” metrics and provision Grafana dashboards on top of:\n\n- **Git + PR/MR facts** (from GitHub/GitLab/local syncs)\n- **Work tracking items** (Jira issues, GitHub issues/Projects, GitLab issues)\n\nJira is **not** a replacement for pull request data — it’s used to track associated project work (throughput, WIP, work-item cycle/lead times). PR metrics still come from the Git provider data (e.g., GitHub PRs / GitLab MRs) synced by the CLI (`dev-hops sync \u003ctarget\u003e --provider ...`).\n\n**Docs**\n\n- Metrics definitions + tables: `docs/metrics.md`\n- Implementation plans, metrics inventory, requirements/roadmap: `docs/project.md`, `docs/metrics-inventory.md`, `docs/roadmap.md`\n- Task tracker configuration (Jira/GitHub/GitLab, status mapping, teams): `docs/task_trackers.md`\n- Grafana dashboards + provisioning: `docs/grafana.md`\n\n### Quickstart (ClickHouse + Grafana)\n\n1. Start ClickHouse + Grafana:\n\n```bash\ndocker compose -f compose.yml up -d\n```\n\n1. Sync Git data into ClickHouse (choose one):\n\n```bash\n# Local repo (commits + stats)\ndev-hops sync git --provider local --db \"clickhouse://localhost:8123/default\" --repo-path .\n\n# GitHub repo (commits + stats)\ndev-hops sync git --provider github --db \"clickhouse://localhost:8123/default\" --owner \u003cowner\u003e --repo \u003crepo\u003e\n\n# GitLab project (commits + stats)\ndev-hops sync git --provider gitlab --db \"clickhouse://localhost:8123/default\" --project-id \u003cid\u003e\n```\n\n1. Compute derived metrics (Git + Work Items):\n\n```bash\n# (Optional) Sync work items from provider APIs (recommended)\ndev-hops sync work-items --provider all --date 2025-02-01 --backfill 30 --db \"clickhouse://localhost:8123/default\"\n\n# One day (derived Git metrics; enriches IC metrics from already-synced work items when available)\ndev-hops metrics daily --date 2025-02-01 --db \"clickhouse://localhost:8123/default\"\n\n# Backfill last 30 days ending at date\ndev-hops metrics daily --date 2025-02-01 --backfill 30 --db \"clickhouse://localhost:8123/default\"\n```\n\n1. Open Grafana:\n\n- \u003chttp://localhost:3000\u003e (default `admin` / `admin`)\n- Dashboards are provisioned under the “Developer Health” folder.\n\n### API (FastAPI)\n\nRun the Developer Health Ops API for the web app:\n\n```bash\ndev-hops api --db \"clickhouse://localhost:8123/default\" --reload\n```\n\nOpenAPI docs are available at \u003chttp://localhost:8000/docs\u003e.\n\n## Container images\n\nThis repo ships two reusable images built from `docker/Dockerfile`, which provides a multi-stage build (`base`, `api`, `runner`). The images cover demo runners and REST APIs:\n\n1. `dev-hops-api` runs `dev-hops api` and exposes `port 8000`.\n2. `dev-hops-runner` uses `dev-hops` as the entrypoint so you can invoke `sync`, `fixtures`, `metrics`, etc., through a container.\n\n### Building the images\n\nUse `scripts/build-images.sh` to build both images (it sets `SETUPTOOLS_SCM_PRETEND_VERSION` so `setuptools_scm` doesn't require `.git`). The base stage installs the package and then drops the source tree so the runtime image contains only the installed wheel. Override the defaults with:\n\n- Note: the API runtime loads packaged SQL files from `dev_health_ops/api/sql`. If you extend the build, make sure those SQL files are included in the wheel (the Docker build will fail fast if they're missing).\n\n- `IMAGE_REGISTRY` (defaults to `ghcr.io/chrisgeo/dev-health-ops`)\n- `VERSION` (tags the images; default `latest`)\n- `SETUPTOOLS_SCM_PRETEND_VERSION` (needed when building from a released archive without Git metadata)\n\n```bash\ncd /path/to/dev-health-ops\nIMAGE_REGISTRY=ghcr.io/myorg/dev-health-ops \\\nVERSION=$(git describe --tags --abbrev=0 2\u003e/dev/null || echo latest) \\\n./scripts/build-images.sh\n```\n\nAny extra arguments (`--no-cache`, `--pull`, etc.) are forwarded to both `docker build` invocations.\n\n### Running the API container\n\nExpose port `8000` and point the container at your ClickHouse backend:\n\n```bash\ndocker run --rm -p 8000:8000 \\\n  -e DATABASE_URI=clickhouse://ch:ch@clickhouse:8123/stats \\\n  dev-hops-api:latest\n```\n\nAdd flags after the image name (e.g., `--reload`) to modify the default `api` invocation (`--host 0.0.0.0 --port 8000` is already applied).\n\n### Using the runner container\n\nMount repositories or fixtures directories and run any `dev-hops` command you need. Share a Docker network with ClickHouse (Compose uses `dev-health-ops_default` by default):\n\n```bash\ndocker run --rm -it \\\n  --network dev-health-ops_default \\\n  -v \"$(pwd)\":/app \\\n  -w /app \\\n  -e DATABASE_URI=clickhouse://ch:ch@clickhouse:8123/stats \\\n  dev-hops-runner:latest \\\n  fixtures generate --db clickhouse://ch:ch@clickhouse:8123/stats --days 14\n```\n\nReplace the final arguments with any `dev-hops` subcommand you need (`sync`, `metrics daily`, etc.). The entrypoint handles argument parsing so you can run the same image in CI, demos, or release flows.\n\n### Automated builds\n\n`docker-images.yml` runs on GitHub when a `v*` tag is pushed or a release is published. It logs into `ghcr.io`, builds both `runner`/`api` targets from `docker/Dockerfile`, and pushes `:latest` plus the tag pulled from the workflow context (`${{ github.ref_name }}` or the release tag). Make sure `GITHUB_TOKEN` has `packages: write` (default) so the workflow can publish into `ghcr.io/chrisgeo/dev-health-ops`.\n\n### “Download” work tracking data (Jira/GitHub/GitLab)\n\nWork items are fetched from provider APIs via a dedicated sync command. This is separate from PR ingestion:\n\n- Configure credentials + mapping (see `docs/task_trackers.md`)\n- Sync work items: `dev-hops sync work-items --provider jira|github|gitlab|all ...` (use `-s` to filter repos; `--auth` for GitHub/GitLab token override)\n- `metrics daily` does not need `--provider` unless you want backward-compatible \"sync-then-compute\" behavior in one step.\n\n`src/dev_health_ops/cli.py` automatically loads a local `.env` file from the repo root (without overriding already-set environment variables). Disable with `DISABLE_DOTENV=1`.\n\n### Sync Teams\n\nYou can sync team definitions into the database from multiple sources. This allows dashboards to group data by teams.\n\n```bash\n# Sync from a local YAML config (default)\ndev-hops sync teams --db \"sqlite+aiosqlite:///stats.db\" --path config/teams.yaml\n\n# Sync from Jira Projects (uses JIRA_* env vars)\ndev-hops sync teams --db \"sqlite+aiosqlite:///stats.db\" --provider jira\n\n# Generate synthetic teams for testing\ndev-hops sync teams --db \"sqlite+aiosqlite:///stats.db\" --provider synthetic\n```\n\n## Database Configuration\n\nThis project supports PostgreSQL, MongoDB, SQLite, and ClickHouse as storage backends.\n\n### Environment Variables\n\n- **`DATABASE_URI`** (optional): Default DB URI for `dev-hops metrics daily --db ...` and for Alembic migrations. Also accepts `DATABASE_URL` as an alias.\n- **`SECONDARY_DATABASE_URI`** (optional): Secondary database URI for `--sink both` mode (writes to both primary and secondary databases).\n- **`DB_ECHO`** (optional): Enable SQL query logging for PostgreSQL and SQLite. Set to `true`, `1`, or `yes` (case-insensitive) to enable. Any other value (including `false`, `0`, `no`, or unset) disables it. Default: `false`. Note: Enabling this in production can expose sensitive data and impact performance.\n- **`REPO_UUID`** (optional): UUID for the repository. If not provided, a deterministic UUID will be derived from the git repository's remote URL (or repository path if no remote exists). This ensures the same repository always gets the same UUID across runs.\n- **`MAX_WORKERS`** (optional): Number of parallel workers for processing git blame data. Higher values can speed up processing but use more CPU and memory. Default: `4`\n- **`LOG_LEVEL`** (optional): Logging level (e.g. `INFO`, `DEBUG`). Default: `INFO`\n- **`DISABLE_DOTENV`** (optional): Set to `1` to disable `.env` loading from the repo root.\n- **`GITHUB_TOKEN`** (optional): Default GitHub token when `--auth` is not provided.\n- **`GITLAB_TOKEN`** (optional): Default GitLab token when `--auth` is not provided.\n- **`GITLAB_URL`** (optional): Default GitLab base URL when `--gitlab-url` is not provided (default: `https://gitlab.com`).\n\n### Command-Line Arguments\n\nYou can also configure the database using command-line arguments, which will override environment variables:\n\n#### Core Arguments\n\n- **`--db`**: Database connection string (required for `sync`; optional for `metrics daily` if `DATABASE_URI` is set)\n- **`--db-type`**: Database backend override (`postgres`, `mongo`, `sqlite`, or `clickhouse`) - optional if URL scheme is clear\n- **`--provider`**: Source provider for sync targets (`local`, `github`, `gitlab`, `synthetic`)\n- **`--auth`**: Authentication token (GitHub/GitLab)\n- **`--repo-path`**: Path to the git repository (for `--provider local`)\n- **`--since`**: Lower-bound date/time filter for sync targets. Uses ISO formats (e.g., `2024-01-01` or `2024-01-01T00:00:00`).\n\n#### Connector-Specific Arguments\n\n- **`--owner`**: GitHub repository owner/organization\n- **`--repo`**: GitHub repository name\n- **`--gitlab-url`**: GitLab instance URL (default: \u003chttps://gitlab.com\u003e)\n- **`--project-id`**: GitLab project ID (numeric)\n\n#### Batch Processing Options\n\nThese unified options work with both GitHub and GitLab connectors:\n\n- **`-s, --search`**: fnmatch-style pattern to filter repositories/projects (e.g., `owner/repo*`, `group/p*`)\n- **`--batch-size`**: Number of repositories/projects to process in each batch (default: 10)\n- **`--group`**: Organization/group name to fetch repositories/projects from\n- **`--max-concurrent`**: Maximum concurrent workers for batch processing (default: 4)\n- **`--rate-limit-delay`**: Delay in seconds between batches for rate limiting (default: 1.0)\n- **`--max-commits-per-repo`**: Maximum commits to analyze per repository/project\n- **`--max-repos`**: Maximum number of repositories/projects to process\n- **`--use-async`**: Use async processing for better performance\n\nExample usage:\n\n```bash\n# Using PostgreSQL (auto-detected from URL)\ndev-hops sync git --provider local --db \"postgresql+asyncpg://user:pass@localhost:5432/stats\"\n\n# Using MongoDB (auto-detected from URL)\ndev-hops sync git --provider local --db \"mongodb://localhost:27017\"\n\n# Local repo filtered to recent activity\ndev-hops sync git --provider local \\\n  --db \"sqlite+aiosqlite:///stats.db\" \\\n  --repo-path /path/to/repo \\\n  --since 2024-01-01\n# Commits and stats are limited to changes on/after this date.\n\n# Using SQLite (file-based, auto-detected)\ndev-hops sync git --provider local --db \"sqlite+aiosqlite:///stats.db\"\n\n# Using SQLite (in-memory)\ndev-hops sync git --provider local --db \"sqlite+aiosqlite:///:memory:\"\n\n# GitHub repository with unified auth\ndev-hops sync git --provider github \\\n  --db \"postgresql+asyncpg://user:pass@localhost:5432/stats\" \\\n  --auth \"$GITHUB_TOKEN\" \\\n  --owner torvalds \\\n  --repo linux\n\n# GitLab project with unified auth\ndev-hops sync git --provider gitlab \\\n  --db \"mongodb://localhost:27017\" \\\n  --auth \"$GITLAB_TOKEN\" \\\n  --project-id 278964\n\n# Batch process repositories matching a pattern (GitHub)\ndev-hops sync git --provider github \\\n  --db \"sqlite+aiosqlite:///stats.db\" \\\n  --auth \"$GITHUB_TOKEN\" \\\n  -s \"chrisgeo/dev-health-*\" \\\n  --group \"chrisgeo\" \\\n  --batch-size 5 \\\n  --max-concurrent 2 \\\n  --max-repos 10 \\\n  --use-async\n\n# Batch process projects matching a pattern (GitLab)\ndev-hops sync git --provider gitlab \\\n  --db \"sqlite+aiosqlite:///stats.db\" \\\n  --auth \"$GITLAB_TOKEN\" \\\n  --gitlab-url \"https://gitlab.com\" \\\n  --group \"mygroup\" \\\n  -s \"mygroup/api-*\" \\\n  --batch-size 5 \\\n  --max-concurrent 2 \\\n  --max-repos 10 \\\n  --use-async\n```\n\n### MongoDB Connection String Format\n\nMongoDB connection strings follow the standard MongoDB URI format:\n\n- **Basic**: `mongodb://host:port`\n- **With authentication**: `mongodb://username:password@host:port`\n- **With database**: `mongodb://username:password@host:port/database_name`\n- **With options**: `mongodb://host:port/?authSource=admin\u0026retryWrites=true`\n\nNote: Include the database name in the URI (e.g., `mongodb://host:port/stats`).\n\n### SQLite Connection String Format\n\nSQLite connection strings use the following format:\n\n- **File-based**: `sqlite+aiosqlite:///path/to/database.db` (relative path) or `sqlite+aiosqlite:////absolute/path/to/database.db` (absolute path - note the four slashes)\n- **In-memory**: `sqlite+aiosqlite:///:memory:` (data is lost when the process exits)\n\nSQLite is ideal for:\n\n- Local development and testing\n- Single-user scenarios\n- Small to medium-sized repositories\n- Environments where running a database server is not practical\n\nNote: SQLite does not use connection pooling since it is a file-based database.\n\n### Performance Tuning\n\nThe script includes several configuration options to optimize performance:\n\n- **`MAX_WORKERS`**: Controls parallel processing of git blame data. Set this based on your CPU cores (e.g., 2-8). Higher values speed up processing but use more CPU and memory.\n\n- **Connection Pooling**: PostgreSQL automatically uses connection pooling with these defaults:\n  - Pool size: 20 connections\n  - Max overflow: 30 additional connections\n  - Connections are recycled every hour\n\n**Example for large repositories:**\n\n```bash\nexport MAX_WORKERS=8\ndev-hops sync git --provider local --db \"sqlite+aiosqlite:///stats.db\" --repo-path .\n```\n\n**Example for resource-constrained environments:**\n\n```bash\nexport MAX_WORKERS=2\ndev-hops sync git --provider local --db \"sqlite+aiosqlite:///stats.db\" --repo-path .\n```\n\n## Performance Optimizations\n\nThis project includes several key performance optimizations to speed up git data processing:\n\n### 1. **Increased Batch Size** (10x improvement)\n\n- **Batching**: Uses batched inserts to reduce database round-trips\n- **Impact**: Significantly reduces database round-trips, improving insertion speed\n\n### 2. **Parallel Git Blame Processing** (4-8x improvement)\n\n- **Implementation**: Uses asyncio with configurable worker pool\n- **Default**: 4 parallel workers processing files concurrently\n- **Impact**: Multi-core CPU utilization, dramatically faster blame processing\n- **Configuration**: Set `MAX_WORKERS=8` for more powerful machines\n\n### 3. **Database Connection Pooling** (PostgreSQL)\n\n- **Pool size**: 20 connections (up from default 5)\n- **Max overflow**: 30 additional connections (up from default 10)\n- **Impact**: Better handling of concurrent operations, reduced connection overhead\n- **Auto-configured**: No manual setup required\n\n### 4. **Optimized Bulk Operations**\n\n- All database insertions use bulk operations\n- MongoDB operations use `ordered=False` for better performance\n- SQLAlchemy uses `add_all()` for efficient batch inserts\n\n### 5. **Smart File Filtering**\n\n- Skips binary files (images, videos, archives, etc.)\n- Skips files larger than 1MB for content reading\n- Reduces unnecessary I/O and processing time\n\n### Expected Performance Improvements\n\nFor a typical repository with 1000 files and 10,000 commits:\n\n| Operation    | Before      | After         | Improvement      |\n| ------------ | ----------- | ------------- | ---------------- |\n| Git Blame    | 50 min      | 6-12 min      | **4-8x faster**  |\n| Commits      | -           | 1-2 min       | **New feature**  |\n| Commit Stats | -           | 2-4 min       | **New feature**  |\n| Files        | -           | 30-60 sec     | **New feature**  |\n| **Total**    | **50+ min** | **10-20 min** | **~3-5x faster** |\n\n_Actual performance depends on hardware, repository size, and configuration._\n\n### PostgreSQL vs MongoDB vs SQLite: Setup and Migration Considerations\n\n#### Using PostgreSQL\n\n- Requires running database migrations with Alembic before first use\n- Provides strong relational data structure\n- Best for complex queries and joins\n- Example setup:\n\n  ```bash\n  # Start PostgreSQL with Docker Compose\n  docker compose up postgres -d\n\n  # Run migrations (Alembic reads DATABASE_URI)\n  export DATABASE_URI=\"postgresql+asyncpg://postgres:postgres@localhost:5333/postgres\"\n  alembic upgrade head\n\n  # Sync a local repo\n  dev-hops sync git --provider local --db \"$DATABASE_URI\" --repo-path .\n  ```\n\n#### Using MongoDB\n\n- No migrations required - collections are created automatically\n- Schema-less design allows for flexible data structures\n- Best for quick setup and document-based storage\n- Example setup:\n\n  ```bash\n  # Start MongoDB with Docker Compose\n  docker compose up mongo -d\n\n  dev-hops sync git --provider local --db \"mongodb://localhost:27017/stats\" --repo-path .\n  ```\n\n#### Using SQLite\n\n- No migrations required - tables are created automatically using SQLAlchemy\n- Simple file-based or in-memory database\n- No external database server required\n- Best for local development, testing, and single-user scenarios\n- Example setup:\n\n  ```bash\n  dev-hops sync git --provider local --db \"sqlite+aiosqlite:///stats.db\" --repo-path .\n  ```\n\n  Or for an in-memory database (data lost when process exits):\n\n  ```bash\n  dev-hops sync git --provider local --db \"sqlite+aiosqlite:///:memory:\" --repo-path .\n  ```\n\n#### Using ClickHouse\n\n- No migrations required - tables are created automatically using `ReplacingMergeTree`\n- Best for analytics and large datasets\n- Example setup:\n\n  ```bash\n  dev-hops sync git --provider local --db \"clickhouse://default:@localhost:8123/default\" --repo-path .\n  ```\n\n#### Switching Between Databases\n\n- The different backends use different storage mechanisms and are not directly compatible\n- Data is not automatically migrated when switching between PostgreSQL, MongoDB, SQLite, and ClickHouse\n- If you need to switch backends, you'll need to re-run the analysis to populate the new database\n- PostgreSQL and MongoDB can run simultaneously on the same machine using different ports (see `compose.yml`)\n\n### Local Repository Pull Request Handling Warning\n\n**Important:** When processing local repositories, pull request records are inferred from merge commit messages and local refs. These inferences are estimation-based and highly volatile:\n\n- Dates (created_at, merged_at) may be inaccurate due to limited information in local repositories\n- PR states (open/closed/merged) are estimated from commit history\n- Some PRs may be missed entirely if they don't match expected patterns\n- The accuracy depends heavily on repository history and commit message conventions\n\nThis behavior is different from GitHub/GitLab connectors, which provide accurate PR data directly from the provider API.\n\n## Example Running Order\n\n1.  Sync teams\n\ndev-hops sync teams --provider config --path config/teams.yaml --db \"\u003cDB_CONN\u003e\"\n\n# or: dev-hops sync teams --provider jira --db \"\u003cDB_CONN\u003e\"\n\n2. Sync git facts (example: GitHub)\n\ndev-hops sync git --provider github --owner \"\u003cORG\u003e\" --repo \"\u003cREPO\u003e\" --db \"\u003cDB_CONN\u003e\"\ndev-hops sync prs --provider github --owner \"\u003cORG\u003e\" --repo \"\u003cREPO\u003e\" --db \"\u003cDB_CONN\u003e\"\ndev-hops sync blame --provider github --owner \"\u003cORG\u003e\" --repo \"\u003cREPO\u003e\" --db \"\u003cDB_CONN\u003e\"\ndev-hops sync cicd --provider github --owner \"\u003cORG\u003e\" --repo \"\u003cREPO\u003e\" --db \"\u003cDB_CONN\u003e\"\ndev-hops sync deployments --provider github --owner \"\u003cORG\u003e\" --repo \"\u003cREPO\u003e\" --db \"\u003cDB_CONN\u003e\"\ndev-hops sync incidents --provider github --owner \"\u003cORG\u003e\" --repo \"\u003cREPO\u003e\" --db \"\u003cDB_CONN\u003e\"\n\n3. Sync work items (and derived work‑item tables)\n\ndev-hops sync work-items --provider github --db \"\u003cDB_CONN\u003e\" --date YYYY-MM-DD --backfill 30\n\n# use --provider jira|gitlab|all as needed\n\n4. Compute daily metrics (uses stored facts)\n\ndev-hops metrics daily --db \"\u003cDB_CONN\u003e\" --date YYYY-MM-DD --backfill 30\n\n5. Compute complexity snapshots\n\ndev-hops metrics complexity --repo-path /path/to/repo --db \"\u003cDB_CONN\u003e\" --date YYYY-MM-DD --backfill 30\n\n## Sample Dashboards\n\n\u003cimg width=\"1807\" height=\"1080\" alt=\"Advanced Work Tracking Phase 2\" src=\"https://github.com/user-attachments/assets/f635d1ec-2c0f-41c6-b57c-e3687f090007\" /\u003e\n\u003cimg width=\"1807\" height=\"1080\" alt=\"CI CD Pipelines Dashboard\" src=\"https://github.com/user-attachments/assets/ba8a03bc-fa4c-4241-9a3b-ab312c9e9f23\" /\u003e\n\u003cimg width=\"1807\" height=\"1080\" alt=\"Code Hotspots Dashboard\" src=\"https://github.com/user-attachments/assets/3ba27599-ec82-424d-a616-330b946ac37b\" /\u003e\n\u003cimg width=\"1080\" height=\"1441\" alt=\"Collaboration Developer Health Dashboard\" src=\"https://github.com/user-attachments/assets/61e09239-07de-4a54-ad47-6ab50641c137\" /\u003e\n\u003cimg width=\"1807\" height=\"1080\" alt=\"Complexity Hotspots Dashboard\" src=\"https://github.com/user-attachments/assets/6958338e-cae7-4f19-8dae-e543f461f811\" /\u003e\n\u003cimg width=\"1807\" height=\"1080\" alt=\"Deployments Dashboard\" src=\"https://github.com/user-attachments/assets/a6cc37c8-ab92-42f4-8d33-2ad85c7a12d6\" /\u003e\n\u003cimg width=\"1807\" height=\"1080\" alt=\"Developer Landscape Dashboard\" src=\"https://github.com/user-attachments/assets/811ee083-80b4-4543-9267-e8afb1efc5e6\" /\u003e\n\u003cimg width=\"1080\" height=\"1137\" alt=\"IC Drilldown Developer Health\" src=\"https://github.com/user-attachments/assets/4223f663-2e10-4119-aa4b-2f187928bebc\" /\u003e\n\u003cimg width=\"1807\" height=\"1080\" alt=\"Incidents Dashboard\" src=\"https://github.com/user-attachments/assets/341ff259-1c97-4c2f-a144-2a26714b7c1c\" /\u003e\n\u003cimg width=\"1807\" height=\"1080\" alt=\"Investment Areas Dashboard\" src=\"https://github.com/user-attachments/assets/87292edd-26f8-4c09-9047-f8a6317a2b90\" /\u003e\n\u003cimg width=\"1807\" height=\"1080\" alt=\"Issue Types - Developer Health\" src=\"https://github.com/user-attachments/assets/c80c4e43-fc02-4ca1-9191-e6fefc268c02\" /\u003e\n\u003cimg width=\"1080\" height=\"1593\" alt=\"Quality   Risk Dashboard\" src=\"https://github.com/user-attachments/assets/5b398dd3-8c42-4f95-b77b-145f4ff71e27\" /\u003e\n\u003cimg width=\"1080\" height=\"1289\" alt=\"Repo Health Dashboard\" src=\"https://github.com/user-attachments/assets/e7e31df3-a073-476c-a76a-9830147bbdbd\" /\u003e\n\u003cimg width=\"1807\" height=\"1080\" alt=\"Well-being Team Level Dashboard\" src=\"https://github.com/user-attachments/assets/830f84e3-b356-4499-be89-7967a1537576\" /\u003e\n\u003cimg width=\"1080\" height=\"1669\" alt=\"Work Tracking Developer Health Dashboard\" src=\"https://github.com/user-attachments/assets/7bb42ae9-bcd3-437b-b60d-6064b9f9ee81\" /\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffull-chaos%2Fdev-health-ops","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffull-chaos%2Fdev-health-ops","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffull-chaos%2Fdev-health-ops/lists"}