{"id":48138443,"url":"https://github.com/4gt-104/matyan-backend","last_synced_at":"2026-04-04T16:48:06.146Z","repository":{"id":345895145,"uuid":"1183543427","full_name":"4gt-104/matyan-backend","owner":"4gt-104","description":"REST API server for Matyan experiment tracking — reads and writes experiment data, metrics, and artifacts to FoundationDB; handles control operations.","archived":false,"fork":false,"pushed_at":"2026-03-29T14:16:23.000Z","size":535,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-29T17:32:25.913Z","etag":null,"topics":["aim","apache-2","asyncio","experiment-tracking","fastapi","foundationdb","kafka","machine-learning","matyan","mlops","object-storage","python","rest-api"],"latest_commit_sha":null,"homepage":"https://4gt-104.github.io/matyan-core/stable/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/4gt-104.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-16T18:00:28.000Z","updated_at":"2026-03-29T14:16:26.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/4gt-104/matyan-backend","commit_stats":null,"previous_names":["4gt-104/matyan-backend"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/4gt-104/matyan-backend","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/4gt-104%2Fmatyan-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/4gt-104%2Fmatyan-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/4gt-104%2Fmatyan-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/4gt-104%2Fmatyan-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/4gt-104","download_url":"https://codeload.github.com/4gt-104/matyan-backend/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/4gt-104%2Fmatyan-backend/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31406117,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T10:20:44.708Z","status":"ssl_error","status_checked_at":"2026-04-04T10:20:06.846Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aim","apache-2","asyncio","experiment-tracking","fastapi","foundationdb","kafka","machine-learning","matyan","mlops","object-storage","python","rest-api"],"created_at":"2026-04-04T16:48:05.989Z","updated_at":"2026-04-04T16:48:06.136Z","avatar_url":"https://github.com/4gt-104.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Matyan Backend\n\nREST API and workers for the Matyan experiment-tracking stack (fork of Aim). Serves reads and control operations from **FoundationDB**; consumes ingestion and control events from **Kafka**; uses **S3/GCS/Azure** for artifact blobs. The UI talks to this API; training clients send data via the frontier, which publishes to Kafka consumed by these workers.\n\n## Layout\n\n- **`src/matyan_backend/`** — Python package: FastAPI app (`app.py`), API routes under `api/` (runs, experiments, tags, projects, dashboards, reports, streaming), `storage/` (FDB + S3/GCS/Azure), `workers/` (ingestion + control Kafka consumers), `jobs/` (FDB lock, used by CLI cleanup commands), `backup/` (export/restore), CLI in `cli.py`.\n- **Entrypoints**: `matyan-backend start` (API server, default port 53800), `matyan-backend ingest-worker`, `matyan-backend control-worker`; plus one-off CLI commands (reindex, backup, restore, finish-stale, cleanup-orphan-blobs, cleanup-tombstones, convert tensorboard).\n\n## Prerequisites\n\n- Python 3.12+. The package uses `uv` in the repo: `uv run matyan-backend` or install then `matyan-backend` CLI.\n- **Runtime dependencies**: FoundationDB (cluster file), Kafka (for workers), blob store. For local dev, typically run FDB + Kafka + S3 (RustFS) via docker-compose.\n\n## Run\n\n- **API server**: `uv run matyan-backend start` (or `matyan-backend start`). Options: `--host`, `--port` (defaults: `0.0.0.0`, 53800). API is under `/api/v1`; health at `/health/ready/`, `/health/live/`, metrics at `/metrics/` when enabled.\n- **Workers**: `uv run matyan-backend ingest-worker` and `uv run matyan-backend control-worker`. Both require Kafka and FDB; ingestion worker also writes to FDB and reads blob storage config for blob references.\n- **CLI (one-off)**: `reindex` (rebuild indexes), `backup` / `restore`, `finish-stale`, `cleanup-orphan-blobs`, `cleanup-tombstones`. See the backend CLI help (`matyan-backend cleanup-orphan-blobs --help`, `matyan-backend cleanup-tombstones --help`) and [References — CLI](../../docs/refs/cli.md) for all options. Cleanup commands are intended for CronJobs or cron; use `--dry-run` to preview and `--lock-ttl-seconds` for FDB-based single-run locking. Optional: `convert tensorboard` to convert TensorBoard logs to backup format.\n\n## Configuration (environment variables)\n\n| Variable | Default | Purpose |\n|----------|---------|---------|\n| `MATYAN_ENVIRONMENT` / `ENVIRONMENT` | `development` | When `production`, strict checks apply (see Production configuration). |\n| `LOG_LEVEL` | `INFO` | Log level (loguru + uvicorn). |\n| `FDB_CLUSTER_FILE` | `fdb.cluster` | Path to FoundationDB cluster file. |\n| `BLOB_BACKEND_TYPE` | `s3` | Storage backend: `s3`, `gcs`, or `azure`. |\n| `S3_ENDPOINT` | `http://localhost:9000` | S3-compatible API URL. |\n| `S3_ACCESS_KEY` / `S3_SECRET_KEY` | (dev defaults) | S3 credentials. |\n| `S3_BUCKET` | `matyan-artifacts` | Bucket for artifacts (when using `s3`). |\n| `S3_REGION` | `us-east-1` | S3 region (default: `us-east-1`). |\n| `GCS_BUCKET` | `matyan-artifacts` | Bucket for artifacts (when using `gcs`). |\n| `AZURE_CONTAINER` | `matyan-artifacts` | Container for artifacts (when using `azure`). |\n| `AZURE_CONN_STR` | `\"\"` | Azure connection string. |\n| `AZURE_ACCOUNT_URL` | `\"\"` | Azure account URL (for `DefaultAzureCredential`). |\n| `BLOB_URI_SECRET` | (dev default) | Fernet key for blob URIs; must be set in production. |\n| `KAFKA_BOOTSTRAP_SERVERS` | `localhost:9092` | Kafka broker list. |\n| `KAFKA_DATA_INGESTION_TOPIC` | `data-ingestion` | Topic for ingestion messages. |\n| `KAFKA_CONTROL_EVENTS_TOPIC` | `control-events` | Topic for control events. |\n| `KAFKA_SECURITY_PROTOCOL` / `KAFKA_SASL_*` | (empty) | Optional Kafka SASL. |\n| `METRICS_ENABLED` | `true` | Expose Prometheus metrics. |\n| `METRICS_PORT` | `9090` | Port for metrics HTTP server (workers). |\n| `INGEST_MAX_MESSAGES_PER_TXN` | `100` | Max messages per FDB transaction (ingestion worker). |\n| `INGEST_MAX_TXN_BYTES` | `8388608` (8 MB) | Target max transaction size; FDB limit is 10 MB. |\n| `CORS_ORIGINS` | (localhost list) | Allowed origins for CORS. |\n\nSource of truth: [config.py](src/matyan_backend/config.py).\n\n## Production configuration\n\nSee **[docs/PRODUCTION_CONFIG.md](docs/PRODUCTION_CONFIG.md)** for enabling production mode (`MATYAN_ENVIRONMENT=production`), required overrides, and supplying secrets via env or a secrets backend.\n\n## Deployment\n\n- **Docker**: Build the backend image (context from repo root); run API and workers as separate processes or containers.\n- **Kubernetes/Helm**: The chart in `deploy/helm/matyan` deploys the backend API, ingestion worker, and control worker as separate Deployments; optional CronJobs for `cleanup-orphan-blobs` and `cleanup-tombstones`. Configure FDB, blob storage (S3, GCS, Azure), and Kafka via chart values; see the chart README. Set `MATYAN_ENVIRONMENT=production` and required env for production.\n\n## Related\n\n- **UI**: matyan-ui calls this backend REST API.\n- **Frontier**: matyan-frontier publishes to Kafka; backend workers consume.\n- **API models**: matyan-api-models shared types (Kafka messages, run creation, etc.).\n- **Monorepo**: This package lives under `extra/matyan-backend` in the matyan-core repo.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F4gt-104%2Fmatyan-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F4gt-104%2Fmatyan-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F4gt-104%2Fmatyan-backend/lists"}