{"id":28824150,"url":"https://github.com/kernelci/kcidb-ng","last_synced_at":"2026-03-15T05:31:12.641Z","repository":{"id":287921253,"uuid":"966239975","full_name":"kernelci/kcidb-ng","owner":"kernelci","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-13T11:22:45.000Z","size":106,"stargazers_count":0,"open_issues_count":11,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-13T11:42:15.389Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kernelci.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-14T16:05:45.000Z","updated_at":"2025-06-13T11:22:04.000Z","dependencies_parsed_at":"2025-04-30T12:48:35.101Z","dependency_job_id":"fe4a6d6e-efe3-4307-b429-8ea5f1d59269","html_url":"https://github.com/kernelci/kcidb-ng","commit_stats":null,"previous_names":["nuclearcat/kcidb-rest","kernelci/kcidb-rest","kernelci/kcidb-ng"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kernelci/kcidb-ng","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kernelci%2Fkcidb-ng","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kernelci%2Fkcidb-ng/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kernelci%2Fkcidb-ng/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kernelci%2Fkcidb-ng/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kernelci","download_url":"https://codeload.github.com/kernelci/kcidb-ng/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kernelci%2Fkcidb-ng/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260660116,"owners_count":23043488,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-19T01:05:39.901Z","updated_at":"2026-03-15T05:31:12.634Z","avatar_url":"https://github.com/kernelci.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KCIDB-ng\n\nKCIDB-ng is fresh rewrite of our wrappers around the KernelCI Database (KCIDB). It provides services for submitting kernel test data to the KCIDB database via HTTP requests, and also\ncome with log analysis capabilities.\n\n## Architecture\n\nThe system consists of several interconnected components:\n\n1. **kcidb-restd-rs** - A Rust-based REST service that:\n   - Receives JSON submissions via HTTP/HTTPS\n   - Authenticates users via JWT\n   - Stores valid submissions in a spool directory\n   - Provides status endpoints for submissions\n\n2. **ingester** - A Python service that:\n   - Processes JSON submissions from the spool directory\n   - Validates them against the KCIDB schema\n   - Loads them into the KCIDB database\n   - Archives processed submissions\n   This service has been replaced with [KernelCI Dashboard's implementation of the ingester](https://github.com/kernelci/dashboard/blob/main/backend/kernelCI_app/management/commands/monitor_submissions.py) and was left here just as a temporary and standalone usage.\n\n3. **logspec-worker** - A Python service that:\n   - Monitors the database for failed tests and builds\n   - Downloads and analyzes log files using the logspec library\n   - Identifies issues and incidents from logs\n   - Submits findings back to KCIDB\n\n4. **PostgreSQL Database** - Stores all KCIDB data\n   - Can be run locally (self-hosted mode)\n\n## Installation\n\n### Prerequisites\n\n- Docker and Docker Compose (the newer `docker compose` plugin, not the legacy `docker-compose`; version 2.0+ required)\n- Git\n\n### Clone the Repository\n\n```bash\ngit clone https://github.com/kernelci/kcidb-ng.git\ncd kcidb-ng\n```\n\nThe legacy `ingester/ingester.py` implementation is deprecated and kept for compatibility only.\nCurrent compose setups use the dashboard-backed ingester (`monitor_submissions`), built from `Dockerfile.django-ingester`.\nPlanned removal of the deprecated ingester is April 30, 2026.\n\n### Quick Start (recommended)\nTo quickly start the KCIDB-ng services with a local PostgreSQL database, run:\n\n```bash\n./self-hosted.sh run\n```\nThis script will:\n- Build and start the Docker containers\n- Initialize the PostgreSQL database\n- Start the REST API, dashboard ingester, and logspec-worker services\n\nTo run from local Dockerfiles (no GHCR image pulls), use:\n\n```bash\n./self-hosted.sh --dev run\n```\n\nAlso available commands:\n- `./self-hosted.sh down` - Stops the services\n- `./self-hosted.sh clean` - Stops and removes all containers, configs, databases, networks, and volumes\n\n### Manual configuration\n\nCreate a `.env` file in the root directory with the following environment variables:\n\n```\n# PostgreSQL configuration\nPOSTGRES_PASSWORD=kcidb\nPS_PASS=kcidb\nPG_URI=postgresql:dbname=kcidb user=kcidb_editor password=kcidb host=db port=5432\n# JWT authentication\nJWT_SECRET=your_jwt_secret\n```\n\n## Usage\n\n### Starting the Services\n\n#### Self-hosted Mode (with local PostgreSQL)\n\nThe self-hosted profile includes a local PostgreSQL database and an initialization service:\n\n```bash\ndocker compose --profile=self-hosted up -d --build\n```\n\nThis command:\n- Builds and/or pulls required images (as defined by compose files)\n- Sets up a local PostgreSQL database\n- Initializes the database schema\n- Starts the REST API, dashboard ingester, and logspec-worker services\n\nNote: By default it is expecting PostgreSQL to be running with default settings, except postgres password which is set to `kcidb`.\nIt will also create a user `kcidb_editor` with password `kcidb` and a database `kcidb`, and user `kcidb_viewer` with password `kcidb` for read-only access.\n\n#### Compose modes\n\nThis project now has two compose workflows:\n\n- Default mode uses images from GitHub Container Registry for `kcidb-rest` and `ingester` (as defined in `docker-compose.yaml` / `docker-compose-all.yaml`).\n- Local development mode rebuilds those images from local Dockerfiles by adding `docker-compose-dev.yaml`.\n\nRun with prebuilt images:\n\n```bash\ndocker compose -f docker-compose.yaml up -d\ndocker compose -f docker-compose-all.yaml up -d\n```\n\nRun local-source builds:\n\n```bash\ndocker compose -f docker-compose.yaml -f docker-compose-dev.yaml up -d --build\ndocker compose -f docker-compose-all.yaml -f docker-compose-dev.yaml up -d --build\n```\n\nFor self-hosted local PostgreSQL with local builds:\n\n```bash\ndocker compose -f docker-compose.yaml -f docker-compose-dev.yaml --profile=self-hosted up -d --build\n```\n\n### Generating tokens\n\nIf your kcidb-ng is installed in isolated environment, you can disable JWT authentication by commenting out the JWT command in `docker-compose.yaml`:\n\n```yaml\n#    command: [\"/usr/local/bin/kcidb-restd-rs\",\"-j\",\"\"]\n```\n\nIf you want to use JWT authentication, you can generate a token using the following command:\n\n```bash\nkcidb-restd-rs/tools/jwt_rest.py --secret YOUR_SECRET --origin YOUR_ORIGIN\n```\n\n### Validating tokens\n\nTo validate a JWT token, you can use the following command:\n\n```bash\ncurl -X GET \\\n  -H \"Authorization: Bearer \u003cjwt_token\u003e\" \\\n  https://localhost:443/authtest\n```\nThis will return a JSON response with the token's validity.\n\n### Sending Data to the API\n\nTo submit data to the REST API:\n\n```bash\ncurl -X POST \\\n  -H \"Authorization: Bearer \u003cjwt_token\u003e\" \\\n  -H \"Content-Type: application/json\" \\\n  -d @submission.json \\\n  https://localhost:443/submit\n```\n\n### Checking Status\n\nYou can check the status of your submission using:\n\n```bash\ncurl -X GET   -H \"Authorization: Bearer \u003cyourtoken\u003e\" https://staging.db.kernelci.org/status?id=wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz\n{\"id\":\"wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz\",\"status\":\"failed\",\"message\":\"File found\"}\n```\n\nPossible status values:\n\n-\t{\"id\":\"0\",\"status\":\"error\",\"message\":\"Empty id\"}\tThe request was invalid (e.g., missing or invalid ID, authentication error, etc.).\n-\t{\"id\":\"wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz\",\"status\":\"inprogress\",\"message\":\"File still in progress\"}\tA submission file with .json.temp exists, indicating the upload is not yet complete.\n-\t{\"id\":\"wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz\",\"status\":\"ready\",\"message\":\"File waiting for processing\"}\tThe submission file exists and is ready for processing, but not yet archived or failed.\n-\t{\"id\":\"wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz\",\"status\":\"processed\",\"message\":\"File archived\"}\tThe submission has been processed and archived.\n-\t{\"id\":\"wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz\",\"status\":\"failed\",\"message\":\"File failed to pass validation\"}\tThe submission failed validation and is in the failed directory.\n-\t{\"id\":\"wvTu6myNQOlM7IEWhHJz8WnnE0GTG1yz\",\"status\":\"notfound\",\"message\":\"File not found\"}\tNo submission file with the given ID was found in any expected location.\n\n\n## Directory Structure\n\n- `/spool`: Stores incoming submissions (managed by docker volumes)\n  - `/spool/failed`: Stores submissions that failed to process\n  - `/spool/archive`: Stores successfully processed submissions\n\n- `/state`: Stores application state (managed by docker volumes)\n  - `processed_builds.db`: Tracks processed builds\n  - `processed_tests.db`: Tracks processed tests\n\n- `/cache`: Caches downloaded log files for logspec-worker\n\n## Development and Debugging\n\n### Viewing Logs\n\n```bash\ndocker logs kcidb-rest\ndocker logs ingester\ndocker logs logspec-worker\ndocker logs postgres\n```\n\n### Connecting to the Database\n\n```bash\ndocker exec -it postgres psql -U kcidb_editor -d kcidb\n```\n\n### Authentication\n\nThe REST API uses JWT for authentication. To disable JWT authentication (not recommended for production):\n\nUncomment this line in docker-compose.yaml:\n```yaml\n#    command: [\"/usr/local/bin/kcidb-restd-rs\",\"-j\",\"\"]\n```\n\n### Manual Log Processing\n\nTo manually process a log file through logspec without submitting it to the database, you can run:\n\n```bash\ndocker exec -it logspec-worker python /app/logspec_worker.py --spool-dir /app/spool --origins microsoft --dry-run\n```\n\n## License\n\nThis project is licensed under the [LGPL-2.1 license](https://www.gnu.org/licenses/old-licenses/lgpl-2.1.en.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkernelci%2Fkcidb-ng","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkernelci%2Fkcidb-ng","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkernelci%2Fkcidb-ng/lists"}