{"id":47082020,"url":"https://github.com/openscilab/pycm-api","last_synced_at":"2026-03-12T07:03:13.282Z","repository":{"id":330521172,"uuid":"1123025085","full_name":"openscilab/pycm-api","owner":"openscilab","description":"PyCM REST API","archived":false,"fork":false,"pushed_at":"2026-03-10T18:55:25.000Z","size":39,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-11T00:32:03.755Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://api.pycm.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openscilab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-26T03:42:39.000Z","updated_at":"2026-03-10T11:54:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/openscilab/pycm-api","commit_stats":null,"previous_names":["openscilab/pycm-api"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/openscilab/pycm-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openscilab%2Fpycm-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openscilab%2Fpycm-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openscilab%2Fpycm-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openscilab%2Fpycm-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openscilab","download_url":"https://codeload.github.com/openscilab/pycm-api/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openscilab%2Fpycm-api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30417686,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-12T06:40:58.731Z","status":"ssl_error","status_checked_at":"2026-03-12T06:40:40.296Z","response_time":114,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-03-12T07:03:09.695Z","updated_at":"2026-03-12T07:03:13.275Z","avatar_url":"https://github.com/openscilab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PyCM API\n\nThis project is a REST API wrapper built around the [PyCM](https://github.com/sepandhaghighi/pycm) library, designed to streamline the creation, storage, comparison, and reporting of confusion matrices for machine learning model evaluation. The API offers comprehensive support for standard classification metrics, multi-label classification scenarios, and performance curve generation including ROC and PR curves. Key capabilities include full confusion matrix management with persistent storage, access to essential metrics like accuracy, precision, recall, and F1-score, and rich visualization options through HTML reports and PNG plots. Users can compare multiple confusion matrices side-by-side to evaluate model performance, handle complex multi-label classification with per-class and per-sample metrics, and generate performance curves complete with Area Under the Curve calculations. The system also incorporates secure user management with API key-based authentication and provides an admin dashboard for overseeing both users and confusion matrices, making it a complete solution for machine learning practitioners who need robust model evaluation tools.\n\n## Installation\n\n### Prerequisites\n\n- Python \u003e= 3.10\n- pip package manager\n\n### Setup Steps\n\n1. **Clone the repository** (if applicable) or navigate to the project directory:\n```bash\ncd pycm-api\n```\n\n2. **Install dependencies**:\n```bash\npip install -r requirements.txt\n```\n\n3. **Set environment variables** for admin authentication. These credentials are required to access administrative endpoints (`/users/`, `/cms/`).\n```bash\nexport PYCM_API_ADMIN=\"your_admin_username\"\nexport PYCM_API_ADMIN_PASSWORD=\"your_admin_password\"\n```\n\n4. **Run the application**:\n```bash\nuvicorn app.main:app --reload\n```\n\nThe `--reload` flag enables auto-reload on code changes (useful for development).\n\n5. **Access the API**:\n- API Base URL: `http://127.0.0.1:8000`\n- Interactive API Documentation: `http://127.0.0.1:8000/docs`\n\n## API Endpoints\n\n### User Management\n\n#### `POST /sign_up/`\nRegister a new user account.\n\n**Request Body**:\n```json\n{\n  \"email\": \"user@example.com\",\n  \"password\": \"secure_password\"\n}\n```\n\n**Response**: User object with `id`, `email`, `api_key`, `credit`, `is_active`, and `cms` (user confusion matrices list) fields. Note that using `api_key` user will be authenticated for using the APIs.\n\n\n#### `POST /sign_in/`\nAuthenticate an existing user.\n\n**Request Body**:\n```json\n{\n  \"email\": \"user@example.com\",\n  \"password\": \"secure_password\"\n}\n```\n\n**Response**: User object (same as sign_up).\n\n---\n\n### Confusion Matrix Operations\n\n#### `POST /cm/create`\nCreate a new confusion matrix.\n\n**Request Body**:\n```json\n{\n  \"api_key\": \"your_api_key_here\",\n  \"actual_vector\": [1, 0, 1, 1, 0, 1, 0, 0],\n  \"predicted_vector\": [1, 0, 1, 0, 0, 1, 1, 0]\n}\n```\n\n**Response**: Confusion matrix object with `uid` (unique identified) field.\n\n**Note**: The `actual_vector` and `predicted_vector` can contain integers, floats, or strings representing class labels.\n\n#### `GET /cm/`\nRetrieve a confusion matrix by UID.\n\n**Query Parameters**:\n- `api_key` (required): Your API key\n- `cm_uid` (required): The UID of the confusion matrix\n\n**Response**: Confusion matrix with metrics:\n```json\n{\n  \"uid\": \"cm_uid_here\",\n  \"accuracy\": 0.75,\n  \"precision\": 0.8,\n  \"recall\": 0.8,\n  \"f1\": 0.8,\n  \"confusion_matrix\": [[3, 1], [1, 3]]\n}\n```\n\n#### `POST /cm/update`\nUpdate an existing confusion matrix.\n\n**Query Parameters**:\n- `cm_uid` (required): The UID of the confusion matrix to update\n\n**Request Body**: Same as `/cm/create`\n\n**Response**: Updated confusion matrix object.\n\n#### `DELETE /cm/{cm_uid}`\nDelete a confusion matrix.\n\n**Query Parameters**:\n- `api_key` (required): Your API key\n\n**Response**: `{\"message\": \"Confusion matrix deleted\"}`\n\n**Note**: Only the owner of the confusion matrix can delete it.\n\n#### `GET /cm/report`\nGenerate an HTML report for a confusion matrix.\n\n**Query Parameters**:\n- `api_key` (required): Your API key\n- `cm_uid` (required): The UID of the confusion matrix\n\n**Response**: HTML content with comprehensive statistics and visualizations.\n\n#### `GET /cm/plot`\nGenerate a PNG plot visualization for a confusion matrix.\n\n**Query Parameters**:\n- `api_key` (required): Your API key\n- `cm_uid` (required): The UID of the confusion matrix\n\n**Response**: PNG image file.\n\n---\n\n### Comparison Operations\n\n#### `POST /compare/`\nCompare multiple confusion matrices.\n\n**Request Body**:\n```json\n{\n  \"api_key\": \"your_api_key_here\",\n  \"cm_uids\": [\"uid1\", \"uid2\", \"uid3\"]\n}\n```\n\n**Response**: Comparison results including:\n- `cm_uids`: List of compared confusion matrix UIDs\n- `best_name`: Identifier of the best performing model\n- `cm_scores`: Dictionary of scores for each confusion matrix\n- `cm_orders`: Ordered list of confusion matrices by performance\n\n---\n\n### Multi-Label Classification\n\n#### `POST /mlcm/`\nCreate a multi-label confusion matrix (one-time use, not persisted).\n\n**Request Body**:\n```json\n{\n  \"api_key\": \"your_api_key_here\",\n  \"actual_vector\": [[\"class1\", \"class2\"], [\"class2\"], [\"class1\"]],\n  \"predicted_vector\": [[\"class1\"], [\"class2\"], [\"class1\", \"class2\"]],\n  \"classes\": [\"class1\", \"class2\"]\n}\n```\nIn this example for the first the sample, the ground truth contains both \"class1\" and \"class2,\" but the model only predicted \"class1.\"\n\n**Response**: Multi-label confusion matrix with:\n- `multihot_actual`: Multi-hot encoded actual labels\n- `multihot_predicted`: Multi-hot encoded predicted labels\n- `classes`: List of class names\n- `cm_by_classes`: Per-class confusion matrices\n- `cm_by_samples`: Per-sample confusion matrices\n\n**Note**: This endpoint does not persist the confusion matrix to the database.\n\n---\n\n### Performance Curves\n\n#### `POST /curve`\nGenerate ROC or PR curves.\n\n**Request Body**:\n```json\n{\n  \"api_key\": \"your_api_key_here\",\n  \"type\": \"ROC\",\n  \"actual_vector\": [0, 1, 0, 1, 1],\n  \"probability_vector\": [[0.1, 0.9], [0.3, 0.7], [0.2, 0.8], [0.4, 0.6], [0.5, 0.5]],\n  \"classes\": [\"class0\", \"class1\"]\n}\n```\n\n**Parameters**:\n- `type`: Either `\"ROC\"` or `\"PR\"`\n- `actual_vector`: Ground truth labels\n- `probability_vector`: Predicted probabilities for each class (list of lists)\n- `classes`: Optional list of class names/labels\n\n**Response**: Curve data including:\n- `thresholds`: List of threshold values\n- `auc_trp`: Dictionary mapping class names to AUC (Area Under Curve) values\n\n---\n\n### Administrative Endpoints (admin only)\n\n#### `GET /users/`\nList all users.\n\n**Query Parameters**:\n- `skip` (optional): Number of users to skip (default: 0)\n- `limit` (optional): Maximum number of users to return (default: 100)\n\n**Response**: List of user objects.\n\n#### `GET /cms/`\nList all confusion matrices.\n\n**Query Parameters**:\n- `skip` (optional): Number of confusion matrices to skip (default: 0)\n- `limit` (optional): Maximum number of confusion matrices to return (default: 100)\n\n**Response**: List of confusion matrix objects.\n\n---\n\n## Usage Examples\n\n```mermaid\nflowchart LR\n\n    subgraph UserFlow[\"Normal User API Flow\"]\n        UStart((Start))\n\n        SignUp[\"POST sign_up\"]\n        SignIn[\"POST sign_in\"]\n\n        CreateCM[\"POST cm create\"]\n        GetCM[\"GET cm\"]\n        UpdateCM[\"POST cm update\"]\n        DeleteCM[\"DELETE cm by uid\"]\n\n        Report[\"GET cm report\"]\n        Plot[\"GET cm plot\"]\n\n        Compare[\"POST compare\"]\n\n        MLCM[\"POST mlcm\"]\n        Curve[\"POST curve\"]\n\n        UEnd((End))\n\n        UStart --\u003e SignUp\n        UStart --\u003e SignIn\n        SignUp --\u003e CreateCM\n        SignIn --\u003e CreateCM\n\n        CreateCM --\u003e GetCM\n        GetCM --\u003e UpdateCM\n        UpdateCM --\u003e GetCM\n\n        GetCM --\u003e Report\n        GetCM --\u003e Plot\n\n        CreateCM --\u003e Compare\n        Compare --\u003e UEnd\n\n        GetCM --\u003e DeleteCM\n        DeleteCM --\u003e UEnd\n\n        UStart --\u003e MLCM\n        UStart --\u003e Curve\n    end\n\n    subgraph AdminFlow[\"Admin Only API Flow\"]\n        AStart((Admin Start))\n\n        ListUsers[\"GET users\"]\n        ListCMs[\"GET cms\"]\n\n        AEnd((End))\n\n        AStart --\u003e ListUsers\n        AStart --\u003e ListCMs\n        ListUsers --\u003e AEnd\n        ListCMs --\u003e AEnd\n    end\n```\n\n### Complete Workflow Example\n\n```python\nimport requests\n\nBASE_URL = \"http://127.0.0.1:8000\"\n\n# 1. Sign up a new user\nsignup_response = requests.post(f\"{BASE_URL}/sign_up/\", json={\n    \"email\": \"ml_engineer@example.com\",\n    \"password\": \"secure_password\"\n})\nuser_data = signup_response.json()\napi_key = user_data[\"api_key\"]\nprint(f\"API Key: {api_key}\")\n\n# 2. Create a confusion matrix\ncm_response = requests.post(f\"{BASE_URL}/cm/create\", json={\n    \"api_key\": api_key,\n    \"actual_vector\": [0, 1, 0, 1, 1, 0, 1, 0],\n    \"predicted_vector\": [0, 1, 1, 1, 1, 0, 0, 0]\n})\ncm_data = cm_response.json()\ncm_uid = cm_data[\"uid\"]\nprint(f\"Confusion Matrix UID: {cm_uid}\")\n\n# 3. Retrieve the confusion matrix\nget_cm_response = requests.get(\n    f\"{BASE_URL}/cm/\",\n    params={\"api_key\": api_key, \"cm_uid\": cm_uid}\n)\ncm_metrics = get_cm_response.json()\nprint(f\"Accuracy: {cm_metrics['accuracy']}\")\nprint(f\"Precision: {cm_metrics['precision']}\")\nprint(f\"Recall: {cm_metrics['recall']}\")\nprint(f\"F1-Score: {cm_metrics['f1']}\")\n\n# 4. Generate HTML report\nreport_response = requests.get(\n    f\"{BASE_URL}/cm/report\",\n    params={\"api_key\": api_key, \"cm_uid\": cm_uid}\n)\nwith open(\"report.html\", \"w\") as f:\n    f.write(report_response.text)\n\n# 5. Download plot\nplot_response = requests.get(\n    f\"{BASE_URL}/cm/plot\",\n    params={\"api_key\": api_key, \"cm_uid\": cm_uid}\n)\nwith open(\"plot.png\", \"wb\") as f:\n    f.write(plot_response.content)\n\n# 6. Compare multiple models\ncompare_response = requests.post(f\"{BASE_URL}/compare/\", json={\n    \"api_key\": api_key,\n    \"cm_uids\": [cm_uid, \"another_cm_uid\"]\n})\ncomparison = compare_response.json()\nprint(f\"Best model: {comparison['best_name']}\")\n```\n\n### Using cURL\n\n```bash\n# Sign up\ncurl -X POST \"http://127.0.0.1:8000/sign_up/\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"email\": \"test@example.com\", \"password\": \"test123\"}'\n\n# Create confusion matrix (replace YOUR_API_KEY with actual key)\ncurl -X POST \"http://127.0.0.1:8000/cm/create\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"api_key\": \"YOUR_API_KEY\",\n    \"actual_vector\": [0, 1, 0, 1],\n    \"predicted_vector\": [0, 1, 1, 1]\n  }'\n\n# Get confusion matrix\ncurl \"http://127.0.0.1:8000/cm/?api_key=YOUR_API_KEY\u0026cm_uid=CM_UID\"\n```\n\n## Storage\n\nThe application uses a hybrid storage approach:\n\n### Database (SQLite)\n- **Location**: `app/DB.db`\n- **Stores**: \n  - User accounts (email, hashed password, API key, credit, status)\n  - Confusion matrix metadata (UID, owner relationship, creation timestamp)\n\n### File System\n- **Confusion Matrix Objects**: `app/cms/`\n  - Serialized PyCM confusion matrix objects\n  - Filename format: `{cm_uid}.obj`\n  \n- **HTML Reports**: `app/reports/`\n  - Generated HTML reports for confusion matrices\n  - Filename format: `{cm_uid}.html`\n  \n- **Plot Images**: `app/plots/`\n  - PNG images of confusion matrix visualizations\n  - Filename format: `{cm_uid}.png`\n\nAll storage directories are created automatically if they don't exist.\n\n## Acknowledgments\n\n[NLnet foundation](https://nlnet.nl) has supported the PyCM API for version **0.1** through the [NGI0 Commons](https://nlnet.nl/commonsfund/) Fund. This fund is set up by [NLnet foundation](https://nlnet.nl) with funding from the European Commission's [Next Generation Internet program](https://ngi.eu), administered by DG Communications Networks, Content, and Technology under grant agreement [**No 101135429**](https://nlnet.nl/project/PyCM-API/).\n\n\u003ca href=\"https://nlnet.nl\"\u003e\u003cimg src=\"https://github.com/openscilab/pycm-api/raw/main/otherfiles/NlNet.svg\" height=\"50px\" alt=\"NLnet foundation\"\u003e\u003c/a\u003e \u0026nbsp;  \u003ca href=\"https://nlnet.nl/commonsfund/\"\u003e\u003cimg src=\"https://github.com/openscilab/pycm-api/raw/main/otherfiles/NGI0Core.svg\" height=\"50px\" alt=\"NGI0 Commons\"\u003e\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenscilab%2Fpycm-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenscilab%2Fpycm-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenscilab%2Fpycm-api/lists"}