{"id":23760655,"url":"https://github.com/george0st/qgate-sln-mlrun","last_synced_at":"2025-04-06T08:11:51.018Z","repository":{"id":184912623,"uuid":"623140009","full_name":"george0st/qgate-sln-mlrun","owner":"george0st","description":"MLRun/Iguazio/Nuclio quality gate solution.","archived":false,"fork":false,"pushed_at":"2024-07-11T21:01:15.000Z","size":3034,"stargazers_count":207,"open_issues_count":0,"forks_count":9,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-07-11T22:46:29.221Z","etag":null,"topics":["artificial-intelligence","data-science","e2e","feature-store","genai","iguazio","machine-learning","mlops","mlrun","mlrun-test","nuclio","quality-assessment","quality-assurance","quality-gate","testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/george0st.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-03T19:22:33.000Z","updated_at":"2024-07-13T20:45:00.949Z","dependencies_parsed_at":"2023-07-30T22:38:38.946Z","dependency_job_id":"9d2e01d5-b1dd-4923-9b6c-e27dbee636bc","html_url":"https://github.com/george0st/qgate-sln-mlrun","commit_stats":null,"previous_names":["george0st/qgate-mlrun"],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/george0st%2Fqgate-sln-mlrun","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/george0st%2Fqgate-sln-mlrun/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/george0st%2Fqgate-sln-mlrun/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/george0st%2Fqgate-sln-mlrun/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/george0st","download_url":"https://codeload.github.com/george0st/qgate-sln-mlrun/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247451654,"owners_count":20940939,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","data-science","e2e","feature-store","genai","iguazio","machine-learning","mlops","mlrun","mlrun-test","nuclio","quality-assessment","quality-assurance","quality-gate","testing"],"created_at":"2024-12-31T20:33:11.371Z","updated_at":"2025-04-06T08:11:51.000Z","avatar_url":"https://github.com/george0st.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![PyPI version fury.io](https://badge.fury.io/py/qgate-sln-mlrun.svg)](https://pypi.python.org/pypi/qgate-sln-mlrun/)\n![coverage](https://github.com/george0st/qgate-sln-mlrun/blob/master/coverage.svg?raw=true)\n![GitHub commit activity](https://img.shields.io/github/commit-activity/w/george0st/qgate-sln-mlrun)\n![GitHub release](https://img.shields.io/github/v/release/george0st/qgate-sln-mlrun)\n\n# QGate-Sln-MLRun\nThe Quality Gate for solution [MLRun](https://www.mlrun.org/) (and [Iguazio](https://www.iguazio.com/)). The main aims of the project are:\n- independent quality test (function, integration, performance, vulnerability, acceptance, ... tests)\n- deeper quality checks before full rollout/use in company environments\n- identification of possible compatibility issues (if any)\n- external and independent test coverage\n- community support\n- etc.\n\nThe tests use these key components, MLRun solution see **[GIT mlrun](https://github.com/mlrun/mlrun)**, \nsample meta-data model see **[GIT qgate-model](https://github.com/george0st/qgate-model)** and this project.\n\n## Test scenarios\nThe quality gate covers these test scenarios (✅ done, ✔ in-progress, ❌ planned):\n - **01 - Project**\n   - ✅ TS101: Create project(s)\n   - ✅ TS102: Delete project(s)\n - **02 - Feature set**\n   - ✅ TS201: Create feature set(s)\n   - ✅ TS202: Create feature set(s) \u0026 Ingest from DataFrame source (one step)\n   - ✅ TS203: Create feature set(s) \u0026 Ingest from CSV source (one step) \n   - ✅ TS204: Create feature set(s) \u0026 Ingest from Parquet source (one step)\n   - ✅ TS205: Create feature set(s) \u0026 Ingest from SQL source (one step)\n   - ✔  TS206: Create feature set(s) \u0026 Ingest from Kafka source (one step)\n   - ✔  TS207: Create feature set(s) \u0026 Ingest from HTTP source (one step)\n - **03 - Ingest data**\n   - ✅ TS301: Ingest data (Preview mode)\n   - ✅ TS302: Ingest data to feature set(s) from DataFrame source\n   - ✅ TS303: Ingest data to feature set(s) from CSV source \n   - ✅ TS304: Ingest data to feature set(s) from Parquet source\n   - ✅ TS305: Ingest data to feature set(s) from SQL source\n   - ✔  TS306: Ingest data to feature set(s) from Kafka source\n   - ✔  TS307: Ingest data to feature set(s) from HTTP source\n - **04 - Ingest data \u0026 pipeline**\n   - ✅ TS401: Ingest data \u0026 pipeline (Preview mode)\n   - ✅ TS402: Ingest data \u0026 pipeline to feature set(s) from DataFrame source\n   - ✅ TS403: Ingest data \u0026 pipeline to feature set(s) from CSV source \n   - ✅ TS404: Ingest data \u0026 pipeline to feature set(s) from Parquet source\n   - ✅ TS405: Ingest data \u0026 pipeline to feature set(s) from SQL source\n   - ✔  TS406: Ingest data \u0026 pipeline to feature set(s) from Kafka source\n   - ❌ TS407: Ingest data \u0026 pipeline to feature set(s) from HTTP source\n - **05 - Feature vector**\n   - ✅ TS501: Create feature vector(s)\n - **06 - Get data from vector**\n   - ✅ TS601: Get data from off-line feature vector(s)\n   - ✅ TS602: Get data from on-line feature vector(s)\n - **07 - Pipeline**\n   - ✅ TS701: Simple pipeline(s)\n   - ✅ TS702: Complex pipeline(s)\n   - ✅ TS703: Complex pipeline(s), mass operation\n - **08 - Build model**\n   - ✅ TS801: Build CART model\n   - ❌ TS802: Build XGBoost model\n   - ❌ TS803: Build DNN model\n - **09 - Serve model**\n   - ✅ TS901: Serving score from CART\n   - ❌ TS902: Serving score from XGBoost\n   - ❌ TS903: Serving score from DNN\n - **10 - Model monitoring/drifting**\n   - ❌ TS1001: Real-time monitoring\n   - ❌ TS1002: Batch monitoring\n - **11 - Performance tests**\n   - ❌ TS1101: Simple pipeline\n   - ❌ TS1102: Complex pipeline(s)\n   - ❌ TS11xx: TBD.\n   \nNOTE: Each test scenario contains addition specific test cases (e.g. with different\ntargets for feature sets, etc.).\n\n## Test inputs/outputs\nThe quality gate tests these inputs/outputs (✅ done, ✔ in-progress, ❌ planned):\n - Outputs (targets)\n   - ✅ RedisTarget, ✅ SQLTarget/MySQL, ✔ SQLTarget/Postgres, ✅ KafkaTarget\n   - ✅ ParquetTarget, ✅ CSVTarget\n   - ✅ File system, ❌ S3, ❌ BlobStorage\n - Inputs (sources)\n   - ✅ Pandas/DataFrame, ✅ SQLSource/MySQL, ❌ SQLSource/Postgres, ❌ KafkaSource\n   - ✅ ParquetSource, ✅ CSVSource\n   - ✅ File system, ❌ S3, ❌ BlobStorage\n\n\nThe current supported [sources/targets in MLRun](https://docs.mlrun.org/en/latest/feature-store/sources-targets.html).\n\n## Sample of outputs\n\n![Sample of outputs](https://github.com/george0st/qgate-sln-mlrun/blob/master/assets/imgs/qgt-mlrun-samples.png?raw=true)\n\nThe PART reports in original form, see:\n - all DONE - [HTML](https://htmlpreview.github.io/?https://github.com/george0st/qgate-sln-mlrun/blob/master/docs/samples/outputs/qgt-mlrun-sample.html), [TXT](https://github.com/george0st/qgate-sln-mlrun/blob/master/docs/samples/outputs/qgt-mlrun-sample.txt?raw=true)\n - with ERRors - [HTML](https://htmlpreview.github.io/?https://github.com/george0st/qgate-sln-mlrun/blob/master/docs/samples/outputs/qgt-mlrun-sample-err.html), [TXT](https://github.com/george0st/qgate-sln-mlrun/blob/master/docs/samples/outputs/qgt-mlrun-sample-err.txt?raw=true)\n\n## Usage\n\nYou can easy use this solution in four steps:\n1. Download content of these two GIT repositories to your local environment\n    - [qgate-sln-mlrun](https://github.com/george0st/qgate-sln-mlrun)\n    - [qgate-model](https://github.com/george0st/qgate-model)\n2. Update file `qgate-sln-mlrun.env` from qgate-model\n   - Update variables for MLRun/Iguazio, see `MLRUN_DBPATH`, `V3IO_USERNAME`, `V3IO_ACCESS_KEY`, `V3IO_API`\n     - setting of `V3IO_*` is needed only in case of Iguazio installation (not for pure free MLRun)\n   - Update variables for QGate, see `QGATE_*` (basic description directly in *.env)\n     - detail setup [configuration](./docs/configuration.md)\n3. Run from `qgate-sln-mlrun`\n   - **python main.py**\n4. See outputs (location is based on `QGATE_OUTPUT` in configuration)\n   - './output/qgt-mlrun-\u003cdate\u003e \u003csequence\u003e.html'\n   - './output/qgt-mlrun-\u003cdate\u003e \u003csequence\u003e.txt'\n\nPrecondition: You have available MLRun or Iguazio solution (MLRun is part of that),\nsee official [installation steps](https://docs.mlrun.org/en/latest/install.html), or directly installation for [Desktop Docker](https://docs.mlrun.org/en/latest/install/local-docker.html). \n\n## Tested with\nThe project was tested with these MLRun versions (see [change log](https://docs.mlrun.org/en/latest/change-log/index.html)):\n - **MLRun** (in Kubernates or Desktop Docker)\n   - ❌ MLRun 1.8.0 (plan Q1/2025)\n   - ✔  MLRun 1.7.0 (? 1.7.1 ?)\n   - ✅ MLRun 1.6.4, 1.6.3, 1.6.2, 1.6.1, 1.6.0\n   - ✅ MLRun 1.5.2, 1.5.1, 1.5.0\n   - ✅ MLRun 1.4.1\n   - ✅ MLRun 1.3.0\n - **Iguazio** (k8s, on-prem, VM on VMware)\n   - ✅ Iguazio 3.5.3 (with MLRun 1.4.1)\n   - ✅ Iguazio 3.5.1 (with MLRun 1.3.0)\n\nNOTE: Current state, only the last MLRun/Iguazio versions are tested \n(the backward compatibility is based on MLRun/Iguazio, [see](https://docs.mlrun.org/en/latest/install.html#mlrun-client-backward-compatibility)).\n\n## Others\n - **To-Do**, the list of expected/future improvements, [see](./docs/todo_list.md)\n - **Applied limits**, the list of applied limits/issues, [see](./docs/applied-limits.md) \n - **How can you test the solution?**, you have to focus on Linux env. or \n Windows with WSL2 ([see](./docs/testing.md) step by step tutorial)\n - **MLRun/Iguazio**, the key changes in a nutshell (customer view), [see](./docs/mlrun-iguazio-release-notes.md)\n - **MLRun local installation**, [see the hack](https://github.com/mlrun/mlrun/blob/development/hack/local/README.md) \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeorge0st%2Fqgate-sln-mlrun","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeorge0st%2Fqgate-sln-mlrun","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeorge0st%2Fqgate-sln-mlrun/lists"}