{"id":19607063,"url":"https://github.com/bookingcom/ml_quality_maturity_framework","last_synced_at":"2025-10-13T22:14:23.795Z","repository":{"id":272913378,"uuid":"784142906","full_name":"bookingcom/ml_quality_maturity_framework","owner":"bookingcom","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-10T10:51:49.000Z","size":3136,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-10-13T22:14:21.809Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc-by-sa-4.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bookingcom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-04-09T09:14:22.000Z","updated_at":"2025-04-10T10:51:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"b32f6cbf-7c9d-4f96-a53b-99158202ac7d","html_url":"https://github.com/bookingcom/ml_quality_maturity_framework","commit_stats":null,"previous_names":["bookingcom/ml_quality_maturity_framework"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bookingcom/ml_quality_maturity_framework","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookingcom%2Fml_quality_maturity_framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookingcom%2Fml_quality_maturity_framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookingcom%2Fml_quality_maturity_framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookingcom%2Fml_quality_maturity_framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bookingcom","download_url":"https://codeload.github.com/bookingcom/ml_quality_maturity_framework/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookingcom%2Fml_quality_maturity_framework/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279017158,"owners_count":26085983,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T10:08:38.210Z","updated_at":"2025-10-13T22:14:23.790Z","avatar_url":"https://github.com/bookingcom.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Machine Learning Quality and Maturity Package\n\nThis repository houses the Machine Learning (ML) Quality and Maturity Framework package developed at Booking.com.\\\nThe structure of the repository is delineated as follows:\n\n* The primary class, located in `assessments.py`, serves the function of generating the final quality and maturity score.\\\nAdditionally, it creates a report detailing the technical gaps identified, along with recommended best practices for their mitigation.\\\nInitialization of this class involves parsing a `.csv` file containing technical gap information for each quality sub-characteristic.\n\n* The file `assessment_automation.py` is dedicated to inferring the gaps present in a model based on data extracted from the ML registry. \nThe ML registry serves as a database housing pertinent information regarding the models under analysis.\n\n* The file `report_automation.py`, provides utilities for automating the generation of multiple reports, \nas well as the aggregation of historical evaluation data into a single dataset.\n\n* The `practice_prioritization` directory contains code responsible for prioritizing the sequence of best practices \nto be applied in addressing identified technical gaps.\n\n### Set up the environment and run the tests\nTo verify the functionality of the repository, create a Python virtual environment, following the instructions below.\nThe package was tested using `python3.11`.\n\n```\npython -m ensurepip --upgrade  # Make sure you have pip installed\npip install virtualenv\npython -m venv venv\nsource ./venv/bin/activate  # Activates the virtual environment\npip install -r requirements.txt \n```\nTo verify everything is working properly you can run the test suite of the package.\\\nYou can do this by typing `tox` from the package root directory.\n\n#### wkhtmltopdf dependency\n\nIn case wkhtmltopdf is missing, install as follows:\n - MacOS: `brew install Caskroom/cask/wkhtmltopdf`\n - Linux: `sudo apt install -y wkhtmltopdf`\n\n### Generate a quality report\nTo generate a quality and maturity report, follow the example provided in the `create_quality_report.ipynb` notebook.\nTo run the notebook first create a Jupyter kernel from the virtual environment:\n\n```\npip install ipykernel\npython -m ipykernel install --user --name=venv\njupyter notebook\n```\n\n#### Automation levels\nWhen generating a quality report, we provide 2 levels of automation with different requirements: \n- **Semi-automated**: This method requires gathering the ML system's quality information through a Google form, using the \nquestions specified in [quality_assessment_form.md](quality_assessment_form.md) and manually converting them into\ntechnical gaps using the quality assessment requirements. The gaps should be stored in a `csv` \nfile ([example](assessments/inputs/gaps_model_1.csv)). The quality report can then be created following the instructions\nin the `create_quality_report.ipynb` notebook.\n- **Fully-automated**: This method requires a registry with the ML systems' metadata. Using the function \n`infer_gaps_from_registry_metadata` in [assessment_automation.py](ml_quality/assessment_automation.py) one can \ngenerate the technical gaps `csv` file with no human intervention, leveraging the information from the registry. The \nquality report is then generated automatically as shown in the `create_quality_report.ipynb` notebook.\n\n### Generate multiple reports from a list of models with technical gaps\nTo generate multiple reports for a list of models, utilize the script `create_reports.py`. The script generates reports for all models listed in `models_to_assess.csv`.\\\nIt also consolidates previous evaluations into a historical report. You can run the script by typing\n```\npython create_reports.py\n```\nThe file `models_to_assess.csv` contains the list of models you want to be assessed, the path of the file with technical gaps,\\\nthe date of the evaluation, the team owning the model and the model family (in case the same evaluation is valid for more than a single ML system).\nThe script will create a `.csv` file with the historical data of all the evaluations stored in the repo. \nThe file created will be under `all_models_summary/historical_summary.csv`\n\n# Example reports\nBelow we provide examples of full quality reports from assessed systems, using the ML quality Python package.  \n\nThe images below show an example of a report for a fully mature system. No technical gaps are present, \nall the fulfilled quality attributes are listed in green.\n![Fully mature system - report page 1](images/report_100_1.png)\n![Fully mature system - report page 2](images/report_100_2.png)\n\n\nExample of a report for a system of maturity level 1 is provided below. The gaps to be fulfilled to pass to the next \nmaturity level are shown in red. The quality attributes to be fulfilled for the subsequent maturity levels are shown in \norange. Below each quality attribute the user can see both the motivation of a certain technical gap and a \nrecommendation to remove it.\n![Level 1 system - report page 1](images/report_84_1.png)\n![Level 1 system - report page 2](images/report_84_2.png)\n\n\n## Reconstruct quality improvements from past assessments\nEach evaluation produces a report in .html and .pdf format but also a `summary.csv` which stores the \nevaluation input for full reproducibility. \nWe uploaded a set of anonymized summary reports from Booking.com `assessments/example_booking_reports_summary` \nfor you to familiarize with the framework. These summaries can be used to re-build historical information\nas shown in the notebook `notebooks/monitor_historical_progress.ipynb` and obtain granular information\nat the quality sub-characteristic level.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbookingcom%2Fml_quality_maturity_framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbookingcom%2Fml_quality_maturity_framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbookingcom%2Fml_quality_maturity_framework/lists"}