{"id":40699828,"url":"https://github.com/dssg/catwalk","last_synced_at":"2026-01-21T12:02:57.626Z","repository":{"id":46084559,"uuid":"97516449","full_name":"dssg/catwalk","owner":"dssg","description":"Training, testing, and evaluating machine learning classifier models","archived":false,"fork":false,"pushed_at":"2022-07-15T18:40:43.000Z","size":108,"stargazers_count":3,"open_issues_count":4,"forks_count":4,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-08-31T10:40:42.341Z","etag":null,"topics":["triage"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dssg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-07-17T19:54:23.000Z","updated_at":"2018-04-10T21:59:55.000Z","dependencies_parsed_at":"2022-09-26T18:30:33.113Z","dependency_job_id":null,"html_url":"https://github.com/dssg/catwalk","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/dssg/catwalk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dssg%2Fcatwalk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dssg%2Fcatwalk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dssg%2Fcatwalk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dssg%2Fcatwalk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dssg","download_url":"https://codeload.github.com/dssg/catwalk/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dssg%2Fcatwalk/sbom","scorecard":{"id":358018,"data":{"date":"2025-08-11","repo":{"name":"github.com/dssg/catwalk","commit":"b421e5bacc133f80d144ff4438dd65574a35b269"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.2,"checks":[{"name":"Code-Review","score":4,"reason":"Found 8/19 approved changesets -- score normalized to 4","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":9,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Warn: project license file does not contain an FSF or OSI license."],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 24 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"22 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2021-856 / GHSA-5545-2q6w-2gh6","Warn: Project is vulnerable to: GHSA-6p56-wp2h-9hxr","Warn: Project is vulnerable to: PYSEC-2019-108 / GHSA-9fq2-x9r6-wfmf","Warn: Project is vulnerable to: PYSEC-2021-857 / GHSA-f7c7-j99h-c22f","Warn: Project is vulnerable to: GHSA-fpfv-jqm9-f5jm","Warn: Project is vulnerable to: PYSEC-2017-1 / GHSA-frgw-fgh6-9g52","Warn: Project is vulnerable to: PYSEC-2020-73","Warn: Project is vulnerable to: PYSEC-2021-142 / GHSA-8q59-q68h-6hv4","Warn: Project is vulnerable to: PYSEC-2018-49 / GHSA-rprw-h62v-c2w7","Warn: Project is vulnerable to: PYSEC-2020-107 / GHSA-jjw5-xxj6-pcv5","Warn: Project is vulnerable to: PYSEC-2024-110 / GHSA-jw8x-6495-233v","Warn: Project is vulnerable to: PYSEC-2020-108","Warn: Project is vulnerable to: PYSEC-2019-156 / GHSA-xp76-357g-9wqq","Warn: Project is vulnerable to: PYSEC-2023-102","Warn: Project is vulnerable to: PYSEC-2023-114","Warn: Project is vulnerable to: PYSEC-2019-124 / GHSA-38fc-9xqv-7f7q","Warn: Project is vulnerable to: PYSEC-2019-123 / GHSA-887w-45rq-vxgf","Warn: Project is vulnerable to: PYSEC-2012-9 / GHSA-hfg2-wf6j-x53p","Warn: Project is vulnerable to: PYSEC-2022-238 / GHSA-h3qr-fjhm-jphw","Warn: Project is vulnerable to: PYSEC-2021-437 / GHSA-5xp3-jfq3-5q8x","Warn: Project is vulnerable to: PYSEC-2020-173 / GHSA-gpvv-69j7-gwj8","Warn: Project is vulnerable to: PYSEC-2023-228 / GHSA-mq26-g339-26xf"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-18T10:06:05.273Z","repository_id":46084559,"created_at":"2025-08-18T10:06:05.273Z","updated_at":"2025-08-18T10:06:05.273Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28632781,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-21T04:47:28.174Z","status":"ssl_error","status_checked_at":"2026-01-21T04:47:22.943Z","response_time":86,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["triage"],"created_at":"2026-01-21T12:02:50.336Z","updated_at":"2026-01-21T12:02:57.613Z","avatar_url":"https://github.com/dssg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Catwalk\n\nTraining, testing, and evaluating machine learning classifier models\n\n[![Build Status](https://travis-ci.org/dssg/catwalk.svg?branch=master)](https://travis-ci.org/dssg/catwalk)\n[![codecov](https://codecov.io/gh/dssg/catwalk/branch/master/graph/badge.svg)](https://codecov.io/gh/dssg/catwalk)\n[![codeclimate](https://codeclimate.com/github/dssg/catwalk.png)](https://codeclimate.com/github/dssg/catwalk)\n\n**Note**: Catwalk is now a bundled component of [Triage](https://github.com/dssg/triage), and future development will take place there. To utilize Catwalk within your custom pipeline, you may still import it from there.\n\n---\n\nAt the core of many predictive analytics applications is the need to train classifiers on large set of design matrices, test and temporally cross-validate them, and generate evaluation metrics about them.\n\nPython's scikit-learn package provides much of this functionality, but it is not trivial to design large experiments with it in a persistable way. Catwalk builds upon the functionality offered by scikit-learn by implementing:\n\n- Saving of modeling results and metadata in a [Postgres database](https://github.com/dssg/results-schema) for later analysis\n- Exposure of computationally-intensive tasks as discrete workloads that can be used with different parallelization solutions (e.g. multiprocessing, Celery)\n- Different model persistence strategies such as on-filesystem or Amazon S3, that can be easily switched between\n- Hashing classifier model configuration to only retrain a model if necessary.\n- Various best practices in areas like input scaling for different classifier types and feature importance\n- Common scikit-learn model evaluation metrics as well as the ability to bundle custom evaluation metrics\n\n\n## Components\n\nThis functionality is concentrated in the following components:\n\n- [catwalk.ModelTrainer](catwalk/model_trainers.py): Train a configured experiment grid on pre-made design matrices, and store each model's metadata and feature importances in a database.\n- [catwalk.Predictor](catwalk/predictors.py): Given a trained model and another matrix (ie, a test matrix), generate prediction probabilities and store them in a database.\n- [catwalk.ModelEvaluator](catwalk/evaluation.py): Given a set of model prediction probabilities, generate metrics (for instance, precision and recall at various thresholds) and store them in a database.\n\n## Usage\n\nBelow is a complete sample usage of the three Catwalk components.\n\n```\n\nimport datetime\n\nimport pandas\nfrom sqlalchemy import create_engine\n\nfrom metta import metta_io as metta\n\nfrom catwalk.storage import FSModelStorageEngine, MettaCSVMatrixStore\nfrom catwalk.model_trainers import ModelTrainer\nfrom catwalk.predictors import Predictor\nfrom catwalk.evaluation import ModelEvaluator\nfrom catwalk.utils import save_experiment_and_get_hash\n\n\n# create a sqlalchemy database engine pointing to a Postgres database\ndb_engine = create_engine(...)\n\n# A path on your filesystem under which to store matrices and models\nproject_path = 'mytestproject/modeling'\n\n# create a toy train matrix from scratch\n# and saving it using metta to generate a unique id for its metadata\n# catwalk uses both the matrix and metadata\ntrain_matrix = pandas.DataFrame.from_dict({\n\t'entity_id': [1, 2],\n\t'feature_one': [3, 4],\n\t'feature_two': [5, 6],\n\t'label': [7, 8]\n}).set_index('entity_id')\ntrain_metadata = {\n\t'feature_start_time': datetime.date(2012, 12, 20),\n\t'end_time': datetime.date(2016, 12, 20),\n\t'label_name': 'label',\n\t'label_timespan': '1y',\n\t'feature_names': ['ft1', 'ft2'],\n}\ntrain_matrix_uuid = metta.archive_matrix(train_metadata, train_matrix, format='csv')\n\n# The MettaCSVMatrixStore bundles the matrix and metadata together\n# for catwalk to use\ntrain_matrix_store = MettaCSVMatrixStore(\n\tmatrix_path='{}.csv'.format(train_matrix_uuid),\n\tmetadata_path='{}.yaml'.format(train_matrix_uuid)\n)\n\n\n# Similarly, create a test matrix\nas_of_date = datetime.date(2016, 12, 21)\n\ntest_matrix = pandas.DataFrame.from_dict({\n\t'entity_id': [3],\n\t'feature_one': [8],\n\t'feature_two': [5],\n\t'label': [5]\n}).set_index('entity_id')\n\ntest_metadata = {\n\t'label_name': 'label',\n\t'label_timespan': '1y',\n\t'end_time': as_of_date,\n}\ntest_matrix_uuid = metta.archive_matrix(test_metadata, test_matrix, format='csv')\n\n# The MettaCSVMatrixStore bundles the matrix and metadata together\n# for catwalk to use\ntest_matrix_store = MettaCSVMatrixStore(\n\tmatrix_path='{}.csv'.format(test_matrix_uuid),\n\tmetadata_path='{}.yaml'.format(test_matrix_uuid)\n)\n\n# The ModelStorageEngine handles the persistence of model pickles\n# In this case, we are using FSModelStorageEngine to use the local filesystem\nmodel_storage_engine = FSModelStorageEngine(project_path)\n\n# To ensure that we can relate all of our persistent database records with\n# each other, we bind them together with an experiment hash. This is based\n# on the hash of experiment configuration that you pass in here, so if the\n# code fails halfway through and has to run a second time, it will use the\n# already-trained models but save the new ones under the same experment\n# hash.\n\n# Here, we will just save a trivial experiment configuration.\n# You can put any information you want in here, as long as it is hashable\nexperiment_hash = save_experiment_and_get_hash({'name': 'myexperimentname'}, db_engine)\n\n# instantiate pipeline objects. these will to the brunt of the work\ntrainer = ModelTrainer(\n\tproject_path=project_path,\n\texperiment_hash=experiment_hash,\n\tmodel_storage_engine=model_storage_engine,\n\tdb_engine=db_engine,\n\tmodel_group_keys=['label_name', 'label_timespan']\n)\npredictor = Predictor(\n\tproject_path,\n\tmodel_storage_engine,\n\tdb_engine\n)\nmodel_evaluator = ModelEvaluator(\n\t[{'metrics': ['precision@'], 'thresholds': {'top_n': [5]}}],\n\tdb_engine\n)\n\n# run the pipeline\ngrid_config = {\n\t'sklearn.linear_model.LogisticRegression': {\n\t\t'C': [0.00001, 0.0001],\n\t\t'penalty': ['l1', 'l2'],\n\t\t'random_state': [2193]\n\t}\n}\n\n# trainer.train_models will run the entire specified grid\n# and return database ids for each model\nmodel_ids = trainer.train_models(\n\tgrid_config=grid_config,\n\tmisc_db_parameters=dict(test=True),\n\tmatrix_store=train_matrix_store\n)\n\nfor model_id in model_ids:\n\tpredictions_proba = predictor.predict(\n\t\tmodel_id=model_id,\n\t\tmatrix_store=test_matrix_store,\n\t\tmisc_db_parameters=dict(),\n\t\ttrain_matrix_columns=['feature_one', 'feature_two']\n\t)\n\n\tmodel_evaluator.evaluate(\n\t\tpredictions_proba=predictions_proba,\n\t\tlabels=test_store.labels(),\n\t\tmodel_id=model_id,\n\t\tevaluation_start_time=as_of_date,\n\t\tevaluation_end_time=as_of_date,\n\t\tas_of_date_frequency='6month'\n\t)\n\n```\nAfter running the above code, results will be stored in your Postgres database in [this structure](https://github.com/dssg/results-schema/blob/master/results_schema/schema.py)\n\nIn addition to being usable on the design matrices of your current project, Catwalk's functionality is used in [triage](https://github.com/dssg/triage) as a part of an entire modeling experiment that incorporates earlier tasks like feature generation and matrix building.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdssg%2Fcatwalk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdssg%2Fcatwalk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdssg%2Fcatwalk/lists"}