{"id":15208892,"url":"https://github.com/databrickslabs/automl-toolkit","last_synced_at":"2025-10-03T01:31:17.671Z","repository":{"id":50823833,"uuid":"184655700","full_name":"databrickslabs/automl-toolkit","owner":"databrickslabs","description":"Toolkit for Apache Spark ML for Feature clean-up, feature Importance calculation suite, Information Gain selection, Distributed SMOTE, Model selection and training, Hyper parameter optimization and selection, Model interprability.","archived":true,"fork":false,"pushed_at":"2021-06-01T13:19:47.000Z","size":165765,"stargazers_count":191,"open_issues_count":10,"forks_count":44,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-01-17T06:07:56.960Z","etag":null,"topics":["apache-spark","feature-engineering","machinelearning","ml","pyspark","scala","spark"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databrickslabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-05-02T21:45:32.000Z","updated_at":"2024-08-23T03:50:53.000Z","dependencies_parsed_at":"2022-09-12T17:50:25.354Z","dependency_job_id":null,"html_url":"https://github.com/databrickslabs/automl-toolkit","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databrickslabs%2Fautoml-toolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databrickslabs%2Fautoml-toolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databrickslabs%2Fautoml-toolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databrickslabs%2Fautoml-toolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databrickslabs","download_url":"https://codeload.github.com/databrickslabs/automl-toolkit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235059234,"owners_count":18929279,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","feature-engineering","machinelearning","ml","pyspark","scala","spark"],"created_at":"2024-09-28T07:03:18.837Z","updated_at":"2025-10-03T01:31:11.753Z","avatar_url":"https://github.com/databrickslabs.png","language":"HTML","funding_links":[],"categories":["Libraries"],"sub_categories":["Distributed Frameworks"],"readme":"# Databricks Labs AutoML Toolkit\n[Release Notes](RELEASE_NOTES.md) |\n[Python API Docs](python/docs/APIDOCs.md) |\n[Python Artifact](python/dist/pyAutoML-0.2.0-py3-none-any.whl) |\n[Developer Docs](APIDOCS.md) |\n[Python Docs](python/docs/APIDOCs.md) |\n[Analysis Tools Docs](ANALYSIS_TOOLS_DOCS.md) |\n[Demo](demos) |\n[Release Artifacts](bin) |\n[Contributors](#core-contribution-team)\n\n\nThis Databricks Labs project is a non-officially-supported end-to-end supervised learning solution for automating:\n* Feature clean-up \n    * Advanced NA fill, covariance calculations, collinearity determination, outlier filtering, and data casting\n* Feature Importance calculation suite\n    * RandomForest or XGBoost determinations\n* Feature Interaction with [Information Gain selection](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence)\n* Feature vectorization\n* Advanced train/test split techniques (including Distributed [SMOTE](https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis#SMOTE) (KSample))\n* Model selection and training\n* Hyper parameter optimization and selection\n    * Hyperspace, Genetic, and MBO-based selection\n* Batch Prediction through serialized [SparkML Pipelines](https://spark.apache.org/docs/latest/ml-pipeline.html)\n* Logging of model results and training runs (using [MLFlow](https://mlflow.org))\n* Model interprability (including distributed [Shapley Values](https://christophm.github.io/interpretable-ml-book/shapley.html) )\n\nThis package utilizes Apache Spark ML and currently supports the following model family types:\n\n* Decision Trees (Regressor and Classifier)\n* Gradient Boosted Trees (Regressor and Classifier)\n* Random Forest (Regressor and Classifier)\n* Linear Regression\n* Logistic Regression\n* Multi-Layer Perceptron Classifier\n* Support Vector Machines\n* XGBoost (Regressor and Classifier)\n\n\u003e NOTE: With the upgrade to Spark 3 (Scala 2.12) LightGBM is **no longer supported** but will be added in a future release. \n\n## Documentation\n\nScala API documentation can be found [here](APIDOCS.md)\n\nPython API documentation can be found [here](python/docs/APIDOCs.md) \n\nAnalytics Package API documentation can be found [here](ANALYSIS_TOOLS_DOCS.md)\n\n## Installing - Recommended!\nDarabricks Labs AutoML can be pulled from maven central with the following coordinates.\nExample - to install 0.7.2 AutoML:\n```\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.databricks.labs\u003c/groupId\u003e\n  \u003cartifactId\u003eautoml-toolkit_2.12\u003c/artifactId\u003e\n  \u003cversion\u003e0.8.1\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n## Building\n\nDatabricks Labs AutoML can be build with either [SBT](https://www.scala-sbt.org/) or [Maven](https://maven.apache.org/).\n\n```text\nThis package requires Java 1.8.x  and scala 2.12.x to be installed on your system prior to building.\n```\n\nAfter cloning this repo onto your local system, navigate to the root directory and execute either:\n\n##### Maven Build\n```sbtshell\nmvn clean install -DskipTests\n```\n\n##### SBT Build\n```sbtshell\nsbt package\n```\nIf there is any StackOverflowError during the build, adjust the stack size on your computer's JVM. Example:\n```sbtshell\n#For Maven\nexport MAVEN_OPTS=-Xss2m\n#For SBT\nexport SBT_OPTS=\"-Xss2M\"\n```\n\n\nThis will skip unit test execution (it is not recommended to run unit tests in local mode against this package as unit testing is asynchronous and incredibly CPU intensive for this code base.)\n\n\n## Setup\n\nOnce the artifact has been built, attach to the Databricks Shard through either the [DBFS API](https://docs.databricks.com/api/latest/dbfs.html) or the GUI.  Once loaded into the account, utilize either the [Libraries API](https://docs.databricks.com/api/latest/libraries.html#install) to attach to a cluster, or utilize the GUI to attach the .jar to the cluster.\n\n```text\nNOTE: It is not recommended to attach this libarary to all clusters on the account.  \n\nUse of an ML Runtime cluster configuration is highly advised to ensure that custom management of dependent \nlibraries and configurations are provided 'out of the box'\n\n```\n\nAttach the following libraries to the cluster:\n* The automl toolkit jar created above. (automatedml_2.12-((version)).jar)\n* If using the PySpark API for the toolkit, the [.whl file](python/docs/APIDOCs.md#Setup) for the PySpark API.\n\n\u003e IMPORTANT NOTE: as of release 0.7.1, the mlflow libraries in pypi and Maven are NO LONGER NEEDED.  Attaching them\n\u003e to your cluster WILL prevent the run from logging and will throw an exception.  DO NOT ATTACH EITHER OF THEM.\n\n## Getting Started\n\nThis package provides a number of different levels of API interaction, from the highest-level \"default only\" FamilyRunner to low-level APIs that allow for highly customizable workflows to be created for automated ML tuning and Inference.\n\nSince v0.6.0 we have included an API to work with the pipeline semantics around feature engineering steps and full predict pipelines.For the purposes of a quick-start intro, the below example is of the highest-level API access point.\n\n```scala\nimport com.databricks.labs.automl.executor.config.ConfigurationGenerator\nimport com.databricks.labs.automl.executor.FamilyRunner\nimport org.apache.spark.ml.PipelineModel\n\nval data = spark.table(\"ben_demo.adult_data\")\nval overrides = Map(\n  \"labelCol\" -\u003e \"label\",\n  \"mlFlowLoggingFlag\" -\u003e false,\n  \"scalingFlag\" -\u003e true,\n  \"oneHotEncodeFlag\" -\u003e true,\n  \"pipelineDebugFlag\" -\u003e true\n)\nval randomForestConfig = ConfigurationGenerator\n        .generateConfigFromMap(\"RandomForest\", \"classifier\", overrides)\n\nval runner = FamilyRunner(data, Array(randomForestConfig)).executeWithPipeline()\n\nrunner.bestPipelineModel(\"RandomForest\").transform(data)\n\n//Serialize it\nrunner.bestPipelineModel(\"RandomForest\").write.overwrite().save(\"tmp/predict-pipeline-1\")\n\n// Load it for running inference\nval pipelineModel = PipelineModel.load(\"tmp/predict-pipeline-1\")\nval predictDf = pipelineModel.transform(data)\n```\nThis example will take the default configuration for all of the application parameters (excepting the overridden parameters in overrides Map) and execute Data Preparation tasks, Feature Vectorization, and automatic tuning of all 3 specified model types.  At the conclusion of each run, the results and model artifacts will be logged to the mlflow location that was specified in the configuration.\n\nFor a listing of all available parameter overrides and their functionality, see the [Developer Docs](APIDOCS.md)\n\n### Inference via Mlflow Run ID\nIt is also possible to use MlFlow Run ID for inference, if Mlflow logging is turned on during training.\nFor usage, see [this](PIPELINE_API_DOCS.md#running-inference-pipeline-directly-against-an-mlflow-run-id-since-v061)\n\nFor all available pipeline APIs. please see [Developer Docs](PIPELINE_API_DOCS.md)\n\n## Feedback\n\nIssues with the application?  Found a bug?  Have a great idea for an addition?\nFeel free to file an issue or contact [Ben](mailto:benjamin.wilson@databricks.com)\n\n## Contributing\nHave a great idea that you want to add?  Fork the repo and submit a PR!\n\n## Legal Information\nThis software is provided as-is and is not officially supported by Databricks through customer technical support channels.\nSupport, questions, and feature requests can be communicated via email -\u003e benjamin.wilson@databricks.com or through the Issues page of this repo.\nPlease see the [legal agreement](LICENSE.txt) and understand that issues with the use of this code will not be answered or investigated by Databricks Support.  \n\n## Core Contribution team\n* Lead Developer: Ben Wilson, Practice Leader, Databricks\n* Developer: Daniel Tomes, RSA Practice Leader, Databricks\n* Developer: Jas Bali, Sr. Solutions Consultant, Databricks\n* Developer: Mary Grace Moesta, Customer Success Engineer, Databricks\n* Developer: Nick Senno, Resident Solutions Architect, Databricks\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabrickslabs%2Fautoml-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabrickslabs%2Fautoml-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabrickslabs%2Fautoml-toolkit/lists"}