{"id":13795194,"url":"https://github.com/jpmml/jpmml-evaluator-spark","last_synced_at":"2026-01-05T23:16:50.125Z","repository":{"id":2644220,"uuid":"47058451","full_name":"jpmml/jpmml-evaluator-spark","owner":"jpmml","description":"PMML evaluator library for the Apache Spark cluster computing system (http://spark.apache.org/)","archived":false,"fork":false,"pushed_at":"2022-04-02T11:16:55.000Z","size":105,"stargazers_count":94,"open_issues_count":9,"forks_count":43,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-05-19T03:00:46.617Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jpmml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-11-29T10:03:37.000Z","updated_at":"2024-03-31T14:17:02.000Z","dependencies_parsed_at":"2022-08-06T12:30:51.591Z","dependency_job_id":null,"html_url":"https://github.com/jpmml/jpmml-evaluator-spark","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpmml%2Fjpmml-evaluator-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpmml%2Fjpmml-evaluator-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpmml%2Fjpmml-evaluator-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpmml%2Fjpmml-evaluator-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jpmml","download_url":"https://codeload.github.com/jpmml/jpmml-evaluator-spark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225157000,"owners_count":17429698,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T23:00:53.179Z","updated_at":"2026-01-05T23:16:50.118Z","avatar_url":"https://github.com/jpmml.png","language":"Java","funding_links":[],"categories":["人工智能"],"sub_categories":["机器学习"],"readme":"JPMML-Evaluator-Spark [![Build Status](https://github.com/jpmml/jpmml-evaluator-spark/workflows/maven/badge.svg)](https://github.com/jpmml/jpmml-evaluator-spark/actions?query=workflow%3A%22maven%22)\n=====================\n\nPMML evaluator library for Apache Spark.\n\n# Features #\n\nThis library provides an Apache Spark ML-compatible wrapper for the [JPMML-Evaluator](https://github.com/jpmml/jpmml-evaluator) library.\n\n# Prerequisites #\n\n* Java Platform, Standard Edition 11 or newer.\n* Apache Spark 3.0.X through 3.5.X.\n\n# Installation #\n\n### Compatibility matrix\n\nActive development branches:\n\n| JPMML-Evaluator-Spark branch | Apache Spark version | Scala version |\n|------------------------------|----------------------|---------------|\n| [`2.0.X`](https://github.com/jpmml/jpmml-evaluator-spark/tree/2.0.X) | 3.0.X through 3.5.X | 2.12.X |\n| [`master`](https://github.com/jpmml/jpmml-evaluator-spark/tree/master/) | 4.0.X and 4.1.X | 2.13.X |\n\nArchived development branches:\n\n| JPMML-Evaluator-Spark branch | Apache Spark version | Scala version |\n|------------------------------|----------------------|---------------|\n| [`1.4.X`](https://github.com/jpmml/jpmml-evaluator-spark/tree/1.4.X) | 3.0.X through 3.5.X | 2.12.X |\n\n### Library\n\nThe JPMML-Evaluator-Spark library JAR file (together with accompanying Java source and Javadocs JAR files) is released via [Maven Central Repository](https://repo1.maven.org/maven2/org/jpmml/).\n\nThe current version is **1.4.0** (1 January, 2026).\n\n```xml\n\u003cdependency\u003e\n\t\u003cgroupId\u003eorg.jpmml\u003c/groupId\u003e\n\t\u003cartifactId\u003ejpmml-evaluator-spark\u003c/artifactId\u003e\n\t\u003cversion\u003e1.4.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n### Runtime uber-JAR file\n\nEnter the project root directory and build using [Apache Maven](https://maven.apache.org/):\n\n```\nmvn clean install\n```\n\nThe build produces two JAR files:\n* `target/jpmml-evaluator-spark-2.1-SNAPSHOT.jar` - Library JAR file.\n* `target/jpmml-evaluator-spark-runtime-2.1-SNAPSHOT.jar` - Runtime uber-JAR file.\n\nUse the library JAR file for integrating JPMML-Evaluator into a full-scale Apache Spark application. Note that you will need to solve a number of dependency conflicts in your build configuration file.\n\nUse the runtime uber-JAR file for _ad hoc_ tasks, such as adding JPMML-Evaluator to a Toree Jupyter notebook.\n\n# Usage #\n\n## Workflow\n\nBuild a `org.jpmml.evaluator.Evaluator` object that will do the actual evaluation work:\n\n```scala\nimport java.io.InputStream\nimport org.jpmml.evaluator.{Evaluator, LoadingModelEvaluatorBuilder}\n\nval pmmlIs: InputStream = ???\n\nval evaluator: Evaluator = try {\n\tnew LoadingModelEvaluatorBuilder()\n\t\t.load(pmmlIs)\n\t\t.build\n} finally {\n\tpmmlIs.close\n}\n\n// Perform self-check\nevaluator.verify\n```\n\nWrap this `Evaluator` object into an `org.jpmml.evaluator.spark.PMMLTransformer` object to make it look and behave like a regular Apache Spark ML transformer.\n\nThere are two built-in implementation classes available:\n\n* `org.jpmml.evaluator.spark.FlatPMMLTransformer` \n* `org.jpmml.evaluator.spark.NestedPMMLTransformer` \n\nThey are functionally identical evaluation-wise.\nThe only difference is about how the result columns are structured (flat layout vs. nested layout).\n\nIt is possible to toggle column groups (ie. inputs, targets and outputs) on and off to keep the results maximally relevant.\n\n```scala\nimport org.jpmml.evaluator.spark.{FlatPMMLTransformer, NestedPMMLTransformer, PMMLTransformer}\n\nval pmmlTransformer = new FlatPMMLTransformer(evaluator)\n//val pmmlTransformer = new NestedPMMLTransformer(evaluator)\n```\n\nA newly-constructed `PMMLTransformer` object is instantly ready for transformation work (ie. it does not exhibit any fitting behaviour).\n\n```scala\nval df = ???\n\nval transformedDf = pmmlTransformer.transform(df)\n```\n\nEvaluation exceptions, if any, are caught and stored in a dedicated exceptions column.\n\n## API\n\n### `PMMLTransformer`\n\nAbstract base class that provides common parameters and functionality.\n\nParameters:\n\n* `inputs: BooleanParam = true`. Copy all columns from the input dataset to the transformed dataset?\n* `targets: BooleanParam = true`. Produce columns for PMML target fields (ie. primary results)?\n* `outputs: BooleanParam = true`. Produce columns for PMML output fields (ie. secondary results)?\n* `exceptionCol: Param[String] = \"pmmlException\"`. The name of the exceptions column.\n* `syntheticTargetName: Param[String] = \"_target\"`. The substitute name for a synthetic target field column (the default name for a synthetic target field is `null`, which is not a valid Apache Spark column name).\n\n### `FlatPMMLTransformer`\n\nConcrete implementation class, which maps all PMML result fields to top-level columns.\n\nParameters: N/A\n\nTransformed schema for the example `DecisionTreeIris` model:\n\n* Four input columns.\n* One PMML target field column.\n* Three PMML output field columns.\n* One exceptions column.\n\n```\nroot\n |-- Sepal.Length: double (nullable = true)\n |-- Sepal.Width: double (nullable = true)\n |-- Petal.Length: double (nullable = true)\n |-- Petal.Width: double (nullable = true)\n |-- Species: string (nullable = true)\n |-- probability(setosa): double (nullable = true)\n |-- probability(versicolor): double (nullable = true)\n |-- probability(virginica): double (nullable = true)\n |-- pmmlException: string (nullable = true)\n```\n\nIf the evaluation fails for some row, then all PMML result fields columns contain `null` for that row.\n\n### `NestedPMMLTransformer`\n\nConcrete implementation class, which maps all PMML target and output fields to a single top-level nested results column.\n\nParameters:\n\n* `resultsCol: Param[String] = \"pmmlResults\"`. The name of the nested results column.\n\nTransformed schema for the example `DecisionTreeIris` model:\n\n* Four input columns.\n* One nested results column, containing one PMML target field and three output fields.\n* One exceptions column.\n\n```\nroot\n |-- Sepal.Length: double (nullable = true)\n |-- Sepal.Width: double (nullable = true)\n |-- Petal.Length: double (nullable = true)\n |-- Petal.Width: double (nullable = true)\n |-- pmmlResults: struct (nullable = true)\n |    |-- Species: string (nullable = true)\n |    |-- probability(setosa): double (nullable = true)\n |    |-- probability(versicolor): double (nullable = true)\n |    |-- probability(virginica): double (nullable = true)\n |-- pmmlException: string (nullable = true)\n```\n\nIf the evaluation fails for some row, then the results column contains `null` for that row.\n\nUse the dot notation to access individual fields afterwards:\n\n```scala\ntransformedDf.select(\"pmmlResults.Species\").show()\n```\n\n# License #\n\nJPMML-Evaluator-Spark is licensed under the terms and conditions of the [GNU Affero General Public License, Version 3.0](https://www.gnu.org/licenses/agpl-3.0.html).\nFor a quick summary of your rights (\"Can\") and obligations (\"Cannot\" and \"Must\") under AGPLv3, please refer to [TLDRLegal](https://tldrlegal.com/license/gnu-affero-general-public-license-v3-(agpl-3.0)).\n\nIf you would like to use JPMML-Evaluator-Spark in a proprietary software project, then it is possible to enter into a licensing agreement which makes it available under the terms and conditions of the [BSD 3-Clause License](https://opensource.org/licenses/BSD-3-Clause) instead.\n\n# Additional information #\n\nJPMML-Evaluator-Spark is developed and maintained by Openscoring Ltd, Estonia.\n\nInterested in using JPMML software in your software? Please contact [info@openscoring.io](mailto:info@openscoring.io)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjpmml%2Fjpmml-evaluator-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjpmml%2Fjpmml-evaluator-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjpmml%2Fjpmml-evaluator-spark/lists"}