{"id":15287730,"url":"https://github.com/raistlintao/sparkmodelhelper","last_synced_at":"2026-05-06T07:37:58.209Z","repository":{"id":57734604,"uuid":"442498400","full_name":"RaistlinTAO/SparkModelHelper","owner":"RaistlinTAO","description":"Scala Library for extracting useful information from trained Spark Model (DecisionTreeClassificationModel)","archived":false,"fork":false,"pushed_at":"2023-05-18T05:10:03.000Z","size":413,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T03:16:25.664Z","etag":null,"topics":["decision-trees","decisiontreeclassifier","evaluation","fastjson","feature-extraction","scala","spark","sparkml","sparkmllib"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RaistlinTAO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-28T15:04:39.000Z","updated_at":"2024-01-14T08:12:29.000Z","dependencies_parsed_at":"2024-12-01T08:10:32.722Z","dependency_job_id":"b1159580-c1ad-4213-83cf-ef12457389b9","html_url":"https://github.com/RaistlinTAO/SparkModelHelper","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RaistlinTAO%2FSparkModelHelper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RaistlinTAO%2FSparkModelHelper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RaistlinTAO%2FSparkModelHelper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RaistlinTAO%2FSparkModelHelper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RaistlinTAO","download_url":"https://codeload.github.com/RaistlinTAO/SparkModelHelper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245200706,"owners_count":20576674,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["decision-trees","decisiontreeclassifier","evaluation","fastjson","feature-extraction","scala","spark","sparkml","sparkmllib"],"created_at":"2024-09-30T15:36:09.965Z","updated_at":"2026-05-06T07:37:58.167Z","avatar_url":"https://github.com/RaistlinTAO.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark Model Helper\n\nA helper that extracting useful information from trained Spark Model\n\n```\nFor my beloved \nWe were made for each other.\nYou are the giver\nI am the wonder\nAnd I am so blessed to be your disciple.\n```\n\n![License](https://img.shields.io/github/license/RaistlinTAO/SparkModelHelper)\n\n![Sonatype](https://img.shields.io/nexus/s/io.github.raistlintao/sparkmodelhelper_2.12?server=https%3A%2F%2Fs01.oss.sonatype.org%2F)\n![Maven Central](https://img.shields.io/maven-central/v/io.github.raistlintao/sparkmodelhelper_2.12.svg)\n\n![Code Size](https://img.shields.io/github/languages/code-size/raistlintao/SparkModelHelper)\n![Repo Size](https://img.shields.io/github/repo-size/RaistlinTAO/SparkModelHelper)\n\nHave you tired staring at the model.toDebugString() for hours and getting no clue at all? Something like this:\n\n```text\nDecisionTreeClassificationModel (uid=dtc_e933455b) of depth 5 with 341 nodes\n  If (feature 518 \u003c= 1.5)\n   If (feature 6 \u003c= 2.5)\n    If (feature 30 \u003c= 20)\n     If (feature 45 \u003c= 7)\n      If (feature 24 \u003c= 2.5)\n        ...\n      Else (feature 160 \u003e 0.99)\n      If (feature 64 \u003c= 3.5)\n       Predict: 0.0\n      Else (feature 64 \u003e 3.5)\n       Predict: 1.0\n```\nWell now you have this helper designed for HUMAN, which matters.\n\n## Contents\n\n- [Spark Model Helper](#spark-model-helper)\n    * [Usage](#usage)\n        + [**DecisionTreeClassificationModel Analysis**](#decisiontreeclassificationmodel-analysis)\n            - [_1. Get Root Feature Index from trained model_](#1-get-root-feature-index-from-trained-model)\n            - [_2. Get the JSON string from\n              DecisionTreeClassificationModel_](#2-get-the-json-string-from-decisiontreeclassificationmodel)\n            - [_3. Return an Object of Model Node-Tree_](#3-return-an-object-of-model-node-tree)\n            - [_4. Return Root to Leaf Path of Rules_](#4-return-root-to-leaf-path-of-rules)\n            - [_5. Customise the Feature_Index_](#5-customise-the-feature_index)\n        + [**DecisionTreeClassificationModel Rule Helper**](#decisiontreeclassificationmodel-rule-helper)\n            - [_Convert Rules into Python or Scala_](#1-convert-rules-into-python-or-scala)\n            - [_Evaluate new data against Rules_](#2-evaluate-new-data-against-rules)\n\n\n##### This helper was built for [Scala 2.12.15](https://www.scala-lang.org/download/2.12.15.html) and [Spark 2.4.8](https://spark.apache.org/docs/2.4.8/). Also tested with the latest Scala and Spark.\n\n# Usage\n\n## **DecisionTreeClassificationModel Analysis**\n\n***\n\n### _1. Get Root Feature Index from trained model_\n\nIn automated ML, sometimes you need to retrain model due to the scalar metric as evaluation result (precision and recall\nfor instance) are not within a desired range. By using rootFeatureIndex we can change the dataframe accordingly.\n\n```scala\n  val helper = new DecisionModelHelper(model)\n  println(\"Root Feature Index: \" + helper.getRootFeature)\n```\n\n### _2. Get the JSON string from DecisionTreeClassificationModel_\n\n```scala\n  val helper = new DecisionModelHelper(model)\n  println(\"toJson: \" + helper.toJson + \"\\n\")\n```\n\nReturn beatified JSON string:\n\n```json\n    {\n      \"featureIndex\": 367,\n      \"gain\": 0.10617781879627633,\n      \"impurity\": 0.3144732024264778,\n      \"threshold\": 1.5,\n      \"nodeType\": \"internal\",\n      \"splitType\": \"continuous\",\n      \"prediction\": 0.0,\n      \"leftChild\": {\n        ....\n      \"path\": \"F(367)|0.3144732024264778|0.0|1.5|\"\n    }\n```\n\n### _3. Return an Object of Model Node-Tree_\n\n```scala\n    val nodeObj = helper.getDecisionNode\n    println(\"getDecisionNode: \" + nodeObj)\n```\n\nObject version of JSON String\n\n### _4. Return Root to Leaf Path of Rules_\n\n```scala\n    val rules = helper.getRulesList(1, 0.2)\n    rules.foreach(rule =\u003e {\n      println(\"Rule: \" + rule.mkString(\", \"))\n    })\n```\nThe above code prints:\n```text\n    Rule: F(396)|0.3144732024264778|0.0|1.5|L, F(12)|0.49791192623975383|1.0|2.5|R, F(223)|0.2998340735773348|1.0|2500000.0|R, F(20)|0.19586076183802947|1.0|3.523971665E10|L, None|0.1902980108641974|1.0|None|E\n```\nThe function returns List[List[String]], the structure of each String is\n```text\n    Feature_Index | impurity | prediction | threshold | node_type\n```\nFor example, Feature Index **45** has impurity **3.5**, prediction **1**, threshold **1.5** and the path goes **right** after this node, the string will be:\n```text\n    F(45)|3.5|1|1.5|R\n```\nThe Leaf nodes will have **\"E\"** as node_type\n\n### _5. Customise the Feature_Index_\nFeature Index is not designed for human reading, especially with large amount of columns.\nThe helper also supports customisation of Features\n```scala\n  val helper = new DecisionModelHelper(model)\n    helper.setFeatureName(\n      Map(0 -\u003e \"UserID\", 1 -\u003e \"UserCity\", 2 -\u003e \"Salary\" ... )\n    )\n```\nThe helper automatically change the F(1) into \"UserCity\" upon called setFeatureName(Map[Int, String])\n```text\n    F(1)|3.5|1|1.5|R\n```\nwill output as \n```text\n    UserCity|3.5|1|1.5|R\n```\n\n## **DecisionTreeClassificationModel Rule Helper**\n\n### _1. Convert Rules into Python or Scala_\n```scala\n    rules.foreach(rule =\u003e {\n      println(\"Rule (Scala): \" + '\\n' + DecisionRuleHelper.getStatementFromPathList(rule, language = Language.Scala))\n      println(\"Rule (Python): \" + '\\n' + DecisionRuleHelper.getStatementFromPathList(rule, language = Language.Python))\n    })\n```\n\nYou can get code directly from this function and just Copy \u0026 Paste\n```text\nRule (Scala): \nif (F(58) \u003c= 1.5 \u0026\u0026 F(56) \u003e 2.5 \u0026\u0026 F(20) \u003e 2500000.0 \u0026\u0026 F(20) \u003c= 3.523971665E10) 1.0\nRule (Python): \nif F(58) \u003c= 1.5 and F(56) \u003e 2.5 and F(20) \u003e 2500000.0 and F(20) \u003c= 3.523971665E10: 1.0\n```\n\n### _2. Evaluate new data against Rules_\nFor testing new data against extracted rules, *Remember ONLY support non-customised Feature Name*\n\n```scala\n    //Load the data into Dataframe from CSV\n    var data = spark.read.format(\"csv\").option(\"header\", value = false).option(\"inferSchema\", value = true).load(\"sample/testing_data.csv\").na.drop(\"all\")\n    println(\"Data has been loaded into Dataframe from CSV file, or you can use existing Dataframe directly\")\n    //Convert Dataframe Schema Datatype into a ListBuffer\n    val typeList = ListBuffer[String]()\n    data.schema.foreach(schemaNode =\u003e {\n      schemaNode.dataType match {\n        case IntegerType =\u003e {\n          typeList += \"INT\"\n        }\n        case DoubleType =\u003e {\n          typeList += \"DOUBLE\"\n        }\n        case LongType =\u003e {\n          typeList += \"LONG\"\n        }\n        case _ =\u003e {\n          typeList += \"DEC\"\n        }\n      }\n    })\n    //Loop Though Every Rules (Start from 1 for Easy Column Name)\n    for (ruleIndex \u003c- 1 to rules.length) {\n      val columns = data.columns\n\n      //Define the User Defined Function: Using DecisionRuleHelper.evaluateRule\n      def combineUdf = udf((row: Row) =\u003e DecisionRuleHelper.evaluateRule(row, rules(ruleIndex - 1), typeList.toList))\n\n      //Create a new column that save the evaluation result of each rows into Column\n      //If data in one row fit the rule, the result is 1, otherwise 0\n      data = data.withColumn(\"Rule\" + ruleIndex.toString, combineUdf(struct(columns.map(col): _*)))\n      println(\"Rule \" + ruleIndex + \" proceed\")\n    }\n    data.write.format(\"csv\").save(\"result/\" + System.currentTimeMillis() / 1000)\n    println(\"Result CSV Saved, or you can use data (Dataframe) directly\")\n```\n\nAfter the loop end, the data itself changed into a new dataframe with several additional columns depends on your rules \nextracted from model(1 rule for 1 column). \nEach additional column represents result of evaluation of current row.\n\nFor example, if row 1 fit Rule 1: \n```\n(F(58) \u003c= 1.5 \u0026\u0026 F(56) \u003e 2.5 \u0026\u0026 F(20) \u003e 2500000.0 \u0026\u0026 F(20) \u003c= 3.523971665E10) 1.0\n```\nYou will get 1 otherwise 0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraistlintao%2Fsparkmodelhelper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraistlintao%2Fsparkmodelhelper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraistlintao%2Fsparkmodelhelper/lists"}