{"id":15145744,"url":"https://github.com/soundcloud/spdt","last_synced_at":"2026-02-16T09:34:32.741Z","repository":{"id":13060707,"uuid":"15741081","full_name":"soundcloud/spdt","owner":"soundcloud","description":"Streaming Parallel Decision Tree","archived":false,"fork":false,"pushed_at":"2024-02-05T19:49:01.000Z","size":304,"stargazers_count":54,"open_issues_count":4,"forks_count":11,"subscribers_count":155,"default_branch":"master","last_synced_at":"2025-01-29T21:24:20.720Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/soundcloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2014-01-08T16:30:40.000Z","updated_at":"2024-08-09T10:58:25.000Z","dependencies_parsed_at":"2022-09-23T10:41:54.715Z","dependency_job_id":"e8b0e227-0c53-4925-b161-2dc84d8129b8","html_url":"https://github.com/soundcloud/spdt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Fspdt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Fspdt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Fspdt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soundcloud%2Fspdt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/soundcloud","download_url":"https://codeload.github.com/soundcloud/spdt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237172118,"owners_count":19266614,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-26T11:42:25.090Z","updated_at":"2025-10-15T00:35:41.740Z","avatar_url":"https://github.com/soundcloud.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"Streaming Parallel Decision Tree\n================================\n\n![tree!](https://github.com/soundcloud/SPDT/blob/master/config/tree.png?raw=true)\n\nA streaming parallel decision tree (SPDT) enables parallelized training of a [decision tree classifier](https://en.wikipedia.org/wiki/Decision_tree_learning) and the ability to make updates to this tree with streaming data. Basic MDL-based pruning is also implemented to prevent overfitting.\n\nThe SPDT algorithm is described by [Ben-Haim and Tom-Tov 2010](http://jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf?origin=publication_detail).\n\nMinimum Description Length (MDL) pruning is desribed by [Mehta et. al 1995](http://www.almaden.ibm.com/cs/projects/iis/hdb/Publications/papers/kdd95_mdl.pdf).\n\n\n## Command-line interface\n\nSPDT has a command-line interface (CLI) for classification experiments and producing models. For an overview of the CLI options, use the `--help` option. A more detailed description of each option follows.\n\n### --model-path\nYou must specify a model path for model output or updates.\n\nIn the case that there is no existing model file at the path specified, a new model will be trained and written to this path with the date appended to the end of the filename.\n\nIn the case that the path points to an existing model file this model is used for one of the following operations:\n\n* if `--training-data` specifies a data file, the model will be updated with the samples inside.\n* if `--test-data` specifies a data file, the model will be used to classify the samples inside.\n* if `--print` specifies a feature map file, a `dot` file will be produced using the feature map.\n\n### --class-labels\nSpecify the observable class labels in the dataset in a comma-separated list. SPDT defaults to binary classification and adapts automatically to the sets `0,1` or `-1,1`.\n\n### --error-weight\nSet a weight on classification errors used during MDL pruning. Higher weights penalize errors more than lower weights. Higher weights result in more complicated trees than those resulting from lower weights.\n\n### --real-features, --boolean-features\nBy default, SPDT assumes all features are real-valued. If your dataset is comprised solely of boolean variables, you must enable them in SPDT:\n\n```\n./bin/spdt --boolean-features \u003cother_opts\u003e\n```\n\nIf your dataset contains a mix of boolean and real-valued features, you must specify which are real-valued in a comma-separated list and enable boolean features:\n\n```\n./bin/spdt --boolean-features --real-features 0,1,2 \u003cother_opts\u003e\n```\n\n### --impurity-limit\nSpecify the maximum class [entropy](https://en.wikipedia.org/wiki/Entropy_\\(information_theory\\)) allowable at leaf nodes. Decision trees continue to split samples at a node until the impurity criteria, which is a threshold on the class entropy at that node, is satisfied. By default, SPDT adds decisions until the training data is completely classified or further separation of the classes is impossible based on the features in the dataset.\n\n### --min-delta\nSpecify the minimum change in entropy neccessary for adding a new decision at a leaf node in the tree. By default, any decrease in entropy results in a new decision.\n\n### --min-feature-frequency\nSpecify a minimum relative frequency of boolean features before they are added to the model. Underrepresented boolean features might be a computational burden if too many exist in the dataset. Furthermore, they might not improve classification performance. The minimum feature frequency enables you to filter these features out.\n\n### --training-iterations\nSpecify the maximum number of times that the algorithm can attempt to expand leaf nodes into new decisions. By setting the training iterations, you also set a limit on the maximum depth of the tree.\n\n### --print\nSpecify a feature-map file for generating a `dot` file with a graph that represents a trained model. In the resulting graph, nodes are labeled with a period-separated concatenation of the node's ID, feature, and class label. For example:\n\n![dot!](https://github.com/soundcloud/SPDT/blob/master/config/dottree.png?raw=true)\n\nTo generate the preceding graph, first provide a `tsv` feature-map file that maps feature indexes from the `LIBSVM` sparse data format to strings:\n\n```\n$ head -n 3 feature_map.tsv\n1       Regular_insulin_dose\n2       NPH_insulin_dose\n3       UltraLente_insulin_dose\n```\n\nNext, call spdt with the print option:\n\n```\n$ ./bin/spdt --model-path diabetes_2015-01-20_143112.spdt --print feature_map.tsv\nINFO Loading model: diabetes_2015-01-20_143112.spdt\nINFO dotfile written to ./spdt.dot\n$\n```\n\nLastly, render the image with the `dot` program:\n\n```\n$ dot -Tpng -o ~/Desktop/dottree.png ./spdt.dot\n```\n\n### Data format\n\nThe SPDT CLI accepts training and testing data in [LIBSVM format](https://stats.stackexchange.com/questions/61328/libsvm-data-format):\n\n```\n\u003cclassLabel: Int\u003e \u003cfeature1: Int\u003e:\u003cvalue1: Double\u003e \u003cfeature2: Int\u003e:\u003cvalue2: Double\u003e ...\n```\n\nThe following example specifies a sample from the class `1`. This sample has features `0`, `1`, and `6` with corresponding values `0.5`, `0.75`, and `0.9`:\n\n```\n+1 0:0.5 1:0.75 6:0.9\n```\n\n## Serving layer\n\nSPDT provides a basic HTTP API that enables classification and model updates. By default, models are loaded from and saved to [HDFS](https://en.wikipedia.org/wiki/Apache_Hadoop#HDFS).\n\n### Running\nThe serving layer loads the most recent model from the HDFS directory that is specified by the environment variable `SPDT_DIRECTORY`. New versions of the model that result from requests to the `/update` endpoint are saved in this directory as snapshots.\n\nYou must also specify the port for the API to serve on using `WEB_PORT` and the base url for a web HDFS service for model storage with `WEB_HDFS`.\n\n```bash\nSPDT_DIRECTORY='/tmp/dir' WEB_HDFS_BASE_URL='http://localhost/webhdfs/v1' WEB_PORT=5000 ./bin/serve\n```\n\n### /classify\nTo classify a sample, send a POST request with a sample represented in JSON to the `/classify` endpoint. Format the sample JSON as follows:\n\n```\n{\"features\":[\u003cfeature1: Int\u003e,\u003cfeature2: Int\u003e,...],\"values\":[\u003cvalue1: Double\u003e, \u003cvalue2: Double\u003e,...]}\n```\n\nThe following sample has features `1` and `2` with corresponding values `0.1` and `0.2`:\n\n```\n{\"features\":[1,2],\"values\":[0.1,0.2]}\n```\n\nYou can use `curl` to see how the `classify` request works:\n\n```\n$ SAMPLE='{\"features\":[1,2],\"values\":[0.1,0.2]}'\n$ curl -H \"Content-Type: application/json\" -d \"$SAMPLE\" -XPOST http://localhost:5000/classify ; echo\n{\"endpoint\":\"/classify\",\"label\":0}\n$\n```\n\n### /update\nTo update the model, send samples in a POST request to the `/update` endpoint. Format the request JSON as follows:\n\n```\n{\"samples\":[\u003csample1: Sample\u003e, \u003csample2: Sample\u003e,...]}\n```\n\nFormat each sample as follows. Note that a `label` field has been added:\n\n```\n{\"label\":\u003clabel: Int\u003e,\"features\":[\u003cfeature1: Int\u003e,\u003cfeature2: Int\u003e,...],\"values\":[\u003cvalue1: Double\u003e, \u003cvalue2: Double\u003e,...]}\n```\n\nA complete example follows:\n\n```\n{\"samples\":[{\"label\":1,\"features\":[1,2],\"values\":[0.1,0.2]},{\"label\":0,\"features\":[3,4],\"values\":[0.3,0.4]}]}\n```\n\nYou can use `curl` to see how the `update` request works:\n\n```\n$ SAMPLE='{\"samples\":[{\"features\":[1,2],\"values\":[0.1,0.2],\"label\":1},{\"features\":[3,4],\"values\":[0.3,0.4],\"label\":0}]}'\n$ curl -H \"Content-Type: application/json\" -d \"$SAMPLE\" -XPOST http://localhost:5000/update ; echo\n{\"endpoint\":\"/update\",\"num_samples\":2,\"updateId\":0}\n$\n```\n\nThe resulting model is saved to HDFS in the directory that you specified with the `SPDT_DIRECTORY` environment variable. The current date is automatically appended to each model's filename. This enables any new instance of the serving layer to load the most recent model.\n\n## Copyright\n\nCopyright (c) 2013 [SoundCloud Ltd.](http://soundcloud.com) | [Trust, Safety\n\u0026 Security Team](mailto:sketchy@soundcloud.com).\n\nSee the [LICENSE](LICENSE) file for details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoundcloud%2Fspdt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoundcloud%2Fspdt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoundcloud%2Fspdt/lists"}