{"id":13482098,"url":"https://github.com/huawei-noah/streamDM","last_synced_at":"2025-03-27T12:32:14.890Z","repository":{"id":33394651,"uuid":"37039699","full_name":"huawei-noah/streamDM","owner":"huawei-noah","description":"Stream Data Mining Library for Spark Streaming","archived":false,"fork":false,"pushed_at":"2023-04-16T14:47:47.000Z","size":3295,"stargazers_count":491,"open_issues_count":4,"forks_count":147,"subscribers_count":67,"default_branch":"master","last_synced_at":"2024-08-01T17:31:38.933Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://streamdm.noahlab.com.hk/","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huawei-noah.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-06-08T01:28:42.000Z","updated_at":"2024-07-22T01:20:26.000Z","dependencies_parsed_at":"2022-09-07T05:41:06.659Z","dependency_job_id":"80948ee9-c1eb-4c59-bb33-b847bd2582ed","html_url":"https://github.com/huawei-noah/streamDM","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huawei-noah%2FstreamDM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huawei-noah%2FstreamDM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huawei-noah%2FstreamDM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huawei-noah%2FstreamDM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huawei-noah","download_url":"https://codeload.github.com/huawei-noah/streamDM/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222252094,"owners_count":16955965,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T17:00:58.957Z","updated_at":"2024-10-30T15:31:44.506Z","avatar_url":"https://github.com/huawei-noah.png","language":"Scala","funding_links":[],"categories":["Table of Contents","Scala","Software","数据科学"],"sub_categories":["Online Machine Learning","Modelling"],"readme":"# streamDM for Spark Streaming\r\n\r\nstreamDM is a new open source software for mining big data streams using [Spark Streaming](https://spark.apache.org/streaming/), started at [Huawei Noah's Ark\r\nLab](http://www.noahlab.com.hk/). streamDM is licensed under Apache Software License v2.0.\r\n\r\n## Big Data Stream Learning \r\n\r\nBig Data stream learning is more challenging than batch or offline learning,\r\nsince the data may not keep the same distribution over the lifetime of the\r\nstream. Moreover, each example coming in a stream can only be processed once, or\r\nthey need to be summarized with a small memory footprint, and the learning\r\nalgorithms must be very efficient. \r\n\r\n### Spark Streaming\r\n\r\n[Spark Streaming](https://spark.apache.org/streaming/) is an extension of the\r\ncore [Spark](https://spark.apache.org)  API that enables stream processing from\r\na variety of sources. Spark is a extensible and programmable framework for\r\nmassive distributed processing of datasets, called Resilient Distributed\r\nDatasets (RDD). Spark Streaming receives input data streams and divides the data\r\ninto batches, which are then processed by the Spark engine to generate the\r\nresults.\r\n\r\nSpark Streaming data is organized into a sequence of DStreams, represented\r\ninternally as a sequence of RDDs.\r\n\r\n### \u003ca name=\"methods\"\u003e\u003c/a\u003eIncluded Methods\r\n\r\nIn this current release of StreamDM v0.2, we have implemented:\r\n\r\n* [SGD Learner](http://huawei-noah.github.io/streamDM/docs/SGD.html) and [Perceptron](http://huawei-noah.github.io/streamDM/docs/SGD.html#perceptron)\r\n* [Naive Bayes](http://huawei-noah.github.io/streamDM/docs/NB.html)\r\n* [CluStream](http://huawei-noah.github.io/streamDM/docs/CluStream.html)\r\n* [Hoeffding Decision Trees](http://huawei-noah.github.io/streamDM/docs/HDT.html)\r\n* [Bagging](http://huawei-noah.github.io/streamDM/docs/Bagging.html)\r\n* [Stream KM++](http://huawei-noah.github.io/streamDM/docs/StreamKM.html)\r\n\r\nwe also implemented following [data generators](http://huawei-noah.github.io/streamDM/docs/generators.html):\r\n\r\n* HyperplaneGenerator\r\n* RandomTreeGenerator\r\n* RandomRBFGenerator\r\n* RandomRBFEventsGenerator\r\n\r\nWe have also implemented [SampleDataWriter](http://huawei-noah.github.io/streamDM/docs/SampleDataWriter.html), which can call data generators \r\nto create sample data for simulation or test.\r\n\r\nIn the next release of streamDM, we are going to add:\r\n\r\n* Classification: Random Forests\r\n* Multi-label: Hoeffding Tree ML, Random Forests ML\r\n* Frequent Itemset Miner: IncMine\r\n\r\nFor future works, we are considering:\r\n* Regression: Hoeffding Regression Tree, Bagging, Random Forests\r\n* Clustering: Clustree, DenStream\r\n* Frequent Itemset Miner: IncSecMine\r\n\r\n## Going Further\r\n\r\nFor a quick introduction to running StreamDM, refer to the [Getting\r\nStarted](http://huawei-noah.github.io/streamDM/docs/GettingStarted.html) document. The StreamDM [Programming\r\nGuide](http://huawei-noah.github.io/streamDM/docs/Programming.html) presents a detailed view of StreamDM. The full API\r\ndocumentation can be consulted [here](http://huawei-noah.github.io/streamDM/api/index.html). \r\n\r\n## Environment\r\n* Spark 2.3.2\r\n* Scala 2.11\r\n* SBT 0.13\r\n* Java 8+\r\n\r\n## Mailing lists\r\n### User support and questions mailing list:\r\n\u003ca href=\"mailto:streamdm-user@googlegroups.com\"\u003estreamdm-user@googlegroups.com\u003c/a\u003e\r\n### Development related discussions:\r\n\u003ca href=\"mailto:streamdm-dev@googlegroups.com\"\u003estreamdm-dev@googlegroups.com\u003c/a\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuawei-noah%2FstreamDM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuawei-noah%2FstreamDM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuawei-noah%2FstreamDM/lists"}