{"id":19876826,"url":"https://github.com/lewuathe/dllib","last_synced_at":"2025-05-02T12:30:31.575Z","repository":{"id":19302382,"uuid":"22539940","full_name":"Lewuathe/dllib","owner":"Lewuathe","description":"dllib is a distributed deep learning library running on Apache Spark","archived":false,"fork":false,"pushed_at":"2017-10-26T17:07:59.000Z","size":3867,"stargazers_count":32,"open_issues_count":1,"forks_count":5,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-07T01:23:05.838Z","etag":null,"topics":["deep-learning","mllib","scala","spark"],"latest_commit_sha":null,"homepage":"https://www.lewuathe.com/dllib/","language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"msharp/elixir-statistics","license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Lewuathe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-08-02T05:29:16.000Z","updated_at":"2023-10-22T10:04:55.000Z","dependencies_parsed_at":"2022-08-21T07:41:02.399Z","dependency_job_id":null,"html_url":"https://github.com/Lewuathe/dllib","commit_stats":null,"previous_names":["lewuathe/neurallib"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lewuathe%2Fdllib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lewuathe%2Fdllib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lewuathe%2Fdllib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Lewuathe%2Fdllib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Lewuathe","download_url":"https://codeload.github.com/Lewuathe/dllib/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252038077,"owners_count":21684620,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","mllib","scala","spark"],"created_at":"2024-11-12T16:34:26.000Z","updated_at":"2025-05-02T12:30:31.099Z","avatar_url":"https://github.com/Lewuathe.png","language":"CSS","funding_links":[],"categories":[],"sub_categories":[],"readme":"dllib [![Build Status](https://travis-ci.org/Lewuathe/dllib.svg?branch=master)](https://travis-ci.org/Lewuathe/dllib) [![codecov](https://codecov.io/gh/Lewuathe/dllib/branch/master/graph/badge.svg)](https://codecov.io/gh/Lewuathe/dllib)  [![GitHub tag](https://img.shields.io/github/tag/Lewuathe/dllib.svg)]() [![license](https://img.shields.io/github/license/Lewuathe/dllib.svg)](https://www.apache.org/licenses/LICENSE-2.0) [![Maven Central](https://img.shields.io/maven-central/v/com.lewuathe/dllib_2.11.svg)]()\n================\n\ndllib is a distributed deep learning framework running on Apache Spark. See more detail in [documentation](http://www.lewuathe.com/dllib/). dllib is designed to be simple and easy to use for Spark users.\n\nSince dllib has completely same interface of MLlib algorithms, libraries in MLlib can be used for feature engineering or transformation.\n\n# How to use\n\ndllib is uploaded on [Spark Packages](http://spark-packages.org/package/Lewuathe/dllib). You can use from `sperk-shell` directly.\n\n```bash\n\n$ ./bin/spark-shell --packages Lewuathe:dllib:0.0.9\n\n```\n\nIf you want use jar package to extend, you can write the configuration in your pom.xml.\n\n```\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.lewuathe\u003c/groupId\u003e\n    \u003cartifactId\u003edllib_2.11\u003c/artifactId\u003e\n    \u003cversion\u003e0.0.9\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n## Example\n\nThis is an example for classification of [MNIST](http://yann.lecun.com/exdb/mnist/) dataset. Full code can be seen [here](https://github.com/Lewuathe/dllib/blob/master/src/main/scala/com/lewuathe/dllib/example/MNISTApp.scala).\n\n```scala\nimport com.lewuathe.dllib.graph.Graph\nimport com.lewuathe.dllib.layer.{AffineLayer, ReLULayer, SoftmaxLayer}\nimport com.lewuathe.dllib.network.Network\n\n// Define the network structure as calculation graph.\nval graph = new Graph(Array(\n  new AffineLayer(100, 784),\n  new ReLULayer(100, 100),\n  new AffineLayer(10, 100),\n  new SoftmaxLayer(10, 10)\n))\n\n// Model keeps whole network parameters which should be trained.\n// Default is in-memory model.\nval model = Model(nn3Graph)\n\nval nn3 = Network(model, graph)\n\n// MultilayerPerceptron defines the optimization algorithms and hyper parameters.\nval multilayerPerceptron = new MultiLayerPerceptron(\"MNIST\", nn3)\n\n// We can pass Dataset of Spark to the network.\nval trainedModel = multilayerPerceptron.fit(df)\n\nval result = trainedModel.transform(df)\n\nresult.filter(\"label = prediction\").count()\n```\n\n# License\n\n[Apache v2](http://www.apache.org/licenses/LICENSE-2.0)\n\n# Author\n\n* Kai Sasaki([@Lewuathe](https://github.com/Lewuathe))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flewuathe%2Fdllib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flewuathe%2Fdllib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flewuathe%2Fdllib/lists"}