{"id":22812131,"url":"https://github.com/tailhq/dynaml","last_synced_at":"2025-04-09T21:20:59.194Z","repository":{"id":27397642,"uuid":"30874016","full_name":"tailhq/DynaML","owner":"tailhq","description":"Scala Library/REPL for Machine Learning Research","archived":false,"fork":false,"pushed_at":"2023-04-21T20:44:11.000Z","size":225741,"stargazers_count":201,"open_issues_count":25,"forks_count":51,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-10-29T22:56:22.547Z","etag":null,"topics":["classification","committee-models","gaussian-processes","kernel-methods","machine-learning","machine-learning-algorithms","machine-learning-api","regression","repl","scala","scala-library","tensorflow"],"latest_commit_sha":null,"homepage":"http://tailhq.github.io/DynaML/","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tailhq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-02-16T15:22:30.000Z","updated_at":"2024-10-07T20:47:42.000Z","dependencies_parsed_at":"2022-09-01T02:01:56.022Z","dependency_job_id":"4b01cdf1-ceb1-4b03-8b15-e5d43c39148f","html_url":"https://github.com/tailhq/DynaML","commit_stats":{"total_commits":1987,"total_committers":6,"mean_commits":331.1666666666667,"dds":0.08605938600905894,"last_synced_commit":"f5551a6dc09c10665ce6cee9da09798dded2aa60"},"previous_names":["transcendent-ai-labs/dynaml"],"tags_count":87,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tailhq%2FDynaML","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tailhq%2FDynaML/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tailhq%2FDynaML/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tailhq%2FDynaML/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tailhq","download_url":"https://codeload.github.com/tailhq/DynaML/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248112365,"owners_count":21049646,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["classification","committee-models","gaussian-processes","kernel-methods","machine-learning","machine-learning-algorithms","machine-learning-api","regression","repl","scala","scala-library","tensorflow"],"created_at":"2024-12-12T12:10:19.156Z","updated_at":"2025-04-09T21:20:59.171Z","avatar_url":"https://github.com/tailhq.png","language":"Scala","readme":"![3dplot](docs-old/images/dynaml_logo3.png)\n\n# DynaML: ML + JVM + Scala\n\n\n[![Join the chat at https://gitter.im/DynaML/Lobby](https://badges.gitter.im/DynaML/Lobby.svg)](https://gitter.im/DynaML/Lobby?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge) ![Scala CI](https://github.com/transcendent-ai-labs/DynaML/workflows/Scala%20CI/badge.svg) [![](https://jitpack.io/v/transcendent-ai-labs/DynaML.svg)](https://jitpack.io/#transcendent-ai-labs/DynaML)\n[![Coverage Status](https://coveralls.io/repos/github/transcendent-ai-labs/DynaML/badge.svg)](https://coveralls.io/github/transcendent-ai-labs/DynaML)\n\n------------------\n\n\u003cbr/\u003e\n\nDynaML is a Scala \u0026 JVM Machine Learning toolbox for research, education \u0026 industry.\n\n\u003cbr/\u003e\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003cth\u003e \u003cimg src=\"docs-old/images/plot3dsmall.jpeg\" alt=\"Plot3d\"\u003e \u003c/th\u003e \n        \u003cth\u003e \u003cimg src=\"docs-old/images/plots_small.png\" alt=\"Plot2d\"\u003e \u003c/th\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n\n------------------\n\n## Motivation\n\n\n - __Interactive.__ Don't want to create Maven/sbt project skeletons\n every time you want to try out ideas? Create and execute [scala worksheets](scripts/randomvariables.sc) \n in the DynaML shell. DynaML comes packaged with a customized version of the [Ammonite](http://ammonite.io) REPL, \n with *auto-complete*, file operations and scripting capabilities.  \n \n - __End to End.__ Create complex pre-processing pipelines with the [data pipes](https://transcendent-ai-labs.github.io/DynaML/pipes/pipes/) API, \n train models ([deep nets](scripts/cifar.sc), [gaussian processes](https://transcendent-ai-labs.github.io/DynaML/core/core_gp/), \n [linear models](https://transcendent-ai-labs.github.io/DynaML/core/core_glm/) and more), \n optimize over [hyper-parameters](https://transcendent-ai-labs.github.io/DynaML/core/core_opt_global/), \n [evaluate](https://transcendent-ai-labs.github.io/DynaML/core/core_model_evaluation/) model predictions and \n [visualise](https://transcendent-ai-labs.github.io/DynaML/core/core_graphics/) results.\n \n - __Enterprise Friendly.__ Take advantage of the JVM and Scala ecosystem, use Apache [Spark](https://spark.apache.org) \n to write scalable data analysis jobs, [Tensorflow](http://tensorflow.org) for deep learning, all in the same toolbox.\n\n------------------\n\n## Getting Started\n\n### Platform Compatibility\n\nCurrently, only *nix and OSX platforms are supported.\n\nDynaML is compatible with Scala `2.12`\n\n### Installation\n\nEasiest way to install DynaML is cloning \u0026 compiling from the [github](/) repository. Please take a look at \nthe [installation](https://transcendent-ai-labs.github.io/DynaML/installation/installation/) instructions in the \n[user guide](https://transcendent-ai-labs.github.io/DynaML/), to make sure that you have the pre-requisites \nand to configure your installation.\n\n------------------\n\n## CIFAR in under 200 lines\n\nBelow is a sample [script](scripts/cifar.sc) where we train a neural network of stacked \n[Inception](https://arxiv.org/pdf/1409.4842.pdf) cells on the [CIFAR-10](https://en.wikipedia.org/wiki/CIFAR-10)\nimage classification task.\n\n```scala\n{\n  import _root_.ammonite.ops._\n  import _root_.io.github.tailhq.dynaml.pipes.DataPipe\n  import _root_.io.github.tailhq.dynaml.tensorflow.{\n    dtflearn,\n    dtfutils,\n    dtfdata,\n    dtfpipe\n  }\n  import _root_.org.platanios.tensorflow.api._\n  import _root_.org.platanios.tensorflow.api.learn.layers.Activation\n  import _root_.org.platanios.tensorflow.data.image.CIFARLoader\n  import _root_.java.nio.file.Paths\n\n  val tempdir = home / \"tmp\"\n\n  val dataSet =\n    CIFARLoader.load(Paths.get(tempdir.toString()), CIFARLoader.CIFAR_10)\n\n  val dtf_cifar_data = dtfdata.tf_dataset(\n    dtfdata.supervised_dataset(\n      dataSet.trainImages.unstack(axis = 0),\n      dataSet.trainLabels.castTo[Long].unstack(axis = -1)\n    ),\n    dtfdata.supervised_dataset(\n      dataSet.testImages.unstack(axis = 0),\n      dataSet.testLabels.castTo[Long].unstack(axis = -1)\n    )\n  )\n\n  println(\"Building the model.\")\n\n  val relu_act =\n    DataPipe[String, Activation[Float]]((x: String) =\u003e tf.learn.ReLU[Float](x))\n\n  val architecture =\n    tf.learn.Cast[UByte, Float](\"Input/Cast\") \u003e\u003e\n      dtflearn.inception_unit[Float](channels = 3, Seq.fill(4)(10), relu_act)(\n        layer_index = 1\n      ) \u003e\u003e\n      dtflearn.inception_unit[Float](channels = 40, Seq.fill(4)(5), relu_act)(\n        layer_index = 2\n      ) \u003e\u003e\n      tf.learn.Flatten[Float](\"Layer_3/Flatten\") \u003e\u003e\n      dtflearn.feedforward[Float](256)(id = 4) \u003e\u003e\n      tf.learn.ReLU[Float](\"Layer_4/ReLU\", 0.1f) \u003e\u003e\n      dtflearn.feedforward[Float](10)(id = 5)\n\n  val loss = tf.learn.SparseSoftmaxCrossEntropy[Float, Long, Float](\n    \"Loss/CrossEntropy\"\n  ) \u003e\u003e\n    tf.learn.Mean(\"Loss/Mean\") \u003e\u003e\n    tf.learn.ScalarSummary(\"Loss/Summary\", \"Loss\")\n\n  val optimizer = tf.train.Adam(0.1f)\n\n  val cifar_model =\n    dtflearn.model[\n      Output[UByte], Output[Long], Output[Float], Float, \n      Tensor[UByte], UINT8, Shape, \n      Tensor[Long], INT64, Shape, \n      Tensor[Float], FLOAT32, Shape](\n      architecture,\n      (UINT8, dataSet.trainImages.shape(1 ::)),\n      (INT64, Shape()),\n      loss\n    )\n\n  val data_ops = dtflearn.model.data_ops[(Output[UByte], Output[Long])](\n    shuffleBuffer = 5000,\n    batchSize = 128,\n    prefetchSize = 10\n  )\n\n  val train_config = dtflearn.model.trainConfig(\n    tempdir / \"cifar_summaries\",\n    data_ops,\n    optimizer,\n    dtflearn.rel_loss_change_stop(0.05, 500),\n    Some(\n      dtflearn.model._train_hooks(\n        tempdir / \"cifar_summaries\",\n        stepRateFreq = 100,\n        summarySaveFreq = 100,\n        checkPointFreq = 100\n      )\n    )\n  )\n\n  val pattern_to_tensor =\n    DataPipe[Seq[(Tensor[UByte], Tensor[Long])], (Tensor[UByte], Tensor[Long])](\n      ds =\u003e {\n        val (xs, ys) = ds.unzip\n\n        (\n          dtfpipe.EagerStack[UByte](axis = 0).run(xs),\n          dtfpipe.EagerStack[Long](axis = 0).run(ys)\n        )\n      }\n    )\n\n  val data_handle_ops = dtflearn.model.tf_data_handle_ops[\n    (Tensor[UByte], Tensor[Long]),\n    (Tensor[UByte], Tensor[Long]),\n    Tensor[Float],\n    (Output[UByte], Output[Long])\n  ](\n    bufferSize = 500,\n    patternToTensor = Some(pattern_to_tensor),\n    concatOpO = Some(dtfpipe.EagerConcatenate[Float]())\n  )\n\n  val data_handle_ops_infer =\n    dtflearn.model.tf_data_handle_ops[Tensor[UByte], Tensor[UByte], Tensor[\n      Float\n    ], Output[UByte]](\n      bufferSize = 1000,\n      patternToTensor = Some(dtfpipe.EagerStack[UByte](axis = 0)),\n      concatOpO = Some(dtfpipe.EagerConcatenate[Float]())\n    )\n\n  cifar_model.train(\n    dtf_cifar_data.training_dataset,\n    train_config,\n    data_handle_ops\n  )\n\n  def accuracy(predictions: Tensor[Long], labels: Tensor[Long]): Float =\n    tfi\n      .equal(predictions.argmax[Long](1), labels)\n      .castTo[Float]\n      .mean()\n      .scalar\n      .asInstanceOf[Float]\n\n  val (trainingPreds, testPreds): (Tensor[Float], Tensor[Float]) = (\n    cifar_model\n      .infer_batch(\n        dtf_cifar_data.training_dataset.map(p =\u003e p._1),\n        data_handle_ops_infer\n      )\n      .left\n      .get,\n    cifar_model\n      .infer_batch(\n        dtf_cifar_data.test_dataset.map(p =\u003e p._1),\n        data_handle_ops_infer\n      )\n      .left\n      .get\n  )\n\n  val (trainAccuracy, testAccuracy) = (\n    accuracy(trainingPreds.castTo[Long], dataSet.trainLabels.castTo[Long]),\n    accuracy(testPreds.castTo[Long], dataSet.testLabels.castTo[Long])\n  )\n\n  print(\"Train accuracy = \")\n  pprint.pprintln(trainAccuracy)\n\n  print(\"Test accuracy = \")\n  pprint.pprintln(testAccuracy)\n}\n```\n\n------------------\n\n\n## Support \u0026 Community\n\n - [User guide](https://transcendent-ai-labs.github.io/DynaML/)\n - [Gitter](https://gitter.im/DynaML/Lobby?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n - [Contributing](https://github.com/transcendent-ai-labs/DynaML/blob/master/CONTRIBUTING.md)\n - [Code of Conduct](https://github.com/transcendent-ai-labs/DynaML/blob/master/CODE_OF_CONDUCT.md)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftailhq%2Fdynaml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftailhq%2Fdynaml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftailhq%2Fdynaml/lists"}