{"id":13525930,"url":"https://github.com/ildefons/motokolearn","last_synced_at":"2025-04-01T06:30:47.574Z","repository":{"id":210972403,"uuid":"705622247","full_name":"ildefons/motokolearn","owner":"ildefons","description":null,"archived":false,"fork":false,"pushed_at":"2023-12-14T14:44:49.000Z","size":463,"stargazers_count":11,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-02T10:34:05.771Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Motoko","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ildefons.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-16T11:28:16.000Z","updated_at":"2024-10-31T13:36:37.000Z","dependencies_parsed_at":"2023-12-14T15:52:36.527Z","dependency_job_id":"770ae140-14e4-46f4-94b2-d5e7c5b48f9e","html_url":"https://github.com/ildefons/motokolearn","commit_stats":null,"previous_names":["ildefons/motokolearn"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ildefons%2Fmotokolearn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ildefons%2Fmotokolearn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ildefons%2Fmotokolearn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ildefons%2Fmotokolearn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ildefons","download_url":"https://codeload.github.com/ildefons/motokolearn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246596586,"owners_count":20802852,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T06:01:23.623Z","updated_at":"2025-04-01T06:30:47.010Z","avatar_url":"https://github.com/ildefons.png","language":"Motoko","funding_links":[],"categories":["Decentralized AI"],"sub_categories":["TON"],"readme":"# motokolearn\n\nMotokolearn is a Motoko package meant to facilitate on-chain training and inference of machine learning models where having a large GPU is not a requirement. \n\n### Why on-chain training and inference of machine learning models is interesting?​\n\n- Web3 services using these on-chain trained models can inherit security and verification capabilities from the underlying Internet Computer protocol ​\n\n- Overall dapp architecture can be simplified by eeliminate dependencies with external web2 providers and/or avoid using pre-compiled WASM modules of pre-trained machine learning models\n\n### Why do we care about small to mid-sized data problems not requiring a GPU?​\n\n- Small to medium size data problems of heterogenous \"tabular\" data are often better solved with ensemble of boosted trees​\n\n- From personal experience, 1) many Kaggle challenges (including those I won) are better solved with ensembles of trees; and 2) last year alone, I consulted with three medium sized startups and all projects involved data bases below 100 megabytes and none of them required the use of large neural network nor GPUs.\n\n## Package installation using github\n\n1) Install mops package manager: https://docs.mops.one/quick-start\n2) Install the Motoko Base library in your project using mops:\n```bash\nmops add base\n```\n3) Install fuzz package:\n```bash\nmops add fuzz\n```\n4) Clone this repository in the base of your project:\n```\ncd \u003cyourproject\u003e\ngit clone https://github.com/ildefons/motokolearn.git\n``` \n\n## Model training \n\n### CART (Classification And Regression Tree) classifier tree\n\n```\nimport mtkl \"../motokolearn/src/Mtklearn/Mtklearn\";\nimport data \"../motokolearn/src/Mtklearn/Datasets\";\n\nactor {\n  let seed = 123456789;\n  let nsamples: Nat = 100;\n  let alldata = data.wine_data;\n  let pos_vec = mtkl.randomSample(0, alldata.size()-1, nsamples, false, seed);\n\n  let train = mtkl.rows(pos_vec, alldata); \n  let test = mtkl.removeRows(pos_vec, alldata); \n    \n  let xcols = Iter.toArray(Iter.range(0, mtkl.transpose(train).size()-2));\n  let ycol = mtkl.transpose(train).size()-1;\n  let xtrain = mtkl.cols(xcols, train);\n  let yaux = mtkl.transpose(mtkl.cols([ycol], train))[0];\n  let ytrain = mtkl.dataMemberVectorToTextVector(yaux);\n  let xtest = mtkl.cols(xcols, test);\n  let yauxtest = mtkl.transpose(mtkl.cols([ycol], test))[0];\n  let ytest = mtkl.dataMemberVectorToTextVector(yauxtest);\n\n  switch(ytrain) {\n      case (#ok(yvec)) {\n        let y_uniques = mtkl.uniquesText(yvec);\n        let myiter = Iter.range(0, xcols.size()-1);\n        let col_ids = Iter.toArray(myiter);\n        let ret_tree = mtkl.fitClassification(xtrain, yvec, 0, y_uniques, 3, 10, col_ids, seed);\n      };\n  };\n};\n\n``` \n\n### CART regression tree\n\n```\nimport mtkl \"../motokolearn/src/Mtklearn/Mtklearn\";\nimport data \"../motokolearn/src/Mtklearn/Datasets\";\n\nactor {\n  let seed = 123456789;\n  let max_depth: Nat = 10;\n  let min_num_samples: Nat = 5;\n  let nsamples: Nat = 300;\n  let alldata = data.diabetes_data;\n  let pos_vec = mtkl.randomSample(0, alldata.size()-1, nsamples, false, seed);\n\n  let train = mtkl.rows(pos_vec, alldata); \n  let test = mtkl.removeRows(pos_vec, alldata); \n    \n  let xcols = Iter.toArray(Iter.range(0, mtkl.transpose(train).size()-2));\n  let ycol = mtkl.transpose(train).size()-1;\n  let xtrain = mtkl.cols(xcols, train);\n  let yaux = mtkl.transpose(mtkl.cols([ycol], train))[0];\n  let ytrain = mtkl.dataMemberVectorToTextVector(yaux);\n  let xtest = mtkl.cols(xcols, test);\n  let yauxtest = mtkl.transpose(mtkl.cols([ycol], test))[0];\n  let ytest = mtkl.dataMemberVectorToTextVector(yauxtest);\n\n  switch(ytrain) {\n      case (#ok(yvec)) {\n        let y_uniques = mtkl.uniquesText(yvec);\n        let myiter = Iter.range(0, xcols.size()-1);\n        let col_ids = Iter.toArray(myiter);\n        let ret_tree = mtkl.fitRegression(xtrain, yvec, 0, y_uniques, min_num_samples, max_depth, col_ids, seed);\n      };\n  };\n};\n\n``` \n\n### Random forest classifier\n\n```\nimport mtkl \"../motokolearn/src/Mtklearn/Mtklearn\";\nimport data \"../motokolearn/src/Mtklearn/Datasets\";\n\nactor {\n\nvar rf_classifier_vec: [mtkl.BinTree] = [mtkl.nilTree()];\n\npublic func doRFClassifier() {\n    let seed = 123456789; \n    let ntrees = 100;\n    let max_depth: Nat = 10;\n    let min_num_samples: Nat = 5;\n    let pct_train: Float = 0.99;\n    let nsamples: Nat = 1000;\n    let alldata = data.digit_data;\n    let pos_vec = mtkl.randomSample(0, alldata.size()-1, nsamples, false, seed);\n\n    let train = mtkl.rows(pos_vec, alldata); \n    let test = mtkl.removeRows(pos_vec, alldata); \n    \n    let xcols = Iter.toArray(Iter.range(0, mtkl.transpose(train).size()-2));\n    let ycol = mtkl.transpose(train).size()-1;\n    let xtrain = mtkl.cols(xcols, train);\n    let yaux = mtkl.transpose(mtkl.cols([ycol], train))[0];\n    let ytrain = mtkl.dataMemberVectorToTextVector(yaux);\n    let xtest = mtkl.cols(xcols, test);\n    let yauxtest = mtkl.transpose(mtkl.cols([ycol], test))[0];\n    let ytest = mtkl.dataMemberVectorToTextVector(yauxtest);\n   \n    switch(ytrain) {\n      case (#ok(yvec)) {\n        let y_uniques = mtkl.uniquesText(yvec);\n        let myiter = Iter.range(0, xcols.size()-1);\n        let col_ids = Iter.toArray(myiter);\n        var ret_tree: mtkl.BinTree = mtkl.nilTree(); \n\n        let rfreturn = await mtkl.fitRandomForestClassifier(xtrain, \n                                                            yvec, \n                                                            y_uniques, \n                                                            ntrees, \n                                                            0, \n                                                            min_num_samples, \n                                                            max_depth, \n                                                            col_ids, \n                                                            pct_train,\n                                                            seed+1);\n        switch(rfreturn) {\n          case (#ok(tree_vec)) {\n            rf_classifier_vec := tree_vec;\n          };\n          case (_) {\n            //\n          };\n        };\n      }; \n      case (_) {\n        //\n      };\n    };\n  };\n```\n\n### Random forest regression\n```\nimport mtkl \"../motokolearn/src/Mtklearn/Mtklearn\";\nimport data \"../motokolearn/src/Mtklearn/Datasets\";\n\nactor {\n\nvar rf_regression_vec: [mtkl.BinTree] = [mtkl.nilTree()];\n\npublic func doRFRegression() : async () {\n\n    let seed = 123456789; \n    let ntrees = 100;\n    let max_depth: Nat = 10;\n    let min_num_samples: Nat = 5;\n    let pct_train = 0.9;\n    let nsamples: Nat = 300;\n    let alldata = data.diabetes_data;\n    let pos_vec = mtkl.randomSample(0, alldata.size()-1, nsamples, false, seed);\n\n    let train = mtkl.rows(pos_vec, alldata); \n    let test = mtkl.removeRows(pos_vec, alldata); \n    \n    let xcols = Iter.toArray(Iter.range(0, mtkl.transpose(train).size()-2));\n    let ycol = mtkl.transpose(train).size()-1;\n    let xtrain = mtkl.cols(xcols, train);\n    let yaux = mtkl.transpose(mtkl.cols([ycol], train))[0];\n    let ytrain = mtkl.dataMemberVectorToFloatVector(yaux);\n    let xtest = mtkl.cols(xcols, test);\n    let yauxtest = mtkl.transpose(mtkl.cols([ycol], test))[0];\n    let ytest = mtkl.dataMemberVectorToFloatVector(yauxtest);\n   \n    switch(ytrain) {\n      case (#ok(yvec)) {\n        let myiter = Iter.range(0, xcols.size()-1);\n        let col_ids = Iter.toArray(myiter);\n        var ret_tree: mtkl.BinTree = mtkl.nilTree(); \n        \n        let rfreturn = await mtkl.fitRandomForestRegression(xtrain, \n                                                            yvec, \n                                                            ntrees, \n                                                            0, \n                                                            min_num_samples, \n                                                            max_depth, \n                                                            col_ids, \n                                                            pct_train,\n                                                            seed+1);\n        switch(rfreturn) {\n          case (#ok(tree_vec)) {\n            rf_regression_vec := tree_vec;\n          };\n          case (_) {\n            //\n          };\n        };\n      }; \n      case (_) {\n        //\n      };\n    };\n  };\n```\n\n## Model inference\n\n### CART classifier tree\n\n```\nlet i = 1;\nlet sample: [mtkl.dataMember] = xtest[i]; \nlet vec = mtkl.predictTreeClassification(sample, mytree);\nlet myindex = Array.indexOf\u003cFloat\u003e(mtkl.max(vec), vec, Float.equal);\nlet xindex: Nat = switch(myindex) {\n  case (?Nat) Nat; \n  case _ 10;\n};\nif (Text.equal(y_uniques[xindex], yvectest[i])) {\n  Debug.print(\"correct\"); \n}\n```\n\n### CART regression tree\n\n```\nlet i = 1;\nlet sample: [mtkl.dataMember] = xtest[i]; \nlet y_hat = mtkl.predictTreeRegression(sample, mytree)[0];\nlet sample_rmse = mtkl.rmse(y_hat, yvectest[i]);\n```\n\n### Random forest classifier\n\n```\nlet i = 1;\nlet sample: [mtkl.dataMember] = xtest[i];\nlet vec = mtkl.predictRFClassification(sample,rf_classifier_vec);\nlet myindex = Array.indexOf\u003cFloat\u003e(mtkl.max(vec), vec, Float.equal);\nlet xindex: Nat = switch(myindex) {\n  case (?Nat) Nat; \n  case _ 10;\n};\nlet text_sample = mtkl.printSample(sample);\n\nif (Text.equal(y_uniques[xindex], yvectest[i])) {\n  Debug.print(\"correct\");   \n}\n```\n\n### Random forest regression\n\n```\nlet i = 0;\nlet sample: [mtkl.dataMember] = xtest[i]; \nlet y_hat = mtkl.predictRFRegression(sample, rf_regression_vec)[0];\nlet sample_rmse = mtkl.rmse(y_hat, yvectest[i]);\n```\n\n## Example of how to instantiate a classification tree\n\n``` \nlet leftLeaf: mtkl.BinTree = ?(null, null, #symbol([0.05,0.9,0.05]), mtkl.nilTree(), mtkl.nilTree());\nlet rightLeaf: mtkl.BinTree  = ?(null, null, #symbol([0.9,0.1,0.0]), mtkl.nilTree(), mtkl.nilTree());\nlet treeRoot: mtkl.BinTree  = ?(?2, ?0.3, #symbol([0.33,0.33,0.33]), leftLeaf, rightLeaf); // tree node evaluate sample based on 2nd feature and th value 0.3\n``` \n\n## Data \n\n### Uploading custom dataset using dfx\n\n1) you need a canister method able to receive data in the right format as well as a canister state variable:\n\n```\n  var actor_data: [[mtkl.dataMember]] = [[#number(1), #number(3), #symbol(\"1\")],\n                                        [#number(2), #number(2), #symbol(\"2\")],\n                                        [#number(3), #number(3), #symbol(\"3\")],\n                                        [#number(4), #number(2), #symbol(\"4\")]];\n  \n  public func setTrainingData(data: [[mtkl.dataMember]]) : async () {\n    actor_data := data;\n  };\n``` \n\n2) You can now call this method using dfx:\n```\ndfx canister call motokolearn_backend setTrainingData '(vec { vec {variant {number=1}; variant {number=11}; variant {symbol=\"1\"};}; vec {variant {number=2}; variant {number=21}; variant {symbol=\"2\"};};  vec {variant {number=3}; variant {number=31}; variant {symbol=\"3\"};};})' \n```\n\n### Copy/paste custom dataset from your device to your canister code \n\nIn case you have a large dataset in csv format, you can use the python notebook to read and convert the dataset into a motoko compatible dataset. then you can copy/paste it into your canister code: ```\u003cyourproject\u003e/motokolearn/notebooks/sklearn_ds_generation.ipynb```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fildefons%2Fmotokolearn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fildefons%2Fmotokolearn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fildefons%2Fmotokolearn/lists"}