{"id":19031483,"url":"https://github.com/plandes/clj-ml-dataset","last_synced_at":"2026-05-17T11:31:21.122Z","repository":{"id":80107484,"uuid":"64146176","full_name":"plandes/clj-ml-dataset","owner":"plandes","description":"Generate, split into folds or train/test and cache a dataset","archived":false,"fork":false,"pushed_at":"2018-11-25T19:01:42.000Z","size":121,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-09-02T16:46:55.462Z","etag":null,"topics":["clojure","dataset","elasticsearch","machine-learning"],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/plandes.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-07-25T15:27:27.000Z","updated_at":"2018-12-16T23:23:23.000Z","dependencies_parsed_at":null,"dependency_job_id":"e7e6b9b0-bef8-4a7c-b167-93d47df4f50e","html_url":"https://github.com/plandes/clj-ml-dataset","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/plandes/clj-ml-dataset","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plandes%2Fclj-ml-dataset","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plandes%2Fclj-ml-dataset/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plandes%2Fclj-ml-dataset/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plandes%2Fclj-ml-dataset/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/plandes","download_url":"https://codeload.github.com/plandes/clj-ml-dataset/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plandes%2Fclj-ml-dataset/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33136663,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-17T09:28:26.183Z","status":"ssl_error","status_checked_at":"2026-05-17T09:27:52.702Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","dataset","elasticsearch","machine-learning"],"created_at":"2024-11-08T21:23:46.930Z","updated_at":"2026-05-17T11:31:21.102Z","avatar_url":"https://github.com/plandes.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Generate, Split Into Folds or Train/Test and Cache a Dataset\n\n[![Travis CI Build Status][travis-badge]][travis-link]\n\n  [travis-link]: https://travis-ci.org/plandes/clj-ml-dataset\n  [travis-badge]: https://travis-ci.org/plandes/clj-ml-dataset.svg?branch=master\n\nThis is a small simple library that automates the parsing and caching of the\nparsed utterances in [Elasticsearch].\n\nThis library can be used to create an dataset but was written with natural\nlanguage processing problems in mind.  When creating machine learning models\nfor use with NLP you often you need to parse a training set data over and over.\nThe output of the parsing become input features to your model, but the parsing\nstep can take a while and if the parsing pipeline changes it has to be\nrepeated.\n\nThis library is designed to make this process less painful.\n\nFeatures:\n\n* Create training/test split datasets.\n* Store dataset in ElasticSearch with any data structure.\n* Supports stratification by class label.\n* Sort datasets by a split ratio, or by folds (useful for cross-fold validation).\n* Provides dataset statistics, per class spreadsheet dataset creation and other\n  metrics.\n* Stores optional class label and ID (unique created by ElasticSearch if\ndesired).\n* Integrates with\n  the [machine learning framework](https://github.com/plandes/clj-ml-model).\n\n\n\u003c!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc --\u003e\n## Table of Contents\n\n- [Obtaining](#obtaining)\n- [Documentation](#documentation)\n- [Setup](#setup)\n- [Example](#example)\n- [Usage](#usage)\n    - [Write a Corpus Access Namespace](#write-a-corpus-access-namespace)\n    - [Using on the REPL](#using-on-the-repl)\n- [File System Based Data Store](#file-system-based-data-store)\n- [Future Enhancements](#future-enhancements)\n- [Known Bugs](#known-bugs)\n- [Building](#building)\n- [Changelog](#changelog)\n- [License](#license)\n\n\u003c!-- markdown-toc end --\u003e\n\n\n## Obtaining\n\nIn your `project.clj` file, add:\n\n[![Clojars Project](https://clojars.org/com.zensols.ml/dataset/latest-version.svg)](https://clojars.org/com.zensols.ml/dataset/)\n\n\n## Documentation\n\nAPI [documentation](https://plandes.github.io/clj-ml-dataset/codox/index.html).\n\n\n## Setup\n\n**Note**: [ElasticSearch] is no longer necessary as there's a\nnew [file sysetm](#file-system-based-data-store) based data store.\n\nIf you don't have an ElasticSearch instance handy I recommend you use the\ndocker image and SSH tunnel the port so you don't have to configure that\nseparately.  Do do this:\n\n1. Install [docker](https://docs.docker.com/engine/installation/)\n2. Start the machine (you might need to run initial *getting started quickly*\n   terminal program): ```bash docker machine start```\n3. Start the Elasticsearch image (for the first time you'll have to wait for\n   the image to download): ```bash docker-compose up -d```\n4. Docker runs a virtual machine to host an image, so you to avoid\n   (re)configuring a dynamic IP address port forward/tunnel it instead: ```base\n   src/bin/docker-tunnel.sh```\n5. Start the repl to your example program: ```bash lein repl```\n\n\n## Example\n\nSee the [example repo](https://github.com/plandes/clj-example-nlp-ml) that\nillustrates how to use this library and contains the code from where these\nexamples originate.  It's highly recommended to clone it and follow along as\nyou peruse this README.\n\n\n## Usage\n\nFirst create a namespace to use as your *database* library.\n\n\n### Write a Corpus Access Namespace\n\n```clojure\n(ns zensols.example.anon-db\n  (:require [clojure.java.io :as io]\n            [clojure.tools.logging :as log])\n  (:require [zensols.actioncli.dynamic :refer (dyn-init-var) :as dyn]\n            [zensols.actioncli.log4j2 :as lu]\n            [zensols.nlparse.parse :as p]\n            [zensols.dataset.db :as db :refer (with-connection)]))\n\n(defn- parse-utterances [add-fn]\n  (doseq [file [(io/file \"answers\")\n                (io/file \"questions\")]]\n    (with-open [reader (io/reader file)]\n      (-\u003e\u003e reader\n           (line-seq)\n           (map p/parse)\n           (map #(add-fn % (:class-label (.getName file))))\n           doall))))\n\n(defn- connection []\n  (swap! conn-inst #(or % (db/elasticsearch-connection\n                           \"example\"\n                           :create-instances-fn parse-utterances))))\n\n(defn- load-corpora []\n  (with-connection (connection)\n    (db/instances-load)))\n\n(defn anons []\n  (with-connection (connection)\n    (db/instances)))\n```\n\nAlso see the [test case](test/zensols/dataset/thaw_test.clj).\n\n\n### Using on the REPL\n\nNow you only need to load the corpora once, then you can get it back and it\ncaches in memory on the first read access:\n```clojure\nuser\u003e (require '[zensols.example.anon-db])\nuser\u003e (load-corpora)\nuser\u003e (-\u003e\u003e (anons) first)\n=\u003e {:class-label answer, :annotation {:text ...\nuser\u003e (count (anons))\n100\n=\u003e (with-connection (connection) (divide-by-set 0.75))\n=\u003e (count (anons))\n75\n=\u003e (count (anons :set-type :test))\n25\n```\n\n## File System Based Data Store\n\nThis library now has a way to unpersist data from JSON text in\nthe\n[zensols.dataset.thaw](https://plandes.github.io/clj-ml-dataset/codox/zensols.dataset.thaw.html) namespace.\nThis can either be program/hand generated or you can \"serialize\" this to disk\nusing\n[`freeze-dataset`](https://plandes.github.io/clj-ml-dataset/codox/zensols.dataset.db.html#freeze-dataset).\n\nSee the [test case](test/zensols/dataset/thaw_test.clj).\n\n\n## Future Enhancements\n\nWhat's nneded is a all\nthe\n[zensols.dataset.db/*](https://plandes.github.io/clj-ml-dataset/codox/zensols.dataset.db.html) functions\nto be multi-method so you don't need to switch between namespaces to get\ndifferent data sources (i.e. [ElasticSearch] vs file system).  It's possible to\nrefactor these namespaces so that the function is only written once in some\nplaces (i.e. `instance-by-id`) and multi-methods for the rest based on\nconnection type.\n\n\n## Building\n\nTo build from source, do the folling:\n\n- Install [Leiningen](http://leiningen.org) (this is just a script)\n- Install [GNU make](https://www.gnu.org/software/make/)\n- Install [Git](https://git-scm.com)\n- Download the source: `git clone --recurse-submodules https://github.com/plandes/clj-ml-dataset \u0026\u0026 cd clj-ml-dataset`\n\n\n## Changelog\n\nAn extensive changelog is available [here](CHANGELOG.md).\n\n\n## License\n\nMIT License\n\nCopyright (c) 2016 - 2018 Paul Landes\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n\n\n\u003c!-- links --\u003e\n[Elasticsearch]: https://www.elastic.co\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplandes%2Fclj-ml-dataset","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fplandes%2Fclj-ml-dataset","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplandes%2Fclj-ml-dataset/lists"}