{"id":16721868,"url":"https://github.com/cnuernber/kmeans-mnist","last_synced_at":"2025-10-05T06:56:54.961Z","repository":{"id":66590117,"uuid":"325876110","full_name":"cnuernber/kmeans-mnist","owner":"cnuernber","description":"An example of Clojure/Julia integration.","archived":false,"fork":false,"pushed_at":"2021-01-26T15:50:36.000Z","size":30,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-12T00:09:09.284Z","etag":null,"topics":["algorithm","clojure","julia","julia-compiler","kmeans"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cnuernber.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-31T21:31:21.000Z","updated_at":"2022-09-05T14:20:50.000Z","dependencies_parsed_at":"2023-03-22T02:18:31.464Z","dependency_job_id":null,"html_url":"https://github.com/cnuernber/kmeans-mnist","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cnuernber/kmeans-mnist","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnuernber%2Fkmeans-mnist","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnuernber%2Fkmeans-mnist/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnuernber%2Fkmeans-mnist/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnuernber%2Fkmeans-mnist/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cnuernber","download_url":"https://codeload.github.com/cnuernber/kmeans-mnist/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnuernber%2Fkmeans-mnist/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278420204,"owners_count":25983812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-05T02:00:06.059Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["algorithm","clojure","julia","julia-compiler","kmeans"],"created_at":"2024-10-12T22:32:35.917Z","updated_at":"2025-10-05T06:56:54.946Z","avatar_url":"https://github.com/cnuernber.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Julia/Clojure KMeans-MNIST\n\nAn example of using using Clojure and Julia to implement KMeans to\ncreate an MNIST classifier which performs extremely well both in terms\nof accuracy and in terms of cpu time.\n\n\n## Usage\n\nIf you are on Ubuntu and using JDK 8, type\n[`source scripts/local-julia-env`](scripts/local-julia-env) to setup the\njulia environment.  If you are not on Ubuntu, then see the script and\nchange/enhance it to fit your system.\n\nFrom a julia prompt, install the StaticArrays package:\n\n```julia\njulia\u003e import(Pkg)\njulia\u003e Pkg.add(\"StaticArrays\")\n```\n\nIn the script you will notice that it exports LD_PRELOAD so that\na special java library is loaded: libjsig.so.  This allows the JVM to forward\nsignals meant for Julia to Julia but still use signals.  This is a requirement\nto run the demonstration.\n\n\nAssuming that you have the proper environment setup, go into the mnist\nnamespace and type:\n\n```clojure\nkmeans-mnist.mnist\u003e (def data (time (test-kmeans 100)))\n...(lots of logging)...\n\"Elapsed time: 6936.24221 msecs\"\n#'kmeans-mnist.mnist/data\nkmeans-mnist.mnist\u003e data\n{:accuracy 0.9577, :confusion-matrix #tech.v3.tensor\u003cint64\u003e[10 10]\n[[1932    1    9    2    1    5   12    2    7    3]\n [   1 2262    8    0    3    2    4   13    5    3]\n [   9    8 1964    8    2    2    5   21   19    4]\n [   2    0    8 1904    1   35    0    6   28   20]\n [   1    3    2    1 1864    4    5    9    7   53]\n [   5    2    2   35    4 1684   10    2   28   14]\n [  12    4    5    0    5   10 1884    0    4    1]\n [   2   13   21    6    9    2    0 1948    7   39]\n [   7    5   19   28    7   28    4    7 1824    9]\n [   3    3    4   20   53   14    1   39    9 1888]]}\n```\n\n\n## Some Highlights\n\nThe algorithm is a modified kmeans-++ where we use a cumulative summation\nfor the `++` part as opposed to a sort operation.  Next we calculate distances\ninline with finding the centroid index thus reducing algorithm steps.  We use\nJulia to implement a series of small numeric kernels that are the tight\nloops of the algorithm but we write the outer algorithm purely in Clojure.  This\nallows us to use Julia exactly where it is quite strong and use Clojure to glue\nthe pieces together.  Specifically the dense arithmetic is done in Julia while\nthe algorithmic outline and the centroid reduction is done in Clojure.\n\nThe kmeans.jl file is somewhat type hinted to allow the Julia compiler as much\nfreedom as possible to specifically optimize the code to the datatypes of the\narguments provided as well as the number of columns.\n\nI implemented a Julia threading primitive,\n[indexed-map-reduce](https://cnuernber.github.io/dtype-next/tech.v3.parallel.for.html#var-indexed-map-reduce)\nthat allows me to efficiently declare context local to a thread before iterating through the\nindexes of the parallelization.  This allows a very efficient multithreaded form\nof reduction where your reduction targets are declared on the stack and the compiler\nhas full visibility of their lifetime allowing it to keep them in CPU registers or\nto declare transitive per-thread datastructures that are never visible outside the\nalgorithm so for instance a hash map that is used simply inside the per-thread\noperation.\n\nCopying data between the two languages is optimized; as long as the datatypes\nmatch a jvm-heap or native-heap backed tensor will have an optimized pathway\nto Julia.\n\nJulia resources are never exposed to the client code; all per-kmeans-run julia objects\nare allocated within the libjulia-clj construct\n[with-stack-context](https://cnuernber.github.io/libjulia-clj/libjulia-clj.julia.html#var-with-stack-context).\n There is a significant performance penalty upon first call of the code as the Julia compiler\nhas quite a bit of work to do.\n\n\n* [Externally defined Julia code](resources/kmeans.jl)\n* [Calling Julia like Clojure](src/kmeans_mnist/jl_kmeans.clj)\n* [Simple, efficient mnist loader](src/kmeans_mnist/mnist.clj)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcnuernber%2Fkmeans-mnist","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcnuernber%2Fkmeans-mnist","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcnuernber%2Fkmeans-mnist/lists"}