{"id":19512486,"url":"https://github.com/vmchale/emd","last_synced_at":"2025-04-26T04:31:39.740Z","repository":{"id":128760953,"uuid":"64163062","full_name":"vmchale/EMD","owner":"vmchale","description":"Earth mover's distance on Nvidia GPUS","archived":false,"fork":false,"pushed_at":"2016-09-10T02:48:26.000Z","size":14015,"stargazers_count":15,"open_issues_count":2,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-04T08:01:52.337Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vmchale.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-07-25T19:45:24.000Z","updated_at":"2023-09-25T01:40:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"4f7ca5a1-569f-40d9-a071-980cff2eeeea","html_url":"https://github.com/vmchale/EMD","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmchale%2FEMD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmchale%2FEMD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmchale%2FEMD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmchale%2FEMD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vmchale","download_url":"https://codeload.github.com/vmchale/EMD/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250931022,"owners_count":21509802,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T23:26:20.851Z","updated_at":"2025-04-26T04:31:39.734Z","avatar_url":"https://github.com/vmchale.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"#EMD\nEarth mover's distance on Nvidia GPUs\n\n##CUDA\nCUDA and relevant NVIDIA drivers have to be installed from their site. This should be done first.\n\nThe NVIDIA website https://developer.nvidia.com/cuda-downloads has everything you need.\n\n##Building\nTo build, simply run\n```\n./build\n```\nin the directory `EMD/`.\n\nIf you are on a linux machine, this will simply copy some binaries instead of building from source (which takes quite some time due to dependencies)\n\nEMD require `devIL` (an image library) and `imagemagick` to run. If `./build` fails to install these for you, you will have to install them manually.\n\nIf './build' fails completely, jump to the Manual Build section.\n\nAfter this, the haskemd module can be used from any directory. Test it with:\n\n```\npython3 test.py\n```\n\n##Python\nThe provided python wrapper takes two (one-dimensional) numpy arrays and computes the EMD:\n```\npython3\n\u003e\u003e\u003eimport haskemd\n\u003e\u003e\u003earrs = haskemd.equalize(haskemd.sinksrand(1023),haskemd.sourcesrand(1023))\n\u003e\u003e\u003ehaskemd.emd(a[0],a[1])\n232386\n```\n`sinksrand(1024)` and `sourcesrand(1024)` just generates test data with 1024 bins, while the `equalize` function makes the distributions have equal mass each. \n\nTo try your own, consider the following example:\n```\n\u003e\u003e\u003eimport haskemd\n\u003e\u003e\u003eimport numpy as np\n\u003e\u003e\u003ea1 = np.array([1,2,3,4])\n\u003e\u003e\u003ea2 = np.array([4,3,2,1])\n\u003e\u003e\u003ehaskemd.emd(a1, a2)\n6\n```\nYou might be prompted for the sudo password while computing the EMD. This is because of a bug upstream in the code for Haskell Accelerate. \n\n##Ground distance\nIn our case, the \"ground distance\" is the hamming distance. Moreover, the matrix for the metric is automatically generated. This means that an array like\n```\nnp.array([0.25,0.5])\n```\nmust have the correct values in the correct places. d\u003csub\u003eij\u003c/sub\u003e is based on the indices. Here, d\u003csub\u003e01\u003c/sub\u003e would mean the distance between the bin containing 0.25 and the bin containing 0.5. In general, we have d\u003csub\u003eij\u003c/sub\u003e=hammingDistance i j\n\n##Hamming distance\nTo get the hamming distance from a integer indices, say 3 and 7, we first convert to binary to get 011 and 111. Then we can compute the hamming distance to get 1 (in this case). Hence d\u003csub\u003e37\u003c/sub\u003e=1\n\n##Stored matrices\nAfter the matrix for a metric is computed, it is stored in the `data/` directory. This means that if you compute an EMD between distributions with 8192 bins, every subsequent calculation between distributions with 8192 bins will be faster. As the number of bins gets larger, it becomes necessary to store these matrices in order to guarantee quick runtime. \n\n##Computers without an NVIDIA GPU\nIf you want to run the code on your CPU, you can can edit the code in `src/Edmonds.hs` Open it up and the first lines should look like this:\n```\n{-# LANGUAGE FlexibleContexts #-}\n\nmodule Edmonds\n         (\n         exec) where\n\nimport Data.Array.Accelerate as A\nimport Data.Array.Accelerate.IO\nimport Data.Array.Accelerate.CUDA\n--import Data.Array.Accelerate.Interpreter\nimport System.Environment\nimport System.IO.Unsafe (unsafePerformIO)\n```\nComment out the line with `Data.Array.Accelerate.CUDA` and uncomment the line with `Data.Array.Accelerate.Interpreter` Your file should now look like this:\n```\n{-# LANGUAGE FlexibleContexts #-}\n\nmodule Edmonds\n         (\n         exec) where\n\nimport Data.Array.Accelerate as A\nimport Data.Array.Accelerate.IO\n--import Data.Array.Accelerate.CUDA\nimport Data.Array.Accelerate.Interpreter\nimport System.Environment\nimport System.IO.Unsafe (unsafePerformIO)\n```\nSave the file and run `stack install` to build and install it. It will be reasonably fast for up to 1000 bins, and after that it will be noticeably slower than the GPU \n\n##Manual Build\nThere are three parts to building manually: downloading dependencies, building the Haskell code, and installing the python module. \n\n###Dependencies\nThe only outside dependencies are: devIL (an image processing library) and imagemagick. The website for devIL is http://openil.sourceforge.net/ and the website for imagemagick is http://www.imagemagick.org/script/binary-releases.php\n\n###Get Stack\nStack is the tool that buils haskell. To install `stack` simply type\n```\nwget -qO- https://get.haskellstack.org/ | sh\n```\nor go to haskellstack.org for help if that fails.\n\n###Building the Haskell\nTo build the Haskell code, type\n```\nstack setup \u0026\u0026 stack install\n```\nin the directory `EMD/`. This may take a long time.\n\n###Installing the python module\nTo install the python module, simply `cd` into `haskemd/` and then type\n```\nsudo python3 setup.py install\n```\nAt this point, everything should be working, so run `python3 test.py` to make sure this is the case.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmchale%2Femd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvmchale%2Femd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmchale%2Femd/lists"}