{"id":24431842,"url":"https://github.com/flashxio/knor","last_synced_at":"2025-11-10T02:04:10.939Z","repository":{"id":62574415,"uuid":"68330180","full_name":"flashxio/knor","owner":"flashxio","description":"A repo to allow validation of performance results in the knor paper and provide a fast, scalable k-means implementation.","archived":false,"fork":false,"pushed_at":"2020-03-31T07:02:21.000Z","size":7299,"stargazers_count":14,"open_issues_count":6,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-06-24T02:41:36.612Z","etag":null,"topics":["algorithm","cluster","distributed-computing","external-memory","kmeans-clustering","numa","streaming"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flashxio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-09-15T20:48:15.000Z","updated_at":"2025-04-29T20:59:51.000Z","dependencies_parsed_at":"2022-11-03T17:30:32.282Z","dependency_job_id":null,"html_url":"https://github.com/flashxio/knor","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/flashxio/knor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashxio%2Fknor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashxio%2Fknor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashxio%2Fknor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashxio%2Fknor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flashxio","download_url":"https://codeload.github.com/flashxio/knor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashxio%2Fknor/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263568040,"owners_count":23481604,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["algorithm","cluster","distributed-computing","external-memory","kmeans-clustering","numa","streaming"],"created_at":"2025-01-20T15:22:07.270Z","updated_at":"2025-11-10T02:04:10.907Z","avatar_url":"https://github.com/flashxio.png","language":"C++","readme":"[![Build\nStatus](https://travis-ci.org/flashxio/knor.svg?branch=master)](https://travis-ci.org/flashxio/knor)\n\n![logo](https://docs.google.com/drawings/d/1JRW7oklR9yem5-w4Lz4mF2-AM8L0e_yH9Zc-HHJMFeU/pub?w=280\u0026h=360)\n# knor\n\nThe k-means NUMA Optimized Routine library or **knor** is a\nhighly optimized and fast library for computing k-means in\nparallel with accelerations for Non-Uniform Memory Access (NUMA) architectures.\n\n*knor* can perform at **10-100 times** the speed of popular packages like Spark's MLlib, Dato (GraphLab) and H\u003csup\u003e2\u003c/sup\u003eO.\n\n*knor* can operate:\n\n1. On a single machine with all data in-memory -- **knori**.\n2. In a cluster in distributed memory -- **knord**.\n3. On a single machine with some data in-memory and the rest on disk\n(Semi-External Memory) -- **knors**.\n\nThese modules are incarnations of algorithms in\n[our publication](https://arxiv.org/abs/1606.08905) which will appear in the\nannuls of HPDC 2017.\n\nBy default *knor* can use a **scalable** adaption of\n[Elkan's](http://users.cecs.anu.edu.au/~daa/courses/GSAC6017/kmeansicml03.pdf)\nalgorithm that *can* drastically reduce the number of distance computations\nrequired for k-means without the memory bloat traditionally associated with\nElkan's algorithm.\n\n# Python\n\nWe expose the **knori** to users through `Python`. We support Linux and Mac OSX.\n\nFor instructions on how to install the `Python` package please see the\n[Python repository](https://github.com/flashxio/knorPy).\n\nIn most cases all that is necessary is:\n\n```\npip install knor\n```\n\n# R\n\nWe expose the **knori** to users through `R`. We support Linux, Mac OSX, and **Windows**.\n\nFor instructions on how to install the `R` package please see the\n[R repository](https://github.com/flashxio/knorR).\n\nIn most cases all that is necessary is:\n\n```\ninstall.packages(\"knor\")\n```\n\n## knor's backbone\n\n**knor** relies on the following:\n\n- [p-threads](https://computing.llnl.gov/tutorials/pthreads/) for\nmultithreading.\n- [numa](https://linux.die.net/man/3/numa) for NUMA optimization.\n- [FlashGraph](https://github.com/flashxio/FlashX) for a semi-external memory\nvertex-centric interface.\n\n## System Requirements\n\n- Linux or Mac OS 10.*\n- At least **4 (GB) of RAM**\n- Administrative privileges\n\n## Installation\n\nThe following is Tested on **Ubuntu 14.04 - 16.04**. We require a fairly\nmodern compiler to take advantage of compile time optimizations:\n\n### Native Auto-Install\n\nWe assume users have [`git`](https://git-scm.com/) already installed.\n\n```\ngit clone --recursive https://github.com/flashxio/knor.git\ncd knor\n./bootstrap.sh\n```\n\n### Docker -- Run in a Container\n\nUsers can also choose to use a Docker container to run *knor*.\n\n**Mac Docker Users NOTE:**\n\n* You must use the `-O` flag for *knori* and *knord* to use\nOpenMP due to a lack of hardware support for our optimizations.\n* *knors* for Mac is unsupported due to the lack of support for low-level\ninterfaces.\n\n#### Quick docker image (Stable release)\n\nUsers can get a version that is stable, but possibly not up to date as follows:\n\n```\ndocker run -i -t flashxio/knor\n```\n\nIf you already have the *knor* docker image then updating to the newest stable\nrelease can be done as follows:\n\n```\ndocker pull flashxio/knor\n```\n\n#### Manual build (Bleeding edge release)\n\nUsers may also obtain a bleeding-edge up to date stable docker image by\nbuilding the image themselves. We assume your\nOS has [`curl`](https://linux.die.net/man/1/curl) and\n[`docker`](https://docs.docker.com/engine/installation/) already installed.\n\n```\ncurl -O https://raw.githubusercontent.com/flashxio/knor/master/Dockerfile\ndocker build -t knor .\ndocker run -i -t knor\n```\n\n### Usage (Running *knor*)\n\nAssume the following:\n\n- `k` is the number of clusters\n- `dim` is the dimensionality of the features\n- `nsamples` is the number of samples\n- `$KNOR_HOME` is the directory created when *knor* was cloned from Github.\n- `nproc` is the number of MPI processes to use for *knord*.\n- `nthread` is the number of threads per process. For *knori* this is the total\nnumber of threads of execution. **NOTE:** For *knord* the total number of threads of\nexecutions is `nproc * nthread`.\n\nTo run modules from the `$KNOR_HOME` directory, you may do the following:\n\n### Test Data\n\nWe maintain a very small dataset for testing:\n[`$KNOR_HOME/test-data/matrix_r50_c5_rrw.bin`](https://github.com/flashxio/knor/blob/master/test-data/matrix_r50_c5_rrw.bin?raw=true).\n\nThis dataset has:\n\n- `nsamples` = `50`\n- `dim` = `5`\n\n**NOTE: knor will fail if you provide more units of parallelism than the\n`nsamples` in the dataset. So use small `-t` and `-T` values like `2`**\n\n#### knori\n\nFor a help message and to see valid flags:\n\n```\nexec/knori\n```\n\nAn example of how to process with file `datafile.bin`:\n\n```\nexec/knori datafile.bin nsamples dim k \\\n    -t random \\\n    -T 2 \\\n    -i 10 \\\n    -o outdir\n```\n\n**For comparison (slower speed)** run our algorithm using\n[OpenMP](http://www.openmp.org/) as follows:\n\n```\nexec/knori datafile.bin nsamples dim k \\\n    -t random \\\n    -T nthread \\\n    -i 10 \\\n    -o outdirOMP -O\n```\n\nIt is also possible to **disable** computataion pruning i.e., using *Minimal*\ntriangle inequality algorithm by using the `-P` flag.\n\n#### knord\n\nFor a help message and to see valid flags:\n\n```\nexec/knord\n```\n\nAn example of how to process with file `datafile.bin`:\n```\nmpirun.mpich -n nproc exec/knord datafile nsamples dim k \\\n    -t random -i 10 -T nthread -o outdir\n```\n\nSee the [mpirun](https://www.open-mpi.org/doc/v2.0/man1/mpirun.1.php) list of\nflags like to allow your processes to distribute\ncorrectly across a cluster. Flags of note are:\n- `--map-by ppr:n:node`, determining how many processes are mapped to a node.\n- `--map-by ppr:n:socket`, determining how processes per socket.\n\n#### knors:\n\nFor semi-external memory use, users must configure the underlying user-space filesystem SAFS as described [here](https://github.com/flashxio/FlashX/wiki/FlashX-Quick-Start-Guide)\n\nFor a help message:\n\n```\nexec/knors\n```\n\nAn example of how to process with file\n`matrix_r50_c5_rrw.adj `:\n\n```\nexec/knors libsem/FlashX/flash-graph/conf/run_graph.txt test-data/matrix_r50_c5_rrw.adj 50 5 8\n```\n\nThe following flags can further aid to accelerate *knors* performance.\n\n```\n-r: size of the row cache in gb\n```\nSpecifies another layer within the memory hierarchy to allow for caching at the granularity of rows of data rather than a page like traditional caches. In practice a small cache of 0.5GB-1GB can accelerate the performance of 10 Million+ samples.\n\n```\n-I: row cache update interval\n```\n The row cache lazily updates users must specify how often to refresh the cache in iterations of the algorithm. Note that as iterations proceed the cache will automatically update less frequently as computation stabilizes.\n\n### Output file from *knor*\n\nThe output file obtained by using the `-o` flag within *knor* produces a\n[YAML](http://yaml.org/) file that is easily readable by any YAML reader.\n\nFor instance, within Python we can use\n[PyYAML](https://pypi.python.org/pypi/PyYAML/) which is installable via\nby `pip install pyyaml` to read the file as follows:\n\n```python\nfrom yaml import load\nwith open(\"kmeans_t.yml\", \"r\") as f:\n    kms = load(f) # Returns a python dictionary\n\nkms.keys()\n\u003e ['niter', 'dim', 'nsamples', 'cluster', 'k', 'centroids', 'size'\n```\n\nThe fields are as follows:\n\n- `niter`: The number of iterations performed.\n- `dim`: The number of attributes within the dataset.\n- `nsamples`: The number of samples within the dataset.\n- `cluster`: To which cluster, [0-k), each sample is assigned.\n- `k`: The number of clusters requested.\n- `centroids`: The `k x dim` centroids matrix with each row\n    representing a cluster's centroid.\n- `size`: The number of samples placed within each cluster.\n\n\n## Data format\n\nWe use two different formats dependent upon if you operate fully in-memory or\nuse semi-external memory.\n\n### *knori* and *knord*\n\nIn-memory and distributed memory routines (*knori*, *knord*) use row-major\nbinary matrices of type `double` i.e., 16 Bytes per entry. For example the\nmatrix:\n\n```\n1 2 3 4\n5 6 7 8\n```\n\nMust be written out in row-major form i.e., the bytes would be organized such\nthat on disk they would look like:\n\n```\n1 2 3 4 5 6 7 8\n```\n\n**NOTE:** The row-major data on disk will be **non-human readable** since it is\nin binary format.\n\n### *knors*\n\nSemi-external memory (*knors*) data is stored in row-major format with\na leading 4KB FlashGraph header. You can convert binary or text data to *knors* format using the `convert_matrix`\nutility.\n\n### Data Conversion\n\nWe provide a single threaded data conversion script to convert data from one\nformat to the other.\n\n#### Plain text (space separated values) to row-major binary\n\nWe consider a file named `example.txt` in our *knor* root directory,\n`$KNOR_HOME` with the following content:\n\n```\n1 2 3 4\n5 6 7 8\n```\n\nTo convert this file from text readable (tsv) format one would do the following:\n\n```\ncd $KNOR_HOME\nutils/convert_matrix # gives help on how to use the converter\n\n\u003e usage: ./convert_matrix in_filename in_format [text/knori/knord/knors]\\\n    out_filename out_format[text/knori/knord/knors] nsamples dim\n\nutils/convert_matrix example.txt \\\n    text \\\n    example.dat\\\n    knori\\\n    2 4\n```\n\n## Formatting data directly from other languages\n\nFor *knori* and *knord* the format can\nbe produced by any binary writing utility in any language. **NOTE: ** *knors*\nis a custom format that should be generated using the `convert_matrix` utility.\nBelow are a couple examples of writing *knori* and *knord* format from some\nhigher-level languages.\n\n### `R`\n\nThe following is an example of how to take an R data frame\nand write out to raw binary row-major format:\n\n```R\n# `mtcars` is a data frame loaded by default into R\n\nmtcars # Take a look at the data\n\nnbytes.per.elem = 8 # NOTE: Always true\nmtvec \u003c- as.vector(t(mtcars)) # Turn dataframe into matrix (`t`) then vector\nwriteBin(mtvec, \"yourfilename.bin\", size=nbytes.per.elem)\n```\n\n### `Python`\n\nThe following is an example of how to take a numpy ndarray\nand write it out to raw binary row-major format:\n\n```python\nimport numpy as np\nnp.random.seed(1) # For consistency\ndat = np.random.rand(16, 8)\ndat.shape # We have a 16 x 8 matrix\n\ntype(dat[0][0]) # Check type is double i.e. 8 bytes = 64 bits\n# dat = dat.astype(np.float64) # if data type is wrong\n\ndat.tofile(\"yourfilename.bin\")\n```\n\n### Updating *knor*\n\nTo update *knor* simply run the update script which pulls directly from\nGithub:\n\n```\ncd $KNOR_HOME\n./update.sh\n```\n\n### Points of note\n\n- Different *knor* routines **may** not produce the same result\ndependent upon which initialization routine is used, **but** every routine\nwill give the same answer over multiple runs.\n- Found a bug? Please [create an issue](https://github.com/flashxio/knor/issues/new) on Github.\n\n## Publications\n\nMhembere, D., Zheng, D., Vogelstein, J. T., Priebe, C. E., \u0026 Burns, R. (2017).\nknor: A NUMA-optimized In-memory, Distributed and Semi-external-memory k-means\nLibrary. arXiv preprint arXiv:1606.08905.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflashxio%2Fknor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflashxio%2Fknor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflashxio%2Fknor/lists"}