{"id":13700456,"url":"https://github.com/cuMF/cumf_als","last_synced_at":"2025-05-04T18:35:13.392Z","repository":{"id":78262714,"uuid":"56935872","full_name":"cuMF/cumf_als","owner":"cuMF","description":"CUDA Matrix Factorization Library with Alternating Least Square (ALS)","archived":false,"fork":false,"pushed_at":"2018-08-14T02:24:37.000Z","size":375,"stargazers_count":179,"open_issues_count":4,"forks_count":46,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-04-26T05:09:52.255Z","etag":null,"topics":["als","cuda","gpu","machine","machine-learning","matrix-factorization"],"latest_commit_sha":null,"homepage":"","language":"Cuda","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cuMF.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2016-04-23T19:00:16.000Z","updated_at":"2025-04-24T08:10:26.000Z","dependencies_parsed_at":"2023-03-12T03:54:12.456Z","dependency_job_id":null,"html_url":"https://github.com/cuMF/cumf_als","commit_stats":null,"previous_names":["wei-tan/cumf"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cuMF%2Fcumf_als","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cuMF%2Fcumf_als/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cuMF%2Fcumf_als/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cuMF%2Fcumf_als/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cuMF","download_url":"https://codeload.github.com/cuMF/cumf_als/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252383222,"owners_count":21739289,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["als","cuda","gpu","machine","machine-learning","matrix-factorization"],"created_at":"2024-08-02T20:00:56.689Z","updated_at":"2025-05-04T18:35:08.361Z","avatar_url":"https://github.com/cuMF.png","language":"Cuda","funding_links":[],"categories":["Cuda"],"sub_categories":[],"readme":"# CuMF: CUDA-Accelerated ALS on multiple GPUs.\n\n## What is matrix factorization?\n\nMatrix factorization (MF) factors a sparse rating matrix R (m by n, with N_z non-zero elements) into a m-by-f and a f-by-n matrices, as shown below.\n\n\u003cimg src=https://github.com/wei-tan/CuMF/raw/master/images/mf.png width=444 height=223 /\u003e\n \nMatrix factorization (MF) is at the core of many popular algorithms, e.g., [collaborative filtering](https://en.wikipedia.org/wiki/Collaborative_filtering), word embedding, and topic model. GPU (graphics processing units) with massive cores and high intra-chip memory bandwidth sheds light on accelerating MF much further when appropriately exploiting its architectural characteristics.\n\n## What is cuMF?\n\n**CuMF** is a CUDA-based matrix factorization library that optimizes alternate least square (ALS) method to solve very large-scale MF. CuMF uses a set of techniques to maximize the performance on single and multiple GPUs. These techniques include smart access of sparse data leveraging GPU memory hierarchy, using data parallelism in conjunction with model parallelism, minimizing the communication overhead among GPUs, and a novel topology-aware parallel reduction scheme.\n\nWith only a single machine with four Nvidia GPU cards, cuMF can be 6-10 times as fast, and 33-100 times as cost-efficient, compared with the state-of-art distributed CPU solutions. Moreover, cuMF can solve the largest matrix factorization problem ever reported yet in current literature.\n\nCuMF achieves excellent scalability and performance by innovatively applying the following techniques on GPUs:  \n\n(1) On one GPU, MF deals with sparse matrices, which makes it difficult to utilize GPU's compute power. We optimize memory access in ALS by various techniques including reducing discontiguous memory access, retaining hotspot variables in faster memory, and aggressively using registers. By this means cuMF gets closer to the roofline performance of a single GPU.\n\n(2) On multiple GPUs, we add data parallelism to ALS's inherent model parallelism. Data parallelism needs a faster reduction operation among GPUs, leading to (3).\n\n(3) We also develop an innovative topology-aware, parallel reduction method to fully leverage the bandwidth between GPUs. By this means cuMF ensures that multiple GPUs are efficiently utilized simultaneously.\n\n## Use cuMF to accelerate Spark ALS\n\nCuMF can be used standalone, or to accelerate the [ALS implementation in Spark MLlib](https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala).\n\nWe modified Spark's ml/recommendation/als.scala ([code](https://github.com/wei-tan/SparkGPU/blob/MLlib/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala)) to detect GPU and offload the ALS forming and solving to GPUs, while retain shuffling on Spark RDD.\n\n\u003cimg src=https://github.com/wei-tan/CUDA-MLlib/raw/master/als/images/spark-gpu.png width=380 height=240 /\u003e\n\nThis approach has several advantages. First, existing apps relying on mllib/ALS need no change. Second, we leverage the best of Spark (to scale-out to multiple nodes) and GPU (to scale-up in one node). Check this GitHub [project](https://github.com/wei-tan/CUDA-MLlib/tree/master/als) for more details. It is also a part of [IBM packages for Apache Spark version 2](http://www-01.ibm.com/support/docview.wss?uid=swg21983421).\n\n## Build\n\nType:\n\n    make clean build\n\nTo see debug message, such as the run-time of each step, type:\n\n    make clean debug\n\n## Input Data\n\nCuMF need training and testing rating matrices in binary format, and in CSR, CSC and COO formats. In ./data/netflix and ./data/ml10M we have already prepared (i)python scripts to download Netflix and Movielens 10M data, and preprocess them, respectively.\n\nFor Netflix data, type:\n\n    cd ./data/netflix/\n    python ./prepare_netflix_data.py\n\nNote: this can take 30+ minutes. You can download this [file](https://drive.google.com/uc?export=download\u0026id=1dR8QVRwiERth1_jeFUwakDXbEzJsqixy) from your brower, extract and put the extracted files in [./data/netflix](./data/netflix) directly.\n\nFor Movielens:\n\n    cd ./data/ml10M/\n    ipython prepare_ml10M_data.py\n\nNote: you will encounter a NaN test RMSE. Please refer to the \"Known Issues\" Section.\n\n## Run\n\nType ./main you will see the following instructions:\n\nUsage: give M, N, F, NNZ, NNZ_TEST, lambda, X_BATCH, THETA_BATCH and DATA_DIR.\n\nE.g., for netflix data set, use:\n\n\t./main 17770 480189 100 99072112 1408395 0.048 1 3 ./data/netflix/\n\t\nE.g., for movielens 10M data set, use:\n\n    ./main 71567 65133 100 9000048 1000006 0.05 1 1 ./data/ml10M/\n    \nE.g., for yahooMusic dataset, use:\n\n\t./main 1000990 624961 100 252800275 4003960 1.4 6 3 ./data/yahoo/\n\n\nPrepare the data as instructed in the previous section, before you run.\n\nNote: rank value F has to be a multiple of 10, e.g., 10, 50, 100, 200.\n\n### Large-Scale Problems\n\nFor Netflix data, you need to adjust the number of batches to solve X (movie features) and Theta (user features). When F is 100, we set X_BATCH and THETA_BATCH to 1 and 3, respectively. Check test_als.sh for the reference settings for different F values.\n\nNote: we checked these settings on Kepler, Maxwell and Pascal GPU cards where there is more than 12 GB RAM. If you have cards with small memory capacity, you need to increase X_BATCH and THETA_BATCH to run more (smaller) batches.\n\nDirectory [hugewiki](./hugewiki) contains the code to solve the much larger hugewiki data set. Read Section 4 of our [paper] (http://arxiv.org/abs/1603.03820) for more details.\n\n## Performance Optimization\n\n### Conjugate Gradient Solver\n\nCuMF offers two solvers:\n\n(1) Direct LU solver provided by cuBLAS (http://docs.nvidia.com/cuda/cublas/#cublas-lt-t-gt-getrfbatched). It requires O(n^3) computation and also the implementation on GPU is slow.\n\n(2) Conjugate gradient method (https://en.wikipedia.org/wiki/Conjugate_gradient_method). We implement our own CG kernel. \n\nYou can use the CG instead of the LU solver, by uncomment #define USE_CG in [als.cu](./als.cu).\n\n### Half Precision (FP16)\n\nThe CG solver can use FP16 to store the left-hand square matrix. Since the CG solver is memory-bound, this can further improve performance.\n\n## Known Issues\nWe are trying to improve the usability, stability and performance. Here are some known issues we are working on:\n\n(1) NaN test error. This is because in some datasets such as movielens 10M, there are users or items with no ratings in training set but some ratings in test set. To overcome this, we have defined a flag in als.cu (#define SURPASS_NAN). If SURPASS_NAN is defined, we check NaN in calculating RMSE and ignore the NaN values. Normally #define SURPASS_NAN should be commented out, as the additional check slows down the computation.\n\n(2) Multi GPU support. We have tested on very large data sets such as [SparkALS](https://databricks.com/blog/2014/07/23/scalable-collaborative-filtering-with-spark-mllib.html) and HugeWiki, on multiple GPUs on one server. We will make our multi GPU support code available soon.\n\n## Teams\n- [Wei Tan](https://github.com/wei-tan)\n- [Shiyu Chang](https://github.com/code-terminator)\n- [Liangliang Cao](https://github.com/llcao)\n\n## References\n\nMore details can be found at:\n\n* Matrix Factorization on GPUs with Memory Optimization and Approximate Computing. ICPP 2018. [arXiv] (https://arxiv.org/abs/1808.03843). \n\n* Accelerate Recommender Systems with GPUs. Nvidia ParallelForAll [blog] ( https://devblogs.nvidia.com/parallelforall/accelerate-recommender-systems-with-gpus/).\n\n* CuMF: Large-Scale Matrix Factorization on Just One Machine with GPUs. Nvidia GTC 2016 talk. [ppt](http://www.slideshare.net/tanwei/s6211-cumf-largescale-matrix-factorization-on-just-one-machine-with-gpus), [video](http://on-demand.gputechconf.com/gtc/2016/video/S6211.html)\n\n* Faster and Cheaper: Parallelizing Large-Scale Matrix Factorization on GPUs. Wei Tan, [Liangliang Cao](https://github.com/llcao), [Liana Fong](https://github.com/llfong). [HPDC 2016], Kyoto, Japan. [(arXiv)](http://arxiv.org/abs/1603.03820)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FcuMF%2Fcumf_als","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FcuMF%2Fcumf_als","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FcuMF%2Fcumf_als/lists"}