https://github.com/alexp11223/hadoopmapreducematrixmult
https://github.com/alexp11223/hadoopmapreducematrixmult
hadoop hadoop-mapreduce mapreduce matrix-multiplication sparse-matrix
Last synced: 6 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/alexp11223/hadoopmapreducematrixmult
- Owner: AlexP11223
- License: mit
- Created: 2018-06-01T05:53:45.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2018-06-02T11:11:48.000Z (over 7 years ago)
- Last Synced: 2025-02-06T23:31:25.822Z (8 months ago)
- Topics: hadoop, hadoop-mapreduce, mapreduce, matrix-multiplication, sparse-matrix
- Language: Kotlin
- Size: 16.6 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
Hadoop MapReduce program for sparse matrix multiplication, implemented in Kotlin. Based on [lendap.wordpress.com/2015/02/16/matrix-multiplication-with-mapreduce](https://lendap.wordpress.com/2015/02/16/matrix-multiplication-with-mapreduce/). Optimized to use custom classes for Mapper key/value instead of strings.
Also includes sequential (not Hadoop/MapReduce) version of matrix multiplication.
Input files are in COO/ijv format, with additional first column for matrix name/id (can be M or N).
Example
| | | |
| --- | --- | --- |
| 4 | 0 | 1 |
| 0 | 6 | 0 |
M,0,0,4
M,0,2,1
M,1,1,6
`tools/` folder contains Python script for generating random sparse matrices in this format.
# How to build
Requirements:
- JDK 8+.
- Maven 3+.
Run Maven **package** phase. This will download all dependencies, run JUnit tests and build JAR file.
(Maven is included in popular Java IDEs such as IntelliJ Idea or Eclipse. You can run it either via your IDE Maven plugin or from command line in separate [Maven installation](https://maven.apache.org/install.html): `mvn package`.)
`target/` folder will contain JAR file.
# Usage
The easiest way is to take `*-fat.jar` from `target/`, it includes all dependencies. Then run it as usual via `yarn jar`.