{"id":16755438,"url":"https://github.com/mxmlnkn/scaromare","last_synced_at":"2026-01-02T08:35:24.681Z","repository":{"id":78492481,"uuid":"62764644","full_name":"mxmlnkn/scaromare","owner":"mxmlnkn","description":null,"archived":false,"fork":false,"pushed_at":"2016-08-26T14:39:54.000Z","size":17018,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-22T19:12:02.636Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TeX","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mxmlnkn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-07-07T01:23:36.000Z","updated_at":"2017-11-17T06:05:11.000Z","dependencies_parsed_at":"2023-07-08T07:00:48.487Z","dependency_job_id":null,"html_url":"https://github.com/mxmlnkn/scaromare","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxmlnkn%2Fscaromare","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxmlnkn%2Fscaromare/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxmlnkn%2Fscaromare/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxmlnkn%2Fscaromare/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mxmlnkn","download_url":"https://codeload.github.com/mxmlnkn/scaromare/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243835942,"owners_count":20355611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T03:22:31.336Z","updated_at":"2026-01-02T08:35:24.639Z","avatar_url":"https://github.com/mxmlnkn.png","language":"TeX","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About this Repository\n\nThe goal of this repository are first tests into using Spark with GPUs by using Rootbeer.\n\nThe more long term goal is to hide the antics of the GPU task, so that a user can provide Rootbeer kernels and after that can use Spark like normal plus the defined kernels.\n\n\n# Starting MontePi Spark Multi GPU Test\n\n    startSpark --time=15:00:00 --nodes=2 --partition=gpu2 --cpus-per-task=2 --gres='gpu:2'\n    squeue -u $USER\n         JOBID   PARTITION NAME     ST      TIME  NODES NODELIST(REASON)\n         7660143 gpu2      start_sp R   14:49:09      2 taurusi[2093-2094]\n    makeOptsSpark=(                                                      \\\n        \"SPARK_ROOT=$HOME/spark-1.5.2-bin-hadoop2.6\"                     \\\n        \"SPARK_JAR=$SPARK_ROOT/lib/spark-assembly-1.5.2-hadoop2.6.0.jar\" \\\n        \"SPARKCORE_JAR=$SPARK_JAR\"                                       \\\n        \"SCALA_ROOT=$(dirname $(which scala))/../lib\"                    \\\n    )\n    ( cd \"$HOME/scaromare/rootbeer1/\" \u0026\u0026 ( cd csrc \u0026\u0026 ./compile_linux_x64 ) \u0026\u0026 ant clean \u0026\u0026 'rm' -f dist/Rootbeer1.jar Rootbeer.jar \u0026\u0026 ant jar \u0026\u0026 ./pack-rootbeer ) \u0026\u0026 make clean \u0026\u0026 make -B \u0026\u0026 java -jar Count.jar\n    folder=$HOME/scaromare/MontePi/multiNode/multiGpu/scala\n    make -C \"$folder\" -B ${makeOptsSpark[@]} MontePi.jar\n    sparkSubmit \"$folder/MontePi.jar\" 268435456 2 2\u003e\u00261 | sed '/ INFO /d'\n\n\n# Known Bugs\n\n  - \n  \n        Exception in thread \"main\" java.lang.NoClassDefFoundError: org/trifort/rootbeer/runtime/Kernel\n\n    It seems Rootbeer.jar wasn't merged into the fat JAR or specified in the class path. This may happen after the Rootbeer commit where it doesn't do that automatically anymore.\n\n  - \n \n        16/05/23 12:01:39 ERROR FatalExceptionHandler: Exception processing: 1 org.trifort.rootbeer.runtime.GpuEvent@5a425c28\n        java.lang.OutOfMemoryError: [FixedMemory.java]\n           currentHeapEnd / bytes currently in use: 2670592 B\n           Bytes requested to allocate            : 64 B\n           total size available in FixedMemory    : 2670592 B\n        (This happens if createContext(size) was called with an insufficient size, or CUDAContext.java:findMemorySize failed to determine the correct size automatically)\n           at org.trifort.rootbeer.runtime.FixedMemory$MemPointer.mallocWithSize(FixedMemory.java:451)\n           at org.trifort.rootbeer.runtime.FixedMemory.mallocWithSize(FixedMemory.java:306)\n           at org.trifort.rootbeer.runtime.Serializer.checkWriteCache(Serializer.java:125)\n           at org.trifort.rootbeer.runtime.Serializer.writeToHeap(Serializer.java:148)\n           at org.trifort.rootbeer.runtime.Serializer.writeToHeap(Serializer.java:49)\n           at org.trifort.rootbeer.runtime.CUDAContext.writeBlocksList(CUDAContext.java:473)\n           at org.trifort.rootbeer.runtime.CUDAContext.access$1500(CUDAContext.java:19)\n           at org.trifort.rootbeer.runtime.CUDAContext$GpuEventHandler.onEvent(CUDAContext.java:408)\n           at org.trifort.rootbeer.runtime.CUDAContext$GpuEventHandler.onEvent(CUDAContext.java:369)\n           at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)\n           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n           at java.lang.Thread.run(Thread.java:724)\n\n   =\u003e This seems to be a critical bug in the automatic memory management, especially `findMemorySize`. Try to use a manual estimate for the needed context size instead. Unfortunately I can't yet give a formula or even some estimates. It shouldn't be much more than all the member variables per Kernel times the kernel count plus some constant for the kernel code.\n\n\n# Compiling on Taurus\n\nCurrently the example is using raw make, without CMake or a configure or similar, so you will have to configure some paths manually, e.g. on my Taurus:\n\n    makeOptsSpark=(\n        \"SPARK_ROOT=$HOME/spark-1.5.2-bin-hadoop2.6\"\n        \"SPARK_JAR=$HOME/spark-1.5.2-bin-hadoop2.6/lib/spark-assembly-1.5.2-hadoop2.6.0.jar\"\n        \"SPARKCORE_JAR=$SPARK_JAR\"\n        \"SCALA_ROOT=$(dirname $(which scala))/../lib\"\n    )\n\nAlso you need to have `zipmerge` and for Rootbeer Compilation `ant` in your `PATH` variable. E.g. I installed zipmerge manually to `~/programs/bin`.\n\nNow you can compile e.g. the example using Spark und multiple GPUs with:\n\n    make ${makeOptsSpark[@]} -C multiNode/multiGpu/scala/\n\n\n# Benchmarking\n\n## Implementations\n\n![Strong Scaling Plot (run times) with up to 32 K20x graphic cards](/MontePi/benchmark/strong-scaling-time-logscale-gpu.png?raw=true)\n\n## Weak Scaling\n\nIf a Spark instance is already running and the address of the master is saved in the variable `MASTER_ADDRESS`, then you can simply run:\n\n    nGpusAvailable=32\n    jarFile=$HOME/scaromare/MontePi/multiNode/multiGpu/scala/MontePi.jar\n    fname=weak-scaling_$(date +%Y-%m-%d_%H-%M-%S)\n    echo '' | tee -- \"$fname-cpu.log\" \"$fname-gpu.log\"\n    printf \"# Total Elements   Elements Per Thread    Partitions   Time / s   Calculated Pi\\n\" | tee -- \"$fname-cpu.dat\" \"$fname-gpu.dat\"\n    for (( nPerPartition=2**37; nPerPartition\u003c=2**41; nPerPartition*=4 )); do\n      for (( iRepetition=0; iRepetition\u003c3; iRepetition++ )); do\n        # Spark + GPU\n        for (( nPartitions=1; nPartitions \u003c= $nGpusAvailable; nPartitions++ )) do\n            sparkSubmit \"$jarFile\" $((nPerPartition*nPartitions)) $nPartitions 2\u003e\u00261 |\n                tee tmp.log |             # log file to extract times in seconds from\n                tee -a \"$fname-gpu.log\" | # complete log file for human to view\n                sed '/ INFO /d'           # delete numerous Spark messages\n            seconds=$(sed -nr '\n                s/.*Rolling the dice.*and took (.*) seconds.*/\\1/p' tmp.log)\n            pi=$(sed -nr 's/.*pi ~ ([0-9.]+).*/\\1/p' tmp.log)\n\n            printf \"%16i %16i %4i %10.5f %s\\n\" $((nPerPartition*nPartitions)) \\\n                $nPerPartition $nPartitions $seconds $pi \u003e\u003e \"$fname-gpu.dat\"\n        done\n      done\n    done\n\n![Weak Scaling Plot (run times) with up to 32 K20x graphic cards](/MontePi/benchmark/weak-scaling-time-gpu.png?raw=true)\n\n![ibid. Speedup](/MontePi/benchmark/weak-scaling-speedup-gpu.png?raw=true)\n\n![ibid. Parallel Efficiency](/MontePi/benchmark/weak-scaling-efficiency-gpu.png?raw=true)\n\n### About the Performance Decrease at roughly the half the GPUs\n\nLet's try to find out, if maybe some all the slower runs have a certain host in common which wasn't used in one of the faster rans. First let's extract the times and corresponding used hostnames using a sed script:\n\n    sed -nr '\n    /INFO/d;                    # delete spark output (more a performance thing)\n    /\\(taurusi[0-9]*,/{\n        :findhosts\n            /\\) : 0[ \\t]*$/{    # skip hosts which are not using GPUs\n                s|\\n[^\\n]*$||   # delete last appended line\n                b findhosts\n            }\n            s|[\\^n]*\\((taurusi[0-9]*),.*|\\1|    # extract hostname\n            N                                   # append next line with \\n\n            /\\(taurusi[0-9]*,/b findhosts       # repeat if the new line contains host\n        s|\\n[^\\n]*$||           # delete last appended line\n        s|[ \\t]*\\n[ \\t]*|,|g    # delete newlines and whitespaces between hostnames\n        :pisearch\n            N\n            / pi ~ /!{          # if not line with pi value found\n                s|\\n[^\\n]*$||   # delete last appended line\n                b pisearch\n            }\n        # extract seconds needed\n        s/([^\\n]*).* pi ~ ([0-9.]+) and took ([0-9.]+) seconds.*/\\3 \\1/\n        p\n    }\n    ' weak-scaling_2016-08-03_22-52-00-gpu.log | column -t \u003e time-hosts.dat\n\nManually delete the only half-done longest run with 222s run time and then try to sort the results in fast and slow runs.\n\n - fast runs: `7.0 +- 0.3`, `17.3 +- 0.3`, `58.4 +- 0.3`\n - slow runs: `8.0 +- 0.3`, `18.3 +- 0.3`, `59.4 +- 0.3`\n\nThere seems to be no correlation with one straggling host.\n\nNote that `taurusi2001`-`taurusi2009` and `taurusi2010`-`taurusi2018` respectively share one chassis switch (Gigabit Ethernet and Infiniband).\n\nThis also doesn't seem to be the reason, because almost all runs have hosts from both chassis, especially also fast runs.\n\nIdea: Maybe the switch gets saturated for a certain amount of connections? E.g. only if more than 8 nodes per chassis are used.\n - Also does not seem to be the case.\n\n\n## Strong Scaling\n\n    nGpusAvailable=32\n    jarFile=$HOME/scaromare/MontePi/multiNode/multiGpu/scala/MontePi.jar\n    fname=strong-scaling_$(date +%Y-%m-%d_%H-%M-%S)\n    echo '' | tee -- \"$fname-cpu.log\" \"$fname-gpu.log\"\n    printf \"# Total Elements    Partitions   Time / s   Calculated Pi\\n\" | tee -- \"$fname-cpu.dat\" \"$fname-gpu.dat\"\n    for (( n=2**37; n\u003c=2**43; n*=4 )); do\n      for (( iRepetition=0; iRepetition\u003c5; iRepetition++ )); do\n        # Spark + GPU\n        for (( nPartitions=1; nPartitions \u003c= $nGpusAvailable; nPartitions++ )) do\n            sparkSubmit \"$jarFile\" $n $nPartitions 2\u003e\u00261 |\n                tee tmp.log |             # log file to extract times in seconds from\n                tee -a \"$fname-gpu.log\" | # complete log file for human to view\n                sed '/ INFO /d'           # delete numerous Spark messages\n            seconds=$(sed -nr '\n                s/.*Rolling the dice.*and took (.*) seconds.*/\\1/p' tmp.log)\n            pi=$(sed -nr 's/.*pi ~ ([0-9.]+).*/\\1/p' tmp.log)\n\n            printf \"%16i %4i %10.5f %s\\n\" $n $nPartitions $seconds $pi \u003e\u003e \"$fname-gpu.dat\"\n        done\n      done\n    done\n\n![Strong Scaling Plot (run times) with up to 32 K20x graphic cards](/MontePi/benchmark/strong-scaling-time-logscale-gpu.png?raw=true)\n\n![ibid. Speedup](/MontePi/benchmark/strong-scaling-speedup-gpu.png?raw=true)\n\n![ibid. Parallel Efficiency](/MontePi/benchmark/strong-scaling-efficiency-gpu.png?raw=true)\n\n\n## Plots\n\nTo make the plots `MontePi/benchmark/plot.py` can be used. Try also `python plot.py --help`.\n\n    python plot.py -s strong-scaling_2016-08-04_03-40-03 -k weak-scaling_2016-08-03_22-52-00\n\n\n## Profiling\n\n    salloc --time=02:00:00 --nodes=1 --partition=gpu2-interactive --cpus-per-task=4 --gres='gpu:4' --mem-per-cpu=1000M\n    make ${makeOptsSpark[@]} -C $HOME/scaromare/MontePi/singleNode/multiGpu/scala/ MontePi.jar\n    srun java -jar $HOME/scaromare/MontePi/singleNode/multiGpu/scala/MontePi.jar 32435123451 4\n    which java\n        /sw/global/tools/java/jdk1.7.0_25/bin/java\n    sbatch --time=00:30:00 --nodes=1 --partition=gpu2 --cpus-per-task=4 --gres='gpu:4' \u003c\u003cEOF\n    #!/bin/bash\n    nvprof --analysis-metrics --metrics all -o $HOME/scaromare/MontePi/profilingDataMultiGpuScala.nvp%p /sw/global/tools/java/jdk1.7.0_25/bin/java -jar $HOME/scaromare/MontePi/singleNode/multiGpu/scala/MontePi.jar $((4*14351234510)) 4\n    EOF\n\nOutput:\n    \n    ==26439== NVPROF is profiling process 26439, command: /sw/global/tools/java/jdk1.7.0_25/bin/java -jar $HOME/scaromare/MontePi/singleNode/multiGpu/scala/MontePi.jar 1435123451 4\n    [MonteCarloPi.scala:\u003cconstructor\u003e] Using the following GPUs : 0 \n    [MonteCarloPi.scala:\u003cconstructor\u003e] MonteCarloPi constructor took 6.281092309 seconds\n    0\n    [MonteCarloPi.scala:runOnDevice] [Host:taurusi2051,GPU:0] Total Thread Count = 26624, KernelConfig = (104,1,1) blocks with each (256,1,1) threads\n    [MonteCarloPi.scala:\u003cconstructor\u003e] Using the following GPUs : 1 \n    [MonteCarloPi.scala:\u003cconstructor\u003e] MonteCarloPi constructor took 7.513381237 seconds\n    0\n    [MonteCarloPi.scala:\u003cconstructor\u003e] Using the following GPUs : 2 \n    [MonteCarloPi.scala:\u003cconstructor\u003e] MonteCarloPi constructor took 7.538209838 seconds\n    0\n    [MonteCarloPi.scala:runOnDevice] runOnDevice configuration took 0.748453808 seconds\n    [MonteCarloPi.scala:runOnDevice] context.run( work ) took 5.4038E-5 seconds\n    [MonteCarloPi.scala:calc] Ran Kernels asynchronously on all GPUs. Took 1.265656465 seconds\n    [MonteCarloPi.scala:calc] Taking from GpuFuture now (Wait for asynchronous tasks).\n    [MonteCarloPi.scala:runOnDevice] [Host:taurusi2051,GPU:1] Total Thread Count = 26624, KernelConfig = (104,1,1) blocks with each (256,1,1) threads\n    [MonteCarloPi.scala:runOnDevice] [Host:taurusi2051,GPU:2] Total Thread Count = 26624, KernelConfig = (104,1,1) blocks with each (256,1,1) threads\n    [MonteCarloPi.scala:\u003cconstructor\u003e] Using the following GPUs : 3 \n    [MonteCarloPi.scala:\u003cconstructor\u003e] MonteCarloPi constructor took 7.580257042 seconds\n    0\n    [MonteCarloPi.scala:runOnDevice] [Host:taurusi2051,GPU:3] Total Thread Count = 26624, KernelConfig = (104,1,1) blocks with each (256,1,1) threads\n\n    [MonteCarloPi.scala:runOnDevice] runOnDevice configuration took 0.965182109 seconds\n    [MonteCarloPi.scala:runOnDevice] context.run( work ) took 4.1006E-5 seconds\n    [MonteCarloPi.scala:calc] Ran Kernels asynchronously on all GPUs. Took 1.017713559 seconds\n    [MonteCarloPi.scala:calc] Taking from GpuFuture now (Wait for asynchronous tasks).\n    [MonteCarloPi.scala:runOnDevice] runOnDevice configuration took 0.974981803 seconds\n    [MonteCarloPi.scala:runOnDevice] context.run( work ) took 3.4606E-5 seconds\n    [MonteCarloPi.scala:calc] Ran Kernels asynchronously on all GPUs. Took 1.004266163 seconds\n    [MonteCarloPi.scala:calc] Taking from GpuFuture now (Wait for asynchronous tasks).\n    [MonteCarloPi.scala:runOnDevice] runOnDevice configuration took 0.960264386 seconds\n    [MonteCarloPi.scala:runOnDevice] context.run( work ) took 2.3148E-5 seconds\n    [MonteCarloPi.scala:calc] Ran Kernels asynchronously on all GPUs. Took 0.976993471 seconds\n    [MonteCarloPi.scala:calc] Taking from GpuFuture now (Wait for asynchronous tasks).\n\n\n\n    [MonteCarloPi.scala:calc] synchronize (take) i.e. kernels took 1.673575898 seconds\n    Join thread after 9.409900961 seconds\n    [MonteCarloPi.scala:calc] synchronize (take) i.e. kernels took 1.296328742 seconds\n    [MonteCarloPi.scala:calc] synchronize (take) i.e. kernels took 1.3390092 seconds\n    [MonteCarloPi.scala:calc] synchronize (take) i.e. kernels took 1.326177838 seconds\n    Join thread after 10.060772521 seconds\n    Join thread after 10.25184158 seconds\n    Join thread after 10.253567369 seconds\n    Rolling the dice 1435123451 times resulted in pi ~ 3.141675075310326 and took 10.253777488 seconds\n    ==26439== Generated result file: $HOME/scaromare/MontePi/profilingDataMultiGpuScala.nvp26439\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmxmlnkn%2Fscaromare","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmxmlnkn%2Fscaromare","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmxmlnkn%2Fscaromare/lists"}