{"id":19584904,"url":"https://github.com/archivesunleashed/aut-benchmark","last_synced_at":"2026-05-13T08:03:15.712Z","repository":{"id":150728557,"uuid":"211103389","full_name":"archivesunleashed/aut-benchmark","owner":"archivesunleashed","description":null,"archived":false,"fork":false,"pushed_at":"2020-02-26T14:04:18.000Z","size":68416,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-09T05:45:37.261Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/archivesunleashed.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-26T14:03:09.000Z","updated_at":"2020-02-26T20:19:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"68b43997-152f-47f3-a0f6-b81e201b218e","html_url":"https://github.com/archivesunleashed/aut-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archivesunleashed%2Faut-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archivesunleashed%2Faut-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archivesunleashed%2Faut-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archivesunleashed%2Faut-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/archivesunleashed","download_url":"https://codeload.github.com/archivesunleashed/aut-benchmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240858536,"owners_count":19868997,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T07:51:32.799Z","updated_at":"2026-05-13T08:03:15.665Z","avatar_url":"https://github.com/archivesunleashed.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AUT Benchmark\n\n## Machine\n\n```\nr5d.4xlarge\n  - 16 virtual cores\n  - 128G RAM\n  - 2 x 300G ephemeral mount points\n  - us-east-1\n```\n\n## Setup\n\n```bash\nsudo mkdir /mnt/xfs\n\nsudo zpool create au-zfs /dev/nvme1n1\nsudo mkfs -t xfs\n\nsudo chown -hR ubuntu:ubuntu /au-zfs /mnt/xfs\n\ncd /au-zfs\nmkdir -p data tmp output/auk/all-text output/auk/all-domains output/auk/gephi output/binaries/audio output/binaries/image output/binaries/pdf output/binaries/spreadsheet output/binaries/word-processor output/binaries/presentation-program output/binaries/text output/binaries/video output/df/audio output/df/image output/df/pdf output/df/spreadsheet output/df/word-processor output/df/presentation-program output/df/text output/df/video\n\ncd /mnt/xfs\nmkdir -p data tmp output/auk/all-text output/auk/all-domains output/auk/gephi output/binaries/audio output/binaries/image output/binaries/pdf output/binaries/spreadsheet output/binaries/word-processor output/binaries/presentation-program output/binaries/text output/binaries/video output/df/audio output/df/image output/df/pdf output/df/spreadsheet output/df/word-processor output/df/presentation-program output/df/text output/df/video\n\n```\n\n## Notes\n\n- ZFS\n  - RAID 0\n  - single device pool\n- S3 buckets\n  - no versions\n  - no encryption\n  - us-east-1\n\n## Jobs\n\n### AUK\n\n#### r-xfs-w-xfs\n\n```bash\nrm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-auk-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-auk-01.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-auk-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-auk-02.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-auk-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-auk-03.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-auk-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-auk-04.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-auk-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-auk-05.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /mnt/xfs/tmp/*\n```\n\n#### r-xfs-w-zfs\n\n```bash\nrm -rf /au-zfs/output/auk; cd /au-zfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-auk-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-auk-01.log; rm -rf /au-zfs/output/auk; cd /au-zfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-auk-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-auk-02.log; rm -rf /au-zfs/output/auk; cd /au-zfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-auk-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-auk-03.log; rm -rf /au-zfs/output/auk; cd /au-zfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-auk-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-auk-04.log; rm -rf /au-zfs/output/auk; cd /au-zfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-auk-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-auk-05.log; rm -rf /au-zfs/output/auk; cd /au-zfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /au-zfs/tmp\n```\n\n#### r-zfs-w-zfs\n\n```bash\nrm -rf /au-zfs/output/auk;  cd /au-zfs/output;  mkdir -p auk/all-text auk/all-domains auk/gephi;  rm -rf /au-zfs/tmp/*;  /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-auk-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-auk-01.log;  rm -rf /au-zfs/output/auk;  cd /au-zfs/output;  mkdir -p auk/all-text auk/all-domains auk/gephi;  rm -rf /au-zfs/tmp/*;  /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-auk-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-auk-02.log;  rm -rf /au-zfs/output/auk;  cd /au-zfs/output;  mkdir -p auk/all-text auk/all-domains auk/gephi;  rm -rf /au-zfs/tmp/*;  /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-auk-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-auk-03.log;  rm -rf /au-zfs/output/auk;  cd /au-zfs/output;  mkdir -p auk/all-text auk/all-domains auk/gephi;  rm -rf /au-zfs/tmp/*;  /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-auk-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-auk-04.log;  rm -rf /au-zfs/output/auk;  cd /au-zfs/output;  mkdir -p auk/all-text auk/all-domains auk/gephi;  rm -rf /au-zfs/tmp/*;  /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-auk-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-auk-05.log;  rm -rf /au-zfs/output/auk;  cd /au-zfs/output;  mkdir -p auk/all-text auk/all-domains auk/gephi;  rm -rf /au-zfs/tmp/*\n```\n\n#### r-zfs-w-xfs\n\n```bash\nrm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-auk-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-auk-01.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-auk-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-auk-02.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-auk-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-auk-03.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-auk-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-auk-04.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-auk-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-auk-05.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; rm -rf /mnt/xfs/tmp/*\n```\n\n### Binaries\n\n#### r-xfs-w-xfs\n\n```bash\nrm -rf /mnt/xfs/output/binaries; cd /mnt/xfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-bin-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-bin-01.log; rm -rf /mnt/xfs/output/binaries; cd /mnt/xfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-bin-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-bin-02.log; rm -rf /mnt/xfs/output/binaries; cd /mnt/xfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-bin-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-bin-03.log; rm -rf /mnt/xfs/output/binaries; cd /mnt/xfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-bin-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-bin-04.log; rm -rf /mnt/xfs/output/binaries; cd /mnt/xfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-bin-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-bin-05.log; rm -rf /mnt/xfs/output/binaries; cd /mnt/xfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /mnt/xfs/tmp/*\n```\n\n#### r-xfs-w-zfs\n\n```bash\nrm -rf /au-zfs/output/binaries; cd /au-zfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-bin-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-bin-01.log; rm -rf /au-zfs/output/binaries; cd /au-zfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-bin-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-bin-02.log; rm -rf /au-zfs/output/binaries; cd /au-zfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-bin-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-bin-03.log; rm -rf /au-zfs/output/binaries; cd /au-zfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-bin-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-bin-04.log; rm -rf /au-zfs/output/binaries; cd /au-zfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-bin-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-bin-05.log; rm -rf /au-zfs/output/binaries; cd /au-zfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /au-zfs/tmp/*\n```\n\n#### r-zfs-w-zfs\n\n```bash\nrm -rf /au-zfs/output/binaries; cd /au-zfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /au-zfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-bin-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-bin-01.log; rm -rf /au-zfs/output/binaries; cd /au-zfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /au-zfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-bin-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-bin-02.log; rm -rf /au-zfs/output/binaries; cd /au-zfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /au-zfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-bin-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-bin-03.log; rm -rf /au-zfs/output/binaries; cd /au-zfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /au-zfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-bin-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-bin-04.log; rm -rf /au-zfs/output/binaries; cd /au-zfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /au-zfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-bin-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-bin-05.log; rm -rf /au-zfs/output/binaries; cd /au-zfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /au-zfs/tmp/*\n```\n\n#### r-zfs-w-xfs\n\n```bash\nrm -rf /mnt/xfs/output/binaries; cd /mnt/xfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-bin-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-bin-01.log; rm -rf /mnt/xfs/output/binaries; cd /mnt/xfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-bin-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-bin-02.log; rm -rf /mnt/xfs/output/binaries; cd /mnt/xfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-bin-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-bin-03.log; rm -rf /mnt/xfs/output/binaries; cd /mnt/xfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-bin-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-bin-04.log; rm -rf /mnt/xfs/output/binaries; cd /mnt/xfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-bin-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-bin.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-bin-05.log; rm -rf /mnt/xfs/output/binaries; cd /mnt/xfs/output; mkdir -p binaries/audio binaries/image binaries/pdf binaries/presentation-program binaries/spreadsheet binaries/text binaries/video binaries/word-processor; rm -rf /mnt/xfs/tmp/*\n```\n\n### DataFrames\n\n#### r-xfs-w-xfs\n\n```bash\nrm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-df-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-df-01.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-df-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-df-02.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-df-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-df-03.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-df-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-df-04.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-xfs-df-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-xfs-df-05.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /mnt/xfs/tmp/*\n```\n\n#### r-xfs-w-zfs\n\n```bash\nrm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-df-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-df-01.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-df-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-df-02.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-df-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-df-03.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-df-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-df-04.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /au-zfs/tmp; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-zfs-df-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-zfs-df-05.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /au-zfs/tmp/*\n```\n\n#### r-zfs-w-zfs\n\n```bash\nrm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /au-zfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-df-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-df-01.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /au-zfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-df-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-df-02.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /au-zfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-df-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-df-03.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /au-zfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-df-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-df-04.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /au-zfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-zfs-df-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/au-zfs/tmp -Djava.io.tmpdir=/au-zfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-zfs-df-05.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /au-zfs/tmp/*\n```\n\n#### r-zfs-w-xfs\n\n```bash\nrm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-df-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-df-01.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-df-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-df-02.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-df-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-df-03.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-df-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-df-04.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /mnt/xfs/tmp/*; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-xfs-df-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --conf spark.local.dir=/mnt/xfs/tmp -Djava.io.tmpdir=/mnt/xfs/tmp --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-xfs-df-05.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; rm -rf /mnt/xfs/tmp/*\n```\n\n### S3 DataFrames\n\n#### r-zfs-w-s3\n\n```bash\naws s3 rb s3://au-df --force; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-s3-df-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-s3-df-01.log; aws s3 rb s3://au-df --force; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-s3-df-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-s3-df-02.log; aws s3 rb s3://au-df --force; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-s3-df-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-s3-df-03.log; aws s3 rb s3://au-df --force; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-s3-df-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-s3-df-04.log; aws s3 rb s3://au-df --force; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-zfs-w-s3-df-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-zfs-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-zfs-w-s3-df-05.log;\n```\n\n#### r-xfs-w-s3\n\n```bash\naws s3 rb s3://au-df --force; sleep 60; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-s3-auk-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-s3-auk-01.log;  aws s3 rb s3://au-df --force; sleep 60; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-s3-auk-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-s3-auk-02.log; aws s3 rb s3://au-df --force; sleep 60; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-s3-auk-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-s3-auk-03.log; aws s3 rb s3://au-df --force; sleep 60; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-s3-auk-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-s3-auk-04.log; aws s3 rb s3://au-df --force; sleep 60; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-xfs-w-s3-auk-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-xfs-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-xfs-w-s3-auk-05.log;\n```\n\n#### r-s3-w-s3\n\n```bash\naws s3 rb s3://au-df --force; sleep 60; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-s3-df-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-s3-df-01.log; aws s3 rb s3://au-df --force; sleep 60; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-s3-df-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-s3-df-02.log; aws s3 rb s3://au-df --force; sleep 60; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-s3-df-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-s3-df-03.log; aws s3 rb s3://au-df --force; sleep 60; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-s3-df-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-s3-df-04.log; aws s3 rb s3://au-df --force; sleep 60; aws s3 mb s3://au-df; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-s3-df-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-s3-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-s3-df-05.log;\n```\n\n#### r-s3-w-xfs\n\n```bash\nrm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-xfs-df-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-xfs-df-01.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-xfs-df-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-xfs-df-02.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-xfs-df-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-xfs-df-03.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-xfs-df-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-xfs-df-04.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-xfs-df-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-xfs-df-05.log;\n```\n\n#### r-s3-w-zfs\n\n```bash\nrm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-zfs-df-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-zfs-df-01.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-zfs-df-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-zfs-df-02.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-zfs-df-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-zfs-df-03.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-zfs-df-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-zfs-df-04.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-zfs-df-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-zfs-df-05.log;\n```\n\n## Orca\n\n#### r-s3-w-orca-auk\n\n```bash\nrm -rf /store/scratch/nruest/benchmark/auk; cd /store/scratch/nruest/benchmark; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-auk-01.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-auk-01.log; rm -rf /store/scratch/nruest/benchmark/auk; cd /store/scratch/nruest/benchmark; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-auk-02.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-auk-02.log; rm -rf /store/scratch/nruest/benchmark/auk; cd /store/scratch/nruest/benchmark; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-auk-03.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-auk-03.log; rm -rf /store/scratch/nruest/benchmark/auk; cd /store/scratch/nruest/benchmark; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-auk-04.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-auk-04.log; rm -rf /store/scratch/nruest/benchmark/auk; cd /store/scratch/nruest/benchmark; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-auk-05.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-auk-05.log; rm -rf /store/scratch/nruest/benchmark/auk; cd /store/scratch/nruest/benchmark; mkdir -p auk/all-text auk/all-domains auk/gephi;\n```\n\n#### r-s2-w-orca-df\n\n```bash\nrm -rf /store/scratch/nruest/benchmark/df; cd /store/scratch/nruest/benchmark; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-df-01.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-df-01.log; rm -rf /store/scratch/nruest/benchmark/df; cd /store/scratch/nruest/benchmark; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-df-02.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-df-02.log; rm -rf /store/scratch/nruest/benchmark/df; cd /store/scratch/nruest/benchmark; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-df-03.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-df-03.log; rm -rf /store/scratch/nruest/benchmark/df; cd /store/scratch/nruest/benchmark; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-df-04.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-df-04.log; rm -rf /store/scratch/nruest/benchmark/df; cd /store/scratch/nruest/benchmark; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-df-05.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g  --jars ~/aut-0.18.0-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-df-05.log; rm -rf /store/scratch/nruest/benchmark/df; cd /store/scratch/nruest/benchmark; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor;\n```\n\n## Rho\n\n#### df\n\n```bash\nrm -rf /mnt/vol1/derivative_data/tmp/df/links-all; /usr/bin/time -o /home/nruest/jcdl-experiment/times/df-01.txt -v ~/bin/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local\\[20\\] --driver-memory 65g --jars /home/nruest/jcdl-experiment/aut-0.18.1-SNAPSHOT-fatjar.jar -i /home/nruest/jcdl-experiment/df.scala 2\u003e\u00261 | tee /home/nruest/jcdl-experiment/logs/df-01.log; rm -rf /mnt/vol1/derivative_data/tmp/df/links-all; /usr/bin/time -o /home/nruest/jcdl-experiment/times/df-02.txt -v ~/bin/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local\\[20\\] --driver-memory 65g --jars /home/nruest/jcdl-experiment/aut-0.18.1-SNAPSHOT-fatjar.jar -i /home/nruest/jcdl-experiment/df.scala 2\u003e\u00261 | tee /home/nruest/jcdl-experiment/logs/df-02.log; rm -rf /mnt/vol1/derivative_data/tmp/df/links-all; /usr/bin/time -o /home/nruest/jcdl-experiment/times/df-03.txt -v ~/bin/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local\\[20\\] --driver-memory 65g --jars /home/nruest/jcdl-experiment/aut-0.18.1-SNAPSHOT-fatjar.jar -i /home/nruest/jcdl-experiment/df.scala 2\u003e\u00261 | tee /home/nruest/jcdl-experiment/logs/df-03.log;rm -rf /mnt/vol1/derivative_data/tmp/df/links-all; /usr/bin/time -o /home/nruest/jcdl-experiment/times/df-04.txt -v ~/bin/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local\\[20\\] --driver-memory 65g --jars /home/nruest/jcdl-experiment/aut-0.18.1-SNAPSHOT-fatjar.jar -i /home/nruest/jcdl-experiment/df.scala 2\u003e\u00261 | tee /home/nruest/jcdl-experiment/logs/df-04.log; rm -rf /mnt/vol1/derivative_data/tmp/df/links-all; /usr/bin/time -o /home/nruest/jcdl-experiment/times/df-05.txt -v ~/bin/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local\\[20\\] --driver-memory 65g --jars /home/nruest/jcdl-experiment/aut-0.18.1-SNAPSHOT-fatjar.jar -i /home/nruest/jcdl-experiment/df.scala 2\u003e\u00261 | tee /home/nruest/jcdl-experiment/logs/df-05.log;\n```\n\n#### rdd\n\n```bash\nrm -rf /mnt/vol1/derivative_data/tmp/rdd/links-all; /usr/bin/time -o /home/nruest/jcdl-experiment/times/rdd-01.txt -v ~/bin/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local\\[20\\] --driver-memory 65g --jars /home/nruest/jcdl-experiment/aut-0.18.1-SNAPSHOT-fatjar.jar -i /home/nruest/jcdl-experiment/rdd.scala 2\u003e\u00261 | tee /home/nruest/jcdl-experiment/logs/rdd-01.log; rm -rf /mnt/vol1/derivative_data/tmp/rdd/links-all; /usr/bin/time -o /home/nruest/jcdl-experiment/times/rdd-02.txt -v ~/bin/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local\\[20\\] --driver-memory 65g --jars /home/nruest/jcdl-experiment/aut-0.18.1-SNAPSHOT-fatjar.jar -i /home/nruest/jcdl-experiment/rdd.scala 2\u003e\u00261 | tee /home/nruest/jcdl-experiment/logs/rdd-02.log; rm -rf /mnt/vol1/derivative_data/tmp/rdd/links-all; /usr/bin/time -o /home/nruest/jcdl-experiment/times/rdd-03.txt -v ~/bin/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local\\[20\\] --driver-memory 65g --jars /home/nruest/jcdl-experiment/aut-0.18.1-SNAPSHOT-fatjar.jar -i /home/nruest/jcdl-experiment/rdd.scala 2\u003e\u00261 | tee /home/nruest/jcdl-experiment/logs/rdd-03.log;rm -rf /mnt/vol1/derivative_data/tmp/rdd/links-all; /usr/bin/time -o /home/nruest/jcdl-experiment/times/rdd-04.txt -v ~/bin/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local\\[20\\] --driver-memory 65g --jars /home/nruest/jcdl-experiment/aut-0.18.1-SNAPSHOT-fatjar.jar -i /home/nruest/jcdl-experiment/rdd.scala 2\u003e\u00261 | tee /home/nruest/jcdl-experiment/logs/rdd-04.log; rm -rf /mnt/vol1/derivative_data/tmp/rdd/links-all; /usr/bin/time -o /home/nruest/jcdl-experiment/times/rdd-05.txt -v ~/bin/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local\\[20\\] --driver-memory 65g --jars /home/nruest/jcdl-experiment/aut-0.18.1-SNAPSHOT-fatjar.jar -i /home/nruest/jcdl-experiment/rdd.scala 2\u003e\u00261 | tee /home/nruest/jcdl-experiment/logs/rdd-05.log;\n```\n\n## EC2 + RADOS\n\n### r-RADOS-w-ec2-xfs-df\n\n```bash\nrm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-xfs-df-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-xfs-df-01.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-xfs-df-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-xfs-df-02.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-xfs-df-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-xfs-df-03.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-xfs-df-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-xfs-df-04.log; rm -rf /mnt/xfs/output/df; cd /mnt/xfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-xfs-df-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-xfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-xfs-df-05.log;\n```\n\n### r-RADOS-w-ec2-zfs-df\n\n```bash\nrm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-zfs-df-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-zfs-df-01.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-zfs-df-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-zfs-df-02.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-zfs-df-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-zfs-df-03.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-zfs-df-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-zfs-df-04.log; rm -rf /au-zfs/output/df; cd /au-zfs/output; mkdir -p df/audio df/image df/pdf df/presentation-program df/spreadsheet df/text df/video df/word-processor; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-zfs-df-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-zfs-df.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-zfs-df-05.log;\n```\n\n### r-RADOS-w-ec2-zfs-auk\n\n```bash\nrm -rf /au-zfs/output/auk; cd /au-zfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-zfs-auk-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-zfs-auk-01.log; rm -rf /au-zfs/output/auk; cd /au-zfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-zfs-auk-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-zfs-auk-02.log; rm -rf /au-zfs/output/auk; cd /au-zfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-zfs-auk-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-zfs-auk-03.log; rm -rf /au-zfs/output/auk; cd /au-zfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-zfs-auk-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-zfs-auk-04.log; rm -rf /au-zfs/output/auk; cd /au-zfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-zfs-auk-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-zfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-zfs-auk-05.log;\n```\n\n### r-RADOS-w-ec2-xfs-auk\n\n```bash\nrm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-xfs-auk-01.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-xfs-auk-01.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-xfs-auk-02.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-xfs-auk-02.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-xfs-auk-03.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-xfs-auk-03.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-xfs-auk-04.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-xfs-auk-04.log; rm -rf /mnt/xfs/output/auk; cd /mnt/xfs/output; mkdir -p auk/all-text auk/all-domains auk/gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-xfs-auk-05.txt -v ~/spark-2.4.4-bin-hadoop2.7/bin/spark-shell --master local[*] --driver-memory 125g  --jars ~/aut-hadoop-2.7.2.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-xfs-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-xfs-auk-05.log;\n```\n\n## JCDL 2020\n\n### r-rados-w-ec2-auk\n\n```bash\nrm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-ec2-auk-01.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-ec2-auk-01.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-ec2-auk-02.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-ec2-auk-02.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-ec2-auk-03.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-ec2-auk-03.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-ec2-auk-04.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-ec2-auk-04.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-ec2-auk-05.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-ec2-auk-05.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi;\n```\n\n### r-s3-w-ec2-auk\n\n```bash\nrm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-ec2-auk-01.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-ec2-auk-01.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-ec2-auk-02.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-ec2-auk-02.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-ec2-auk-03.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-ec2-auk-03.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-ec2-auk-04.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-ec2-auk-04.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-ec2-auk-05.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-ec2-auk-05.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi;\n```\n\n### r-local-w-ec2-auk\n\n```bash\nrm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-local-w-ec2-auk-01.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-local-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-local-w-ec2-auk-01.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-local-w-ec2-auk-02.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-local-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-local-w-ec2-auk-02.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-local-w-ec2-auk-03.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-local-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-local-w-ec2-auk-03.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-local-w-ec2-auk-04.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-local-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-local-w-ec2-auk-04.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-local-w-ec2-auk-05.txt -v /home/ubuntu/spark/bin/spark-shell --master local[*] --driver-memory 120g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-local-w-ec2-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-local-w-ec2-auk-05.log; rm -rf /dev/5467/gephi /dev/5467/all-text /dev/5467/all-domains; cd /dev/5467; mkdir -p all-text all-domains gephi;\n```\n\n### r-rados-w-orca-auk\n\n```bash\nrm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-orca-auk-01.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-orca-auk-01.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-orca-auk-02.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-orca-auk-02.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-orca-auk-03.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-orca-auk-03.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-orca-auk-04.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-orca-auk-04.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-rados-w-orca-auk-05.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-rados-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-rados-w-orca-auk-05.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi;\n```\n\n### r-local-w-orca-auk\n\n```bash\nrm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-local-w-orca-auk-01.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-local-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-local-w-orca-auk-01.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-local-w-orca-auk-02.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-local-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-local-w-orca-auk-02.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-local-w-orca-auk-03.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-local-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-local-w-orca-auk-03.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-local-w-orca-auk-04.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-local-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-local-w-orca-auk-04.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-local-w-orca-auk-05.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-local-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-local-w-orca-auk-05.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi;\n```\n\n### r-s3-w-orca-auk\n\n```bash\nrm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-auk-01.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-auk-01.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-auk-02.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-auk-02.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-auk-03.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-auk-03.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-auk-04.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-auk-04.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi; /usr/bin/time -o ~/aut-benchmark/times/r-s3-w-orca-auk-05.txt -v /store/share/spark/bin/spark-shell --master local\\[\\*\\] --driver-memory 500g --jars ~/aut-0.50.1-SNAPSHOT-fatjar.jar -i ~/aut-benchmark/src/main/scala/r-s3-w-orca-auk.scala 2\u003e\u00261 | tee ~/aut-benchmark/logs/r-s3-w-orca-auk-05.log; rm -rf /store/scratch/nruest/web_archives/winnipeg/5467/gephi /store/scratch/nruest/web_archives/winnipeg/5467/all-text /store/scratch/nruest/web_archives/winnipeg/5467/all-domains; cd /store/scratch/nruest/web_archives/winnipeg/5467; mkdir -p all-text all-domains gephi;\n```\n\n## [Results](https://docs.google.com/spreadsheets/d/1ZBDc5O0yZ52PLoh-SynbJJmWSHoemQIn0FyQMb5IMZE/edit?usp=sharing)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farchivesunleashed%2Faut-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farchivesunleashed%2Faut-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farchivesunleashed%2Faut-benchmark/lists"}