{"id":23360658,"url":"https://github.com/msteindorfer/research-paper-oopsla15-artifact","last_synced_at":"2025-08-23T15:32:24.907Z","repository":{"id":150194144,"uuid":"41114192","full_name":"msteindorfer/research-paper-oopsla15-artifact","owner":"msteindorfer","description":"Artifact accompanying our OOPSLA'15 publication.","archived":true,"fork":false,"pushed_at":"2023-08-16T13:28:53.000Z","size":68204,"stargazers_count":32,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-12-21T11:15:30.881Z","etag":null,"topics":["computer-science","data-structures-and-algorithms","immutable-collections","research","research-data","research-paper"],"latest_commit_sha":null,"homepage":"https://michael.steindorfer.name/publications/oopsla15.pdf","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/msteindorfer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-20T19:11:06.000Z","updated_at":"2024-01-30T04:42:16.000Z","dependencies_parsed_at":"2023-04-17T17:48:51.769Z","dependency_job_id":null,"html_url":"https://github.com/msteindorfer/research-paper-oopsla15-artifact","commit_stats":null,"previous_names":["msteindorfer/research-paper-oopsla15-artifact"],"tags_count":null,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msteindorfer%2Fresearch-paper-oopsla15-artifact","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msteindorfer%2Fresearch-paper-oopsla15-artifact/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msteindorfer%2Fresearch-paper-oopsla15-artifact/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msteindorfer%2Fresearch-paper-oopsla15-artifact/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/msteindorfer","download_url":"https://codeload.github.com/msteindorfer/research-paper-oopsla15-artifact/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230706584,"owners_count":18268295,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-science","data-structures-and-algorithms","immutable-collections","research","research-data","research-paper"],"created_at":"2024-12-21T11:15:33.301Z","updated_at":"2025-08-23T15:32:24.881Z","avatar_url":"https://github.com/msteindorfer.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Getting Started Guide\n\nOur evaluation consists of microbenchmarks and real-world benchmarks, both are fully automated. The system requirements to execute them are as follows:\n\n* Operating system: Apple OS X or Linux\n* Command line tools: \n\t* java (version 8),\n\t* maven (version \u003e= 3.2), \n\t* make (we used GNU Make 3.81),\n\t* ant (we used version 1.9.5), \n\t* R and RScript (we used version 3.2.0)\n* Internet connection (for automatically downloading dependencies)\n\nThe benchmarks requires heap sizes of 4GB, thus machines with at leas 8GB RAM are recommended.\n\nWe assume familiarity with UNIX terminals and command line tools. We do not go into details how to install the above mentioned command line tools. We ourselves used our artifact, both, under Apple OS X and Linux.\n\nWe wanted to make the use of our artifact as simple as possible. If the system requirements are fulfilled, the reproduction of our results require the execution of three commands in a console/terminal:\n\nMoving into the artifacts directory:\n\u003e cd oopsla15-artifacts\n\nSetting up and compiling the artifacts:\n\u003e make prepare\n\nRunning microbenchmarks and real-world benchmarks:\n\u003e make run\n\nRunning result analysis and post-processing:\n\u003e make postprocessing\n\nThe first command does not consume time. The second command should take approximately five minutes to complete and should complete without errors. The third command however will take several hours or even days. E.g., in our real-world evaluation the slowest single invocation completes in 30 minutes. For statistical testing of our results we invoke every benchmarks multiple times. Step four, the analysis and postprocessing takes around a minute or less usually.\n\nAs an alternative to **make prepare** and **make run** we provide a **make run_prebuilt** command that runs a prebuilt benchmarks JAR file. If you experience any issues in running the experiments, you might start with the **make run_prebuilt** command.\n\nWe further included all results that we obtained form step number three. Consequently our results can be evaluated without the necessity to execute our automated benchmark suite. We provide an extra command for this purpose:\n\u003e make postprocessing_cached\n\nFurthermore, in section \"Running the Benchmarks on Smaller Samples\" we will point out how to run the experiments on smaller subsets, that consume less time.\n\nTo manually inspect what the **make** commands do, have a look at *oopsla15-artifacts/Makefile*.\n\n## Key Data Items of our Evaluation\nOur cached results are contained in the folder *oopsla15-benchmarks/resources/r*. \n\nThe following files contain data from the microbenchmarks that are discussed in Section 6 of the paper:\n\n* *results.all-20150817_0732.log*: comma-separated values (CSV) file containing microbenchmark results of runtimes of individual operations\n* *map-sizes-and-statistics-32bit-20150817_0732.csv*: CSV file containing memory footprints in a 32-bit JVM setting.\n* *map-sizes-and-statistics-64bit-20150817_0732.csv*: CSV file containing memory footprints in a 64-bit JVM setting.\n\nThese CSV files are then processed by *benchmarks.r*, a R script, and produce directly the boxplots of Figures 4, 5, 6 and 7 of the papers. The boxplots are named *all-benchmarks-vf_pdb_persistent_(current|memoized)_by_vf_(scala|clojure)-(set|map)-boxplot.pdf*. \n\nThe following files contain data from the real-world benchmarks that are discussed in Section 7 of the paper:\n\n* *results.all-real-world-20150404_1013.log*: a CSV file containing the runtime results of our real-word benchmarks (i.e., of the control-flow graph dominator tree calculations).\n\n## Key Source Items of our Artifact\nOur CHAMP hash trie implementations can be found under *pdb.values/src/org/eclipse/imp/pdb/facts/util/Trie(Set|Map)_5Bits.java*, and MEMCHAMP under *pdb.values/src/org/eclipse/imp/pdb/facts/util/Trie(Set|Map)_5Bits_Memoized_LazyHashCode.java*, for people interested in manually inspecting our implementations.\n\nProjects *pdb.values.persistent.(clojure|scala)* contain simple interface facades that enables cross-library benchmarks under a common API.\n\nThe benchmark implementations can be found in the *oopsla15-benchmarks* project.  Files *Dominators(Champ|Clojure).java* and *DominatorsScala_Default.scala* implement the real-word experiment (Section 7 of the paper). For Champ and Scala there are addtional dominator implementations with postfix *LazyHashCode* for the normalized experiments.\n\nFiles *Jmh(Set|Map)Benchmarks.java* measure the runtimes of individual operations, whereas *CalculateFootprints.java* performs footprint measurements (cf. Section 6, Figures 4, 5, 6 and 7).  Note that the benchmarks contain default parameters for their invocation, the actual parameters are set in *runMicrobenchmarks.sh* and *runRealWorldEvaluation.sh*.\n\n## Running the Benchmarks on Smaller Samples \n\nIn order to run the microbenchmarks on smaller-sized examples we recommend (some of) the following changes in *runMicrobenchmarks.sh*:\n\n* Lines 15-25: each lines performs a benchmark invocations for a specific benchmark/input size combination. To reduce the number of experiments, simply comment out multiple lines.\n* Line 10: AGGREGATED_SETTINGS contains the general settings we use for microbenchmarking (-wi = warmup iterations, -i = benchmark invocations, -p run = a list of identifiers that will create different trees with other random data seeds). To minimize the number of runs set the following parameters: -wi 1 -i 3 -p run=0. These parameters render the benchmarks useless from a statistical point of view, but enable experimentation with the framework.\n* Lines 64-65: those lines perform the footprint calculations that consume quite some time (~15 minutes). Thus for quick experiments one could comment out those lines.\n\nIn order to run the real-world benchmarks on smaller-sized examples we recommend (some of) the following changes in *runRealWorldEvaluation.sh*:\n\n* Line 12: Set the following parameters (note that the size parameter is introduced to limit testing to a certain subset size instead of all tested data points): -wi 1 -i 3 -p size=8","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsteindorfer%2Fresearch-paper-oopsla15-artifact","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmsteindorfer%2Fresearch-paper-oopsla15-artifact","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsteindorfer%2Fresearch-paper-oopsla15-artifact/lists"}