{"id":13467142,"url":"https://github.com/smarr/are-we-fast-yet","last_synced_at":"2025-05-16T04:05:45.978Z","repository":{"id":37492935,"uuid":"47463198","full_name":"smarr/are-we-fast-yet","owner":"smarr","description":"Are We Fast Yet? Comparing Language Implementations with Objects, Closures, and Arrays","archived":false,"fork":false,"pushed_at":"2025-02-22T20:29:04.000Z","size":3933,"stargazers_count":346,"open_issues_count":18,"forks_count":39,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-04-08T14:10:21.599Z","etag":null,"topics":["benchmark","benchmarking","comparison","dynamic-languages","language","language-implementations","performance"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/smarr.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.bib","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-12-05T16:27:05.000Z","updated_at":"2025-03-25T15:08:07.000Z","dependencies_parsed_at":"2024-01-13T04:16:36.266Z","dependency_job_id":"c56cb78f-f749-4412-82c4-3b3b28e53fe9","html_url":"https://github.com/smarr/are-we-fast-yet","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smarr%2Fare-we-fast-yet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smarr%2Fare-we-fast-yet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smarr%2Fare-we-fast-yet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smarr%2Fare-we-fast-yet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/smarr","download_url":"https://codeload.github.com/smarr/are-we-fast-yet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254464895,"owners_count":22075570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","benchmarking","comparison","dynamic-languages","language","language-implementations","performance"],"created_at":"2024-07-31T15:00:53.514Z","updated_at":"2025-05-16T04:05:45.943Z","avatar_url":"https://github.com/smarr.png","language":"Java","funding_links":[],"categories":["Java","Uncategorized"],"sub_categories":["Uncategorized"],"readme":"Are We Fast Yet? Comparing Language Implementations with Objects, Closures, Arrays, and Strings\n===================================================================================================\n\n[![Build Status](https://github.com/smarr/are-we-fast-yet/actions/workflows/ci.yml/badge.svg)](https://github.com/smarr/are-we-fast-yet/actions/workflows/ci.yml)\n\n## Goal\n\nThe goal of this project is to assess whether a language implementation is\n*highly optimizing* and thus able to remove the overhead of programming\nabstractions and frameworks. We are interested in *comparing language\nimplementations* (not _languages_!) with each other and optimize their compilers as well as the\nrun-time representation of objects, closures, arrays, and strings.\n\nThis is in contrast to other projects such as the [Computer Language Benchmark\ngame][CLBG], which encourage finding the\nsmartest possible way to express a problem in a language to achieve best\nperformance, an equally interesting but different problem.\n\n#### Approach\n\nTo allow us to compare the degree of optimization done by the implementations\nas well as the absolute performance achieved, we set the following basic rules:\n\n  1. The benchmark is 'identical' for all languages.  \n     This is achieved by relying only on a widely available and commonly used\n     subset of language features and data types.\n\n  2. The benchmarks should use language 'idiomatically'.  \n     This means, they should be realized as much as possible with idiomatic\n     code in each language, while relying only on the core set of abstractions.\n\nFor the detailed set of rules see [the guidelines](docs/guidelines.md) document.\nFor a description of the set of common language abstractions see [the *core*\nlanguage](docs/core-language.md) document.\n\nThe initial publication describing the project is [Cross-Language Compiler\nBenchmarking: Are We Fast Yet?][3] and can be cited as ([bib file][28]):\n\n  \u003e Stefan Marr, Benoit Daloze, Hanspeter Mössenböck. 2016.\n  \u003e [Cross-Language Compiler Benchmarking: Are We Fast Yet?][4]\n  \u003e In Proceedings of the 12th Symposium on Dynamic Languages (DLS '16). ACM.\n\n#### Disclaimer: This is an Academic Project to Facilitate Research on Languages\n\nTo facilitate our research, we want to be able assess the\neffectiveness of compiler and runtime optimizations for a common set of\nabstractions between languages. As such, many other relevant aspects such as\nGC, standard libraries, and language-specific abstractions are not included\nhere. However, by focusing on this one aspect, we know exactly what is compared.\n\n## Current Status\n\nCurrently, we have 14 benchmarks ported to ten different languages, including\nC++, [Crystal], Java, JavaScript, Lua, Python, Ruby, [SOM Smalltalk][SOM], [SOMns][1] (a\n[Newspeak implementation][2]), and Smalltalk (Squeak/Pharo).\n\nThe graph below shows some older results for\ndifferent implementations after warmup, to ensure peak performance is reported:\n\n![Peak Performance of Java, Node.js, JRuby, JRuby+Truffle, MRI, and SOMns,\nlast update 2016-06-20](docs/figures/all-langs-overview-1.png?raw=true)\n\nA detailed overview of the results is in [docs/performance.md](docs/performance.md).\n\nThe benchmarks are listed below. A detailed analysis including metrics for the\nbenchmarks is in [docs/metrics.md](docs/metrics.md).\n\n#### Macro Benchmarks\n\n - [CD] is a simulation of an airplane collision detector. Based on\n   WebKit's JavaScript [CDjs]. Originally, CD was designed to evaluate\n   real-time JVMs.\n\n - [DeltaBlue] is a classic VM benchmark used to tune, e.g.,\n   Smalltalk, Java, and JavaScript VMs. It implements a constraint\n   solver.\n\n - [Havlak] implements a loop recognition algorithm. It has been used\n   to compare C++, Java, Go, and Scala performance.\n\n - [Json] is a JSON string parsing benchmark derived from the\n   `minimal-json` Java library.\n\n - [Richards] is a classic benchmark simulating an operating system\n   kernel. The used code is based on [Wolczko's Smalltalk\n   version][DeltaBlue].\n\n#### Micro Benchmarks\n\nMicro benchmarks are based on [SOM Smalltalk][SOM] benchmarks unless noted otherwise.\n\n - Bounce simulates a ball bouncing within a box.\n\n - List recursively creates and traverses lists.\n\n - Mandelbrot calculates the classic fractal. It is derived from the\n   [Computer Languages Benchmark Game][CLBG].\n\n - NBody simulates the movement of planets in the solar system. It is\n   derived from the [Computer Languages Benchmark Game][CLBG].\n\n - Permute generates permutations of an array.\n\n - Queens solves the eight queens problem.\n\n - Sieve finds prime numbers based on the sieve of Eratosthenes.\n\n - Storage creates and verifies a tree of arrays to stress the garbage\n   collector.\n\n - Towers solves the Towers of Hanoi game.\n\n\n## Contributing\n\nConsidering the large number of languages out there, we are open to\ncontributions of benchmark ports to new languages. We would also be interested\nin new benchmarks that are in the range of 300 to 1000 lines of code.\n\nWhen porting to a new language, please carefully consider [the\nguidelines](docs/guidelines.md) and description of [the *core*\nlanguage](docs/core-language.md) to ensure that we can compare results.\n\nA list of languages we would definitely be interested in is on the [issues\ntracker](https://github.com/smarr/are-we-fast-yet/issues?q=is%3Aissue+is%3Aopen+label%3A%22contribution+request%22).\n\nThis includes languages like Dart, Scala, and Go. Other interesting\nports could be for Racket, Clojure, or CLOS, but might require more carefully\nthought-out rules for porting. Similarly, a port to Rust need additional\ncare to account for the absence of a garbage collector and should be guided by our C++ port.\n\n## Getting the Code and Running Benchmarks\n\n### Quick Start\n\nTo obtain the code, benchmarks, and documentation, checkout the git repository:\n\n```bash\ngit clone --depth 1 https://github.com/smarr/are-we-fast-yet.git\n```\n\n#### Run Benchmarks for a Specific Language\n\nThe benchmarks are sorted by language in the [`benchmarks`](benchmarks) folder.\nEach language has its own harness. For JavaScript and Ruby, the benchmarks are\nexecuted like this:\n\n```bash\ncd benchmarks/JavaScript\nnode harness.js Richards 5 10\ncd ../Ruby\nruby harness.rb Queens 5 20\n```\n\nThe harness takes three parameters: benchmark name, number of iterations, and\nproblem size. The benchmark name corresponds to a class or file of a benchmark.\nThe *number of iterations* defines how often a benchmark should be executed. The\nproblem size can be used to influence how long a benchmark takes. Note that some\nbenchmarks rely on magic numbers to verify their results. Those might not be\nincluded for all possible problem sizes.\n\nThe [rebench.conf](rebench.conf#L31) file specifies the supported problem sizes\nfor each benchmark.\n\n### Using the Full Benchmark Setup\n\nEach port of the benchmarks comes with a `build.sh` file, which either runs any\nbuild steps needed, or with `./build.sh style` runs code style checks.\nThough, the repository does not contain setup steps for the various languages anymore.\nWe abandoned the idea of maintaining a full setup, since it took too much work.\n\nBenchmark are configured and executed with the\n[ReBench](https://github.com/smarr/ReBench) tool.\n\nReBench can be installed via the Python package manager pip:\n\n```\npip install ReBench\n```\n\nThe benchmarks can be executed with the following command in the root folder,\nassuming they have be previously built:\n\n```\nrebench -d --without-nice rebench.conf all\n```\n\nThe `-d` gives more output during execution, and `--without-nice` means that\nthe `nice` tool enforcing high process priority is not used. We don't use it\nhere to avoid requiring root rights.\n\n*Note:* The [rebench.conf](rebench.conf) file specifies how and which\nbenchmarks to execute. It also defines the arguments to be passed to the\nbenchmarks.\n\n## Academic Work using this benchmark suite\n\n- [Improving Native-Image Startup Performance](https://www.dag.inf.usi.ch/wp-content/uploads/cgo25.pdf)  \n  M. Basso, A. Prokopec, A. Rosà, W. Binder.\n  Proceedings of the 2025 IEEE/ACM International Symposium on Code Generation and Optimization, CGO 2025.\n\n- [Interactive Programming for Microcontrollers by Offloading Dynamic Incremental Compilation](https://doi.org/10.1145/3679007.3685062)  \n  F. Mochizuki, T. Yamazaki, S. Chiba.\n  Proceedings of the 21st ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes, MPLR 2024.\n\n- [Towards Realistic Results for Instrumentation-Based Profilers for JIT-Compiled Systems](https://stefan-marr.de/downloads/mplr24-burchell-et-al-towards-realistic-results-for-instrumentation-based-profilers-for-jit-compiled-systems.pdf)  \n  H. Burchell, O. Larose, S. Marr.\n  Proceedings of the 21st ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes, MPLR 2024.\n\n- [Taking a Closer Look: An Outlier-Driven Approach to Compilation-Time Optimization](https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ECOOP.2024.20)  \n  F. Huemer, D. Leopoldseder, A. Prokopec, R. Mosaner, H. Mössenböck.\n  Proceedings of the 38th European Conference on Object-Oriented Programming, ECOOP 2024.\n\n- [Live Objects All The Way Down: Removing the Barriers between Applications and Virtual Machines](https://stefan-marr.de/downloads/prog23-pimas-et-al-live-objects-all-the-way-down.pdf)  \n  J. E. Pimás, S. Marr, D. Garbervetsky.\n  The Art, Science, and Engineering of Programming, Programming'24.\n\n- [Don’t Trust Your Profiler: An Empirical Study on the Precision and Accuracy of Java Profilers](https://stefan-marr.de/downloads/mplr23-burchell-et-al-dont-trust-your-profiler.pdf)  \n  H. Burchell, O. Larose, S. Kaleba, S. Marr.\n  Proceedings of the 20th ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes, MPLR'23,\n\n- [AST vs. Bytecode: Interpreters in the Age of Meta-Compilation](https://stefan-marr.de/downloads/oopsla23-larose-et-al-ast-vs-bytecode-interpreters-in-the-age-of-meta-compilation.pdf)  \n  O. Larose, S. Kaleba, H. Burchell, S. Marr.\n  Proceedings of the ACM on Programming Languages, OOPSLA'23\n\n- [Collecting Cyclic Garbage across Foreign Function Interfaces: Who Takes the Last Piece of Cake?][19]  \n  T. Yamazaki, T. Nakamaru, R. Shioya, T. Ugawa, S. Chiba.\n  Proceedings of the ACM on Programming Languages, PLDI 2023.\n\n- [Simple Object Machine Implementation in a Functional Programming Language][20]  \n  Filip Říha. Bachelor's Thesis, CTU Prague, 2023.\n\n- [Supporting multi-scope and multi-level compilation in a\n   meta-tracing just-in-time compiler][23]  \n  Y. Izawa. PhD Dissertation. Tokyo Institute of Technology, 2023.\n\n- [Optimizing the Order of Bytecode Handlers in Interpreters using a Genetic Algorithm][25]  \n  W. Huang, S. Marr, T. Ugawa.\n  The 38th ACM/SIGAPP Symposium on Applied Computing, SAC 2023.\n\n- [Who You Gonna Call: Analyzing the Run-time Call-Site Behavior of Ruby Applications][21]  \n  S. Kaleba, O. Larose, R. Jones, S. Marr.\n  Proceedings of the 18th Symposium on Dynamic Languages, DLS 2022.\n\n- [Generating Virtual Machine Code of JavaScript Engine for Embedded Systems][18]  \n  Y. Hirasawa, H. Iwasaki, T. Ugawa, H. Onozawa. Journal of Information Processing. 2022.\n  \n- [Profile Guided Offline Optimization of Hidden Class Graphs for JavaScript VMs in Embedded Systems][24]  \n  T. Ugawa, S. Marr, R. Jones.\n  Proceedings of the 14th ACM SIGPLAN International Workshop on Virtual Machines and Intermediate Languages, VMIL 2022.\n\n- [Implementation Strategies for Mutable Value Semantics][27]  \n  D. Racordon, D. Shabalin, D. Zheng, D. Abrahams, B. Saeta.\n  Journal of Object Technology, 2022.\n\n- [Fusuma: Double-Ended Threaded Compaction][22]  \n  H. Onozawa, T. Ugawa, H. Iwasaki. \n  Proceedings of the 2021 ACM SIGPLAN International Symposium on Memory Management, ISMM 2021.\n\n- [A Surprisingly Simple Lua Compiler – Extended Version][17]  \n  H. M. Gualandi, R. Ierusalimschy. Journal of Computer Languages. 2021.\n\n- [Contextual Dispatch for Function Specialization][16]  \n  O. Flückiger, G. Chari, M. Yee, J. Ječmen, J. Hain, J. Vitek. Proceedings of the ACM on Programming Languages, OOPSLA 2020.\n\n- [GraalSqueak: Toward a Smalltalk-based Tooling Platform for Polyglot Programming][15]  \n  F. Niephaus, T. Felgentreff, R. Hirschfeld. Proceedings of 16th International Conference on Managed Programming Languages \u0026 Runtimes, MPLR 2019.\n  \n- [Scopes and Frames Improve Meta-Interpreter Specialization][26]  \n  V. Vergu, A. Tolmach, E. Visser. 33rd European Conference on Object-Oriented Programming, ECOOP 2019.\n\n- [Self-Contained Development Environments][14]  \n  G. Chari, J. Pimás, J. Vitek, O. Flückiger. Proceedings of the 14th ACM SIGPLAN International Symposium on Dynamic Languages. DLS 2018.\n  \n- [Interflow: Interprocedural Flow-Sensitive TypeInference and Method Duplication][13]  \n  D. Shabalin, M. Odersky. Proceedings of the 9th ACM SIGPLAN International Symposium on Scala 2018.\n\n- [Specializing a Meta-Interpreter: JIT Compilation of DynSem Specifications on the Graal VM][12]  \n  V. Vergu, E. Visser. Proceedings of the 15th International Conference on Managed Languages and Runtimes, ManLang 2018.\n\n- [Newspeak and Truffle: A Platform for Grace?][11]  \n  S. Marr, R. Roberts, J. Noble, Grace'18, p. 3, 2018. Presentation.\n\n- [Parallelization of Dynamic Languages: Synchronizing Built-in Collections][10]  \n  B. Daloze, A. Tal, S. Marr, H. Mössenböck, E. Petrank \n  Proceedings of the ACM on Programming Languages, OOPSLA 2018\n\n- [Efficient and Deterministic Record \u0026 Replay for Actor Languages][9]  \n  D. Aumayr, S. Marr, C. Béra, E. Gonzalez Boix, H. Mössenböck \n  Proceedings of the 15th International Conference on Managed Languages and Runtimes, ManLang 2018. \n\n- [Fully Reflective Execution Environments: Virtual Machines for More Flexible Software][8]  \n  G. Chari, D. Garbervetsky, S. Marr, S. Ducasse \n  IEEE Transactions on Software Engineering, IEEE TSE, p. 1–20, 2018.\n\n- [Garbage Collection and Efficiency in Dynamic Metacircular Runtimes][7]  \n  J. Pimás, J. Burroni, J., B. Arnaud, S. Marr \n  Proceedings of the 13th ACM SIGPLAN International Symposium on Dynamic Languages, DLS 2017.\n\n- [Applying Optimizations for Dynamically-typed Languages to Java][6]  \n  M. Grimmer, S. Marr, M. Kahlhofer, C. Wimmer, T. Würthinger, H. Mössenböck \n  Proceedings of the 14th International Conference on Managed Languages and Runtimes, ManLang 2017.\n\n- [Efficient and Thread-Safe Objects for Dynamically-Typed Languages][5]  \n  B. Daloze, S. Marr, D. Bonetta, Hanspeter Mössenböck\n  In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016.\n\n\n\n [1]: https://github.com/smarr/SOMns\n [2]: http://www.newspeaklanguage.org/\n [3]: https://stefan-marr.de/papers/dls-marr-et-al-cross-language-compiler-benchmarking-are-we-fast-yet/\n [4]: https://stefan-marr.de/downloads/dls16-marr-et-al-cross-language-compiler-benchmarking-are-we-fast-yet.pdf\n [5]: https://2016.splashcon.org/event/splash-2016-oopsla-efficient-and-thread-safe-objects-for-dynamically-typed-languages\n [6]: https://stefan-marr.de/downloads/manlang17-grimmer-et-al-applying-optimizations-for-dynamically-typed-languages-to-java.pdf\n [7]: https://stefan-marr.de/downloads/dls17-pimas-et-al-garbage-collection-and-efficiency-in-dynamic-metacircular-runtimes.pdf\n [8]: http://stefan-marr.de/downloads/tse18-chari-et-al-fully-reflective-execution-environments.pdf\n [9]: http://stefan-marr.de/downloads/manlang18-aumayr-et-al-efficient-and-deterministic-record-and-replay-for-actor-languages.pdf\n [10]: http://stefan-marr.de/downloads/oopsla18-daloze-et-al-parallelization-of-dynamic-languages-synchronizing-built-in-collections.pdf\n [11]: http://stefan-marr.de/downloads/grace18-marr-et-al-newspeak-and-truffle-a-platform-for-grace.pdf\n [12]: https://pure.tudelft.nl/portal/en/publications/specializing-a-metainterpreter(bc672ab9-d2bb-42d9-b67f-3f051868e3d7).html\n [13]: https://github.com/densh/talks/blob/master/2018-06-16-interflow-preprint-v1.pdf\n [14]: http://janvitek.org/pubs/dls18.pdf\n [15]: https://fniephaus.com/2019/mplr19-graalsqueak.pdf\n [16]: http://janvitek.org/pubs/oopsla20-cd.pdf\n [17]: http://www.inf.puc-rio.br/~hgualandi/papers/Gualandi-2022-SCP.pdf\n [18]: https://www.jstage.jst.go.jp/article/ipsjjip/30/0/30_679/_pdf\n [19]: https://dl.acm.org/doi/abs/10.1145/3591244\n [20]: https://dspace.cvut.cz/bitstream/handle/10467/109352/F8-BP-2023-Riha-Filip-thesis.pdf\n [21]: https://stefan-marr.de/downloads/dls22-kaleba-et-al-analyzing-the-run-time-call-site-behavior-of-ruby-applications.pdf\n [22]: https://dl.acm.org/doi/10.1145/3459898.3463903\n [23]: https://prg.is.titech.ac.jp/wp-content/uploads/2023/03/izawa-phd-dissertation.pdf\n [24]: https://stefan-marr.de/downloads/vmil22-ugawa-et-al-profile-guided-offline-optimization-of-hidden-class-graphs.pdf\n [25]: https://stefan-marr.de/downloads/acmsac23-huang-et-al-optimizing-the-order-of-bytecode-handlers-in-interpreters-using-a-genetic-algorithm.pdf\n [26]: https://drops.dagstuhl.de/opus/volltexte/2019/10796/pdf/LIPIcs-ECOOP-2019-4.pdf\n [27]: http://www.jot.fm/issues/issue_2022_02/article2.pdf\n [28]: https://github.com/smarr/are-we-fast-yet/blob/master/CITATION.bib\n\n [CD]:        https://www.cs.purdue.edu/sss/projects/cdx/\n [CDjs]:      https://github.com/WebKit/webkit/tree/master/PerformanceTests/JetStream/cdjs\n [DeltaBlue]: http://www.wolczko.com/java_benchmarking.html\n [Havlak]:    https://days2011.scala-lang.org/sites/days2011/files/ws3-1-Hundt.pdf\n [Json]:      https://github.com/ralfstx/minimal-json\n [Richards]:  http://www.cl.cam.ac.uk/~mr10/Bench.html\n [SOM]:       http://som-st.github.io/\n [CLBG]:      https://benchmarksgame-team.pages.debian.net/benchmarksgame/\n [Crystal]:   http://crystal-lang.org/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmarr%2Fare-we-fast-yet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmarr%2Fare-we-fast-yet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmarr%2Fare-we-fast-yet/lists"}