{"id":18718582,"url":"https://github.com/clasp-developers/cl-bench","last_synced_at":"2026-01-25T12:01:55.477Z","repository":{"id":204522000,"uuid":"712031343","full_name":"clasp-developers/cl-bench","owner":"clasp-developers","description":"Mirror of https://gitlab.common-lisp.net/ansi-test/cl-bench.git","archived":false,"fork":false,"pushed_at":"2023-10-30T17:35:24.000Z","size":257,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-12-28T10:48:03.778Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Common Lisp","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clasp-developers.png","metadata":{"files":{"readme":"README","changelog":"NEWS","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"support.lisp","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-30T16:50:05.000Z","updated_at":"2023-10-30T17:45:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"9ab84717-2e55-4eba-abc8-248fc6d0722c","html_url":"https://github.com/clasp-developers/cl-bench","commit_stats":null,"previous_names":["clasp-developers/cl-bench"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clasp-developers%2Fcl-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clasp-developers%2Fcl-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clasp-developers%2Fcl-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clasp-developers%2Fcl-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clasp-developers","download_url":"https://codeload.github.com/clasp-developers/cl-bench/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239581766,"owners_count":19662960,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T13:22:09.709Z","updated_at":"2025-11-10T15:30:16.101Z","avatar_url":"https://github.com/clasp-developers.png","language":"Common Lisp","funding_links":[],"categories":[],"sub_categories":[],"readme":"Common Lisp benchmarking suite\n==============================\n\nThis package contains Lisp code intended for performance benchmarking\nof different Common Lisp implementations. The tests it runs include\n\n  - the well-known Gabriel benchmarks, with a number of bugfixes from\n    Henry Baker and limited rewriting to bring them a little closer to\n    modern Common Lisp coding style\n  \n  - hashtable exercising\n  \n  - READ-LINE exercising\n  \n  - mathematical functions: factorial, fibonnaci, ackermann's number\n\n  - some bignum-intensive code from Bruno Haible\n    \n  - various aspects of CLOS: time taken to compile and execute\n    DEFCLASS forms which create a class hierarchy, instance creation\n    time, time taken to compile and execute DEFMETHOD forms, execution\n    of method combinations, both with and without :after methods. \n\n  - various operations on arrays, bitvectors and strings\n\n  - performance-intensive kernels such as CRC generation and an\n    implementation of the DEFLATE algorithm\n \nExcept for the CLOS COMPILER tests, timings do not include compilation\ntime. The garbage collector is run before each test to try to make the\ntimings more repeatable. For certain targets, we assume that the times\nreported by GET-INTERNAL-RUN-TIME and GET-INTERNAL-REAL-TIME are\naccurate. Timings for a given Common Lisp environment may be quite\nsensitive to the optimization settings; these are set at the beginning\nof the Makefile.\n\nCommon Lisp is a very large language, so it is difficult to evaluate\nthe performance of all aspects of an implementation. Remember that the\nonly real benchmark is your application: this code is only\nrepresentative of real-life programs to a limited extent.\n\nRunning the suite\n=================\n\nPut cl-bench directory in the asdf:*central-registry* or link it in\n~/quicklisp/local-projects/ directory and load the CL-BENCH. CL-BENCH\ndepends on the ALEXANDRIA and TRIVIAL-GARBAGE libraries, while\nCL-BENCH/REPORT depends additionally on CL-WHO.\n\n   (ql:quickload 'cl-bench)\n   (cl-bench:bench-run)\n\nBenchmark results will be in the output/ directory in the CL-BENCH\nsource tree directory.\n\nGenerating reports\n==================\n\nTo generate reports after running the tests on various\nimplementations, issue the following:\n\n   (ql:quickload 'cl-bench/report)\n   (cl-bench::bench-analysis-page)\n\nBenchmarks report will be generated to the output/report.html file in\nthe CL-BENCH source tree directory.\n\nRunning the suite (obsolete)\n============================\n\nEach implementation has a run-\u003cimpl\u003e.sh file, and a setup-\u003cimpl\u003e.lisp\nfile in the sysdep directory. To execute the test for CMUCL, for\nexampe, type\n\n   bash run-cmucl.sh\n\nThis will create files with the optimization setting requested in the\nMakefile, then invoke CMUCL to compile the files (lots of warnings and\ncompilation notes are to be expected), then run the various tests.\nExecution takes quite a while: around 30 minutes on a 1GHz PIII. A\nlarge part of this time is due to the bignum benchmarks, which you can\ncomment out if they don't interest you. You can also reduce the number\nof executions of each particular test by editing the file \"tests.lisp\"\n-- modify the :runs parameter of the DEFBENCH forms in the file\n\"tests.lisp\". For the results to be meaningful, the time for each test\nshould be more than a second. However, when comparing implementations\nit is not the absolute time which counts, but the time relative to the\nother implementations. Also note that you may wish to reduce the size\nof the problems, particularly in the array sections, because with some\nsetups (CMUCL on Linux for example), your system may become unusable\nif you exceed available RAM. You can do this by editing the file\n\"tests.lisp\" and modifying the arguments to the :code parameters.\n\nRepeat this operation for other implementations you have on your\nmachine. I have tried it with CMUCL, SBCL, CLISP, OpenMCL, Poplog\nCL and LispWorks on various platforms. GCL and ECL are able to run\nsome of the tests.\n\nIf you're not running on a Unix platform, you may not be able to\nscript the different steps of a run. In this case, follow the\nfollowing steps:\n\n   1. Say \"make clean optimize-lisp\" to create source files that contain\n      optimization declarations. The optimize settings can be changed\n      in the Makefile. \n\n   2. Load the file \"generate.lisp\" into your implementation. This\n      should create two files \"do-compilation-script.lisp\" and\n      \"do-execute-script.lisp\", that contain a sequence of compilation\n      and load steps.\n\n   3. Load the file \"sysdep/setup-\u003cyourimpl\u003e.lisp\", which you may need\n      to write yourself. This requires a definition of a BENCH-GC\n      function that triggers a full garbage collection.\n\n   4. Load the file \"do-compilation-script.lisp\" into your\n      implementation, which should result in all the source files\n      being compiled.\n\n   5. Load the file \"do-execute-script.lisp\", which should cause all\n      the tests to be executed. \n\nFor each tested implementation, you should have a file in output/\nnamed \"CL-benchmark-\u003cdate\u003e\".  These files will have the following\nformat:\n\n,---- /var/tmp/CL-benchmark-20010821T2208 ---\n| ;; -*- lisp -*-  CMU Common Lisp CVS sources, level-1 built 2001-08-22 on maftia1\n| ;;\n| ;; Implementation *features*:\n| ;; (:PCL-STRUCTURES :PORTABLE-COMMONLOOPS :PYTHON :PCL :COMPLEX-FP-VOPS :PPRO\n| ;;  :PENTIUM :MP :HASH-NEW :RANDOM-MT19937 :PROPAGATE-FUN-TYPE\n| ;;  :PROPAGATE-FLOAT-TYPE :CONSTRAIN-FLOAT-TYPE :CMU18C :CMU18 :GENCGC :X86\n| ;;  :LINUX :GLIBC2 :UNIX :COMMON :COMMON-LISP :ANSI-CL :CMU\n| ;;  :IEEE-FLOATING-POINT)\n| ;;\n| ;; Function                      real     user     sys       consed\n| ;; ----------------------------------------------------------------\n| ;; Boyer                         1.50     1.22     0.28     54349520\n| ;; Browse                        0.97     0.79     0.18     36219256\n| ;; DDerviv                       0.88     0.50     0.39     67197656\n| ;; Deriv                         1.64     0.87     0.77    127195824\n| ;; Destructive                   0.30     0.24     0.05     12819928\n| ;; div2-test-1                   0.52     0.32     0.20     38398176\n| ;; div2-test2                    0.66     0.42     0.24     47999936\n| ;; FFT                           0.40     0.40     0.00            0\n| ;; frpoly/fixnum                 0.65     0.54     0.10     19172440\n| ;; frpoly/bignum                 1.54     1.25     0.28     55628704\n| ;; frpoly/float                  7.59     6.50     1.09    213052408\n| ;; Puzzle                        0.82     0.82     0.00            0\n| ;; CTak                          0.81     0.81     0.00            0\n| ;; Tak                           0.53     0.54     0.00            0\n| ;; RTak                          0.35     0.36     0.00            0\n| ;; takl                          1.60     1.60     0.00            0\n| ;; stak                          1.15     1.14     0.00            0\n| ;; fprint                        0.25     0.25     0.01      1948416\n| ;; fread                         0.82     0.68     0.13     28487280\n| ;; traverse                      5.28     5.24     0.03      4493288\n| ;; triangle                      2.29     2.28     0.00       499712\n| ;; factorial                     0.37     0.18     0.20     26120296\n| ;; fib                           2.39     2.39     0.00            0\n| ;; hashtable                     0.72     0.69     0.04      9888912\n| ;; CLOS/defclass                 2.82     2.31     0.12     32757328\n| ;; CLOS/defmethod               10.94    10.09     0.55    120612624\n| ;; CLOS/instantiate              7.13     6.27     0.86    229048352\n| ;; CLOS/methodcalls              6.56     5.22     1.34    301057608\n| ;; CLOS/method+after            12.02    11.09     0.93    197058816\n| ;; CLOS/complex-methods          0.38     0.38     0.00       286600\n| ;; 1D-arrays                     2.46     2.17     0.30     60002400\n| ;; 2D-arrays                    20.57    19.50     1.07    240000240\n| ;; bitvectors                   18.75    18.51     0.23     50003200\n| ;; fill-strings                 21.12    15.23     5.88   1000016000\n| ;; fill-strings/adjustable      57.10    56.25     0.85    259729520\n| \n| (\"CMU Common Lisp CVS sources, level-1 built 2001-08-22 on maftia1\"\n|  (\"fill-strings/adjustable\" 57.1 56.25 0.85 259729520)\n|  (\"fill-strings\" 21.12 15.23 5.88 1000016000)\n|  (\"bitvectors\" 18.75 18.51 0.23 50003200)\n|  (\"2D-arrays\" 20.57 19.5 1.07 240000240) (\"1D-arrays\" 2.46 2.17 0.3 60002400)\n|  (\"CLOS/complex-methods\" 0.38 0.38 0.0 286600)\n|  (\"CLOS/method+after\" 12.02 11.09 0.93 197058816)\n|  (\"CLOS/methodcalls\" 6.56 5.22 1.34 301057608)\n|  (\"CLOS/instantiate\" 7.13 6.27 0.86 229048352)\n|  (\"CLOS/defmethod\" 10.94 10.09 0.55 120612624)\n|  (\"CLOS/defclass\" 2.82 2.31 0.12 32757328) (\"hashtable\" 0.72 0.69 0.04 9888912)\n|  (\"fib\" 2.39 2.39 0.0 0) (\"factorial\" 0.37 0.18 0.2 26120296)\n|  (\"triangle\" 2.29 2.28 0.0 499712) (\"traverse\" 5.28 5.24 0.03 4493288)\n|  (\"fread\" 0.82 0.68 0.13 28487280) (\"fprint\" 0.25 0.25 0.01 1948416)\n|  (\"stak\" 1.15 1.14 0.0 0) (\"takl\" 1.6 1.6 0.0 0) (\"RTak\" 0.35 0.36 0.0 0)\n|  (\"Tak\" 0.53 0.54 0.0 0) (\"CTak\" 0.81 0.81 0.0 0) (\"Puzzle\" 0.82 0.82 0.0 0)\n|  (\"frpoly/float\" 7.59 6.5 1.09 213052408)\n|  (\"frpoly/bignum\" 1.54 1.25 0.28 55628704)\n|  (\"frpoly/fixnum\" 0.65 0.54 0.1 19172440) (\"FFT\" 0.4 0.4 0.0 0)\n|  (\"div2-test2\" 0.66 0.42 0.24 47999936) (\"div2-test-1\" 0.52 0.32 0.2 38398176)\n|  (\"Destructive\" 0.3 0.24 0.05 12819928) (\"Deriv\" 1.64 0.87 0.77 127195824)\n|  (\"DDerviv\" 0.88 0.5 0.39 67197656) (\"Browse\" 0.97 0.79 0.18 36219256)\n|  (\"Boyer\" 1.5 1.22 0.28 54349520))\n`----\n\nThe first section of the file is intended to be human readable, and\nthe second section to be READ by a Common Lisp implementation. For\neach test, you should see the elapsed user time, and possibly (if this\nhas been coded for your implementation) elapsed system time and the\nnumber of bytes consed during the test execution. \n\nThe data in the different output/CL-benchmark-* files is analysed by\nthe file \"report.lisp\", to generate a report comparing the performance\nof the different implementations. This file needs to be run in a\nCommon Lisp implementation; the one you use will be considered the\n\"reference\" implementation. In the report which is generated, for each\ntest the timing for the reference implementation will be shown, as\nwell as the _relative times_ for each of the other tested\nimplementations. A relative time means that a score under 1 is better,\nand a score of 2 means it is two times slower -- for that test -- than\nthe reference implementation. If a given test doesn't work in a\nparticular implementation (for example CLISP doesn't do non-standard\nmethod combination), its entry will be -1.0.\n\nHere is an example of the type of results you can obtain, for x86 and\nSPARC: \n\n,---- PentiumIII at 1GHz, 256MB RAM, Linux 2.4.2 ---\n|\n| Benchmark                 Reference  CLISP  CMU C  SBCL \n| ----------------------------------------------------------------\n| BOYER                          2.36   4.54   0.67   0.94\n| BROWSE                         1.04   2.15   0.65   1.04\n| DDerviv                        1.19   1.96   0.48   1.06\n| Deriv                          2.27   1.93   0.42   1.04\n| DESTRUCTIVE                    1.52   2.79   0.89   1.06\n| DIV2-TEST-1                    1.73   1.84   0.51   1.09\n| DIV2-TEST-2                    0.85   1.87   0.46   1.12\n| FFT                            0.22  36.09   1.14   1.14\n| FRPOLY/FIXNUM                  0.79   5.81   0.81   0.96\n| FRPOLY/BIGNUM                  1.99   2.03   0.68   0.96\n| FRPOLY/FLOAT                   0.78   3.79   0.72   0.99\n| PUZZLE                         0.79  23.09   1.15   9.95\n| CTAK                           0.86   6.28   1.10   1.06\n| TAK                            0.91   8.86   1.18   1.34\n| RTAK                           0.91   8.86   1.13   1.34\n| TAKL                           1.67   7.33   1.16   1.22\n| STAK                           1.15   6.66   1.15   1.10\n| FPRINT                         1.42   1.12   1.05   2.37\n| TRAVERSE                       4.35   6.75   1.19   1.64\n| TRIANGLE                       2.01  17.22   1.14   1.27\n| CASCOR                         4.06  80.47   1.23   0.92\n| RICHARDS                       0.58  24.78   1.22   1.07\n| FACTORIAL                      0.50   3.56   0.68   1.38\n| FIB                            0.39   5.67   1.13   1.08\n| BIGNUM/ELEM-100-1000           1.33   0.11   1.02   0.98\n| BIGNUM/ELEM-1000-100           6.03   0.07   1.11   0.93\n| BIGNUM/ELEM-10000-1            6.14   0.05   1.00   0.93\n| BIGNUM/PARI-100-10             1.51   0.06   0.96   0.91\n| BIGNUM/PARI-200-5             17.32   0.02   0.99   0.88\n| HASH-STRINGS                   1.65   3.00   0.82   0.99\n| HASH-INTEGERS                  0.73   2.30   0.68   1.05\n| BOEHM-GC                      10.18   3.94   0.39   1.19\n| CLOS/defclass                  2.64   0.35   0.83   2.33\n| CLOS/defmethod                13.93   0.02   0.70   1.94\n| CLOS/instantiate               7.23   1.02   0.63   1.39\n| CLOS/methodcalls               8.43   1.33   0.50   1.37\n| CLOS/method+after             13.90   0.45   0.79   2.64\n| CLOS/complex-methods           0.35  -1.00   1.06   5.40\n| 1D-ARRAYS                      3.74   4.57   0.69   1.26\n| 2D-ARRAYS                     15.19   3.25   0.92   4.00\n| BITVECTORS                     2.92   0.61   0.70   0.95\n| FILL-STRINGS                  10.62   1.15   0.76   3.03\n| fill-strings/adjustable       18.13   1.08   0.92   1.65\n| BENCH-STRING-CONCAT            9.99  -1.00   0.81   2.34\n| \n| Reference implementation: CMU Common Lisp 18c\n| Impl CLISP: CLISP 2.27 (released 2001-07-17) (built 3204291355) (memory 3205854943)\n| Impl CMU C: CMU Common Lisp 18d-pre, level-1 built 2001-12-14 on maftia1\n| Impl SBCL : SBCL 0.7.0\n| Linux maftia1 2.4.2-2 #1 Sun Apr 8 20:41:30 EDT 2001 i686 unknown\n`----\n\n\n,---- UltraSPARCIIe at 500MHz, 640MB RAM, SunOS 5.8 ---\n|\n| Benchmark                 Reference  CMU C  CLISP\n| -----------------------------------------------------\n| BOYER                          3.98   0.91   8.03\n| BROWSE                         1.72   0.91   2.85\n| DDerviv                        2.02   0.75   3.21\n| Deriv                          3.63   0.81   3.13\n| DESTRUCTIVE                    3.11   1.01   4.18\n| DIV2-TEST-1                    2.19   0.83   3.92\n| DIV2-TEST-2                    1.12   0.82   3.85\n| FFT                            0.74   1.03  28.86\n| FRPOLY/FIXNUM                  1.87   1.01   7.89\n| FRPOLY/BIGNUM                  4.59   1.29   3.07\n| FRPOLY/FLOAT                   1.65   0.96   5.68\n| PUZZLE                         2.07   0.95  30.62\n| CTAK                           2.74   1.01   9.04\n| TAK                            1.84   1.00  14.08\n| RTAK                           1.84   1.01  13.95\n| TAKL                           3.37   1.01  11.63\n| STAK                           2.32   1.01   8.87\n| FPRINT                         4.17   1.02   1.12\n| TRAVERSE                       5.84   0.99  13.74\n| TRIANGLE                       5.53   0.86  15.57\n| CASCOR                        10.53   0.73  52.81\n| RICHARDS                       2.35   0.94  22.46\n| FACTORIAL                      1.46   1.48   2.88\n| FIB                            0.94   0.99   6.71\n| BIGNUM/ELEM-100-1000           2.80   1.24   0.28\n| BIGNUM/ELEM-1000-100          10.14   1.19   0.44\n| BIGNUM/ELEM-10000-1           11.38   1.35   0.41\n| BIGNUM/PARI-100-10             2.76   1.15   0.09\n| BIGNUM/PARI-200-5             27.19   1.06   0.05\n| READ-LINE                      3.39   1.06   1.19\n| HASH-STRINGS                   5.42   1.20   2.19\n| HASH-INTEGERS                  1.61   0.76   2.00\n| BOEHM-GC                      19.97   0.76   4.14\n| CLOS/defclass                  4.78   1.01   0.81\n| CLOS/defmethod                27.61   0.89   0.03\n| CLOS/instantiate              20.93   0.85   1.28\n| CLOS/methodcalls              23.62   1.08   1.94\n| CLOS/method+after             33.70   1.07   0.78\n| CLOS/complex-methods           1.41   0.92  -1.00\n| 1D-ARRAYS                     10.77   0.92   3.51\n| 2D-ARRAYS                     56.66   1.40   2.61\n| BITVECTORS                     5.35   0.86   0.42\n| FILL-STRINGS                  18.88   1.07   0.97\n| fill-strings/adjustable       45.09   1.46   1.41\n| BENCH-STRING-CONCAT           48.10   0.90  -1.00\n| \n| Reference implementation: CMU Common Lisp 18c, Built 2000-11-27\n| Impl CMU C: CMU Common Lisp 18d-pre, level-1 built 2001-12-12 on liszt\n| Impl CLISP: CLISP 2.27.2 (released 2001-10-05) (built on moustacho)\n| SunOS eagles 5.8 Generic_108528-10 sun4u sparc SUNW,Sun-Blade-100\n`----\n\n\n\nNote that the test suite doesn't take compilation time into account\n(except for the CLOS-related tests, where the compiler may be used at\nruntime to create effective methods). You can use the time taken to\nload do-compilation-script.lisp as a rough proxy for compilation time.\n\n\n\n   \"Life is short and it was not meant to be spent making people feel guilty\n    about instruction pipelines being only partly full or caches being missed.\"\n     -- Kent Pitman in \u003csfw7ksm3b7k.fsf@shell01.TheWorld.com\u003e\n\n\nThanks\n======\n\nRaymond Toy, Christophe Rhodes, Peter Van Eynde, Sven Van\nCaekenberghe, Christophe Rhodes, Kevin Layers and possibly others that\nI have forgotten to note.\n\n\n\n\nRelated work\n============\n\n  -  @misc{ gabriel86performance,\n      author = \"R. Gabriel\",\n      title = \"Performance and Evaluation of Lisp Systems\",\n      text = \"R. P. Gabriel. Performance and Evaluation of Lisp Systems. MIT Press, Cambridge,\n        Massachusetts, 1986.\",\n      year = \"1986\" }\n\n  - Scheme benchmarks by Will Clinger (Larceny and TwoBit compilers)\n    \u003cURL:http://www.ccs.neu.edu/home/will/GC/sourcecode.html\u003e\n\n  - Bagley's Programming Language Shootout,\n    \u003cURL:http://www.bagley.org/~doug/shootout/\u003e\n\n \n\nEric Marsden \u003cemarsden@laas.fr\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclasp-developers%2Fcl-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclasp-developers%2Fcl-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclasp-developers%2Fcl-bench/lists"}