{"id":20153590,"url":"https://github.com/seqan/bench","last_synced_at":"2025-04-09T21:33:12.897Z","repository":{"id":29271081,"uuid":"32803786","full_name":"seqan/bench","owner":"seqan","description":"SeqAn Benchmarks","archived":false,"fork":false,"pushed_at":"2017-03-03T09:39:05.000Z","size":6362,"stargazers_count":3,"open_issues_count":0,"forks_count":4,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-23T23:26:55.356Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/seqan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-03-24T14:29:04.000Z","updated_at":"2020-09-10T06:21:10.000Z","dependencies_parsed_at":"2022-08-29T19:31:29.100Z","dependency_job_id":null,"html_url":"https://github.com/seqan/bench","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seqan%2Fbench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seqan%2Fbench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seqan%2Fbench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seqan%2Fbench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/seqan","download_url":"https://codeload.github.com/seqan/bench/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248114964,"owners_count":21050146,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T23:19:49.581Z","updated_at":"2025-04-09T21:33:12.866Z","avatar_url":"https://github.com/seqan.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"Seqan Bench App\n===============\n\nIntroduction\n------------\n\nThis application provides a graphical user interface for benchmarks of\nbiological problems. The benchmarks are pre-implemented by using the SeqAn\nlibrary, but can be exchanged to test the performance of other libraries.\n\nDownload\n--------\n\nExecutables for Linux, OS X and Windows are available\n[Here](https://github.com/marehr/bench/releases).\n\nUsage\n-----\n\nUnpack (howto on\n[Windows](https://wiki.haskell.org/How_to_unpack_a_tar_file_in_Windows)) the\ndownloaded file and execute `seqan-bench-app[.exe]`. The application will\nautomatically download the needed data and preprocess it.\n\nPress run to start the benchmarks.\n\n![image](./resources/help/app/screenshot1.png)\n\nAs you can see, the app shows the current OS, Memory, CPU frequency (can be used\nas an indicator if the CPU frequency was correctly fixed), the number of threads\nand a project title.\n\nThe project title is intended to give the measured benchmark-results a\nrecognizable name. In this example the title is `seqan-2.1-gcc-linux`,\nindicating that the benchmark was run on Linux using the SeqAn library 2.1 and\nGCC compiler.\n\n![image](./resources/help/app/screenshot2.png)\n\n✔ complete: means that the benchmark executed without error\n\n✔ output: means that the output of the benchmark was as specified. This check\nwill be done by a validator (see below if you want to know how this works).\n\n![image](./resources/help/app/screenshot3.png)\n\nAfter the benchmark was executed, there is a summary how much time it took and\nsave buttons to save the results into a json file or to generate a website.\nClicking on `seqan-2.1-gcc-linux` in the summary will expand the results again.\n\n![image](./resources/help/app/screenshot4.png)\n\nThis is an example how a generated website looks like. It will give you a score.\n1000 means same performance as our test system. Less than 1000 means your\nbenchmark/system was slower than our test system, more than 1000 means that your\nbenchmark/system was faster. A score of 2000, 3000, ... means that your\nbenchmark/system is 2, 3, ... times fast than our test system.\n\nOur current test system uses a `Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz` CPU\nfixed by [cpufrequtils](http://www.thinkwiki.org/wiki/How_to_use_cpufrequtils)\nat 2.3 GHz. We might change the base frequency to 1.9 GHz in the future, because\nthat is the base frequency of AVX2 on this CPU.\n\n### Start benchmark app headless\n\n``` terminal\n# in the folder where seqan-bench-app[.exe] is\n\n# using the app to launch the headless version\n./seqan-bench-app[.exe] bench_cli.js\n\n# or with node directly\nnode bench_cli.js\n```\n\nWill start a headless benchmark.\n\nSee `./seqan-bench-app[.exe] bench_cli.js --help` (or `node bench_cli.js --help`) for more information:\n\n``` terminal\nUsage: node bench_cli.js -tc [THREADS] -c [CONFIG] -o [OUTPUT] --html [HTML]\n\nOptions:\n  --version      Show version number                                             [boolean]\n  -v, --verbose  be verbose                                                      [boolean]\n  -t, --threads  The number of threads the benchmark should use in the multi-core run.\n                                                                     [number] [default: 4]\n  -c, --config   CONFIG file that defines the benchmarks.\n                                                  [string] [default: \"config/config.json\"]\n  -o, --output   Write results to OUTPUT.\n                                        [string] [default: \"results-20160601-140243.json\"]\n  --merge        Merge results given by `./bench_cli --output result(1..5).json`.\n                 Usage: `./bench_cli --merge result1.json [result2.json..] --output\n                 merged_result.json --html merged_website`                         [array]\n  --html         Generate static website to HTML, that shows the results.         [string]\n  -h, --help     Show help                                                       [boolean]\n\nExamples:\n  node bench_cli.js -c config.json               Run benchmarks defined in config.json and\n       -o results.json                           write results to results.json\n```\n\nBenchmarks\n----------\n\nCurrently, we define the following benchmarks:\n\n-   indices\n    -   [exact search](\n        https://github.com/marehr/bench/wiki/index_exact_search)\n    -   [approximate search, 1 error with hamming distance](\n        https://github.com/marehr/bench/wiki/index_one_error_approximate_search)\n    -   [approximate search, 2 error with hamming distance](\n        https://github.com/marehr/bench/wiki/index_three_errors_approximate_search)\n    -   [approximate search, 3 error with hamming distance](\n        https://github.com/marehr/bench/wiki/index_two_errors_approximate_search)\n-   k-mers\n    -   [counting 10-mers](\n        https://github.com/marehr/bench/wiki/kmers_10mer_counting)\n    -   [counting 15-mers](\n        https://github.com/marehr/bench/wiki/kmers_15mer_counting)\n    -   [counting 50-mers](\n        https://github.com/marehr/bench/wiki/kmers_50mer_counting)\n-   pairwise alignments\n    -   [global alignment, linear gap model on DNA alphabet](\n        https://github.com/marehr/bench/wiki/pairwise_alignment_global_affine_dna)\n    -   [global alignment, linear gap model on amino acid alphabet](\n        https://github.com/marehr/bench/wiki/pairwise_alignment_global_affine_protein)\n    -   [global alignment, affine gap model on DNA alphabet](\n        https://github.com/marehr/bench/wiki/pairwise_alignment_global_linear_dna)\n    -   [global alignment, affine gap model on amino acid alphabet](\n        https://github.com/marehr/bench/wiki/pairwise_alignment_global_linear_protein)\n    -   [local alignment, linear gap model on DNA alphabet](\n        https://github.com/marehr/bench/wiki/pairwise_alignment_local_affine_dna)\n    -   [local alignment, linear gap model on amino acid alphabet](\n        https://github.com/marehr/bench/wiki/pairwise_alignment_local_affine_protein)\n    -   [local alignment, affine gap model on DNA alphabet](\n        https://github.com/marehr/bench/wiki/pairwise_alignment_local_linear_dna)\n    -   [local alignment, affine gap model on amino acid alphabet](\n        https://github.com/marehr/bench/wiki/pairwise_alignment_local_linear_protein)\n\nBuilding from source\n--------------------\n\n### Requirements\n\n-   [cmake](https://cmake.org) (≥ 3.0),\n-   [Node.js](https://nodejs.org/) (≥ 4.2),\n    -   `npm install` in the source directory will install all required\n        `node_modules`.\n-   [python](https://nodejs.org/) (both python ≥ 2.7 and python ≥ 3.4 work),\n    -   For windows packaging `pip install py2exe` is needed\n    -   Note: At the time of writing this readme, py2exe required python 3.4 to\n        work.\n-   [NW.js](http://nwjs.io/) (≥ 14.5) and\n-   the [SeqAn](https://github.com/seqan/seqan) (≥ 2.0) library.\n\n### Prepare build\n\n(On windows you could use this terminal emulator\n\u003chttps://git-for-windows.github.io/\u003e)\n\nDownload source code\n\nNote: We use git submodule to seperate our benchmarks from the actual tool\nso you need to clone the repository recursively (Git version \u003e 1.6.5)\n\n``` terminal\ngit clone --recursive https://github.com/seqan/bench.git bench_app\ncd bench_app\n```\n\nDownload seqan source code\n\n``` terminal\ngit clone https://github.com/seqan/seqan.git\n```\n\nInstall needed node modules\n\n``` terminal\nnpm install\n```\n\nInstall NW.js\n\n``` terminal\nwget http://dl.nwjs.io/v0.14.5/nwjs-sdk-v0.14.5-linux-x64.tar.gz\ntar xfvz nwjs-sdk-v0.14.5-linux-x64.tar.gz\n```\n\n### Build benchmarks\n\n``` terminal\n# in bench/bench_app\ncd benchmarks\n\ncmake ../benchmarks_src \\\n    -DCMAKE_MODULE_PATH=../seqan/util/cmake/ \\\n    -DCMAKE_INCLUDE_PATH=../seqan/include/ \\\n    -DCMAKE_RUNTIME_OUTPUT_DIRECTORY=. \\\n    -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE=.\n\ncmake --build . --config Release\n\n# on linux/osx you can speed up the build with -j \u003cthreads\u003e\ncmake --build . --config Release -- -j 4\n```\n\n### Build validators\n\n(On Windows install [py2exe](http://www.py2exe.org/), i.e `pip install py2exe`)\n\n``` terminal\n# in bench/bench_app\ncd validators\n\ncmake ../validators_src \\\n    -DCMAKE_MODULE_PATH=../seqan/util/cmake/ \\\n    -DCMAKE_INCLUDE_PATH=../seqan/include/ \\\n    -DCMAKE_RUNTIME_OUTPUT_DIRECTORY=. \\\n    -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE=.\n\ncmake --build . --config Release\n\n# on linux/osx you can speed up the build with -j \u003cthreads\u003e\ncmake --build . --config Release -- -j 4\n```\n\n### Start benchmark app\n\n``` terminal\n# in bench/bench_app\n./nwjs-sdk-v0.14.5-linux-x64/nw .\n```\n\n(Make sure you run `npm install`)\n\nYou can debug the gui by right click and `inspect`.\n\n\u003e **Note**\n\u003e\n\u003e On Windows you need to change the extension of the executables in\n\u003e\n\u003e -   `config/config.json`\n\u003e     -   all `.benchmarks.\u003cbenchmark_id\u003e.command`, e.g. replace:\n\u003e\n\u003e             \"./benchmarks/index_exact_search [...]\" by\n\u003e             \"./benchmarks/index_exact_search.exe [...]\"\n\u003e\n\u003e -   `resources/config/validators.json`\n\u003e     -   all `.\u003cvalidator_id\u003e`, e.g. replace:\n\u003e\n\u003e             \"index_search_hamming_distance_validator.py\" by\n\u003e             \"index_search_hamming_distance_validator.exe\"\n\u003e\n\n### Package benchmark app\n\nTo ship the build and upload it to github, you can use `package.js` which will\ndo the above steps automatically.\n\n``` terminal\n# in bench/bench_app\nnode package.js\n```\n\nThis will create `seqan-bench-app-linux-x64.tar.gz` on linux.\n\nNew benchmark\n-------------\n\n### Benchmark source code\n\nThe source code of the benchmarks are in `benchmarks_src`. A new benchmark can\nbe added to `benchmarks_src/CMakeLists.txt`. The benchmarks should follow the\nfolder structure, that means that the pairwise alignment benchmarks should be\nadded to the `benchmarks_src/pairwise_alignments` folder.\n\nSee below in the Benchmark config section, how the benchmark is expected to be\nexecuted.\n\nThe runtime of a benchmark should be between 2 and 15 minutes. There are some\nexceptions, if the required space would be to much, e.g. like in the case of\nindex\\_exact\\_search.\n\n### Benchmark id\n\nA benchmark should get a unique id, which has the prefix of the containing\nfolder. For example the benchmark for a global pairwise alignment using affine\ngaps cost for the dna alphabet has the id\n`pairwise_alignment_global_affine_dna`, because it is within the\n`pairwise_alignments` folder.\n\n### Benchmark config / How a benchmark will be called\n\nThe benchmark must be added to the `config/config.json`.\n\n``` javascript\n{\n    \"project\": {\n        \"title\": \"seqan-2.1\"\n    },\n\n    \"benchmarks\": {\n        //[...],\n        \"pairwise_alignment_global_affine_dna\": {\n            \"execute\": true,\n            \"command\": \"./benchmarks/pairwise_alignment_global_affine_dna \u003cinput parameters\u003e\",\n            \"expected_runtime\": 245.53,\n            \"repeats\": 1\n        }\n    }\n}\n```\n\n-   execute: whether the benchmark should be run\n-   command: the benchmark executable (relative to the bench app) and the needed\n    input parameters\n    -   The bench app will automatically append `\u003coutput file\u003e -tc \u003cthreads\u003e`.\n        Your benchmark must write the output into the `\u003coutput file\u003e` and only\n        use at most `\u003cthreads\u003e` threads.\n\n        For example: A one-thread execution of\n        pairwise\\_alignment\\_global\\_affine\\_dna, where \u0026lt;input parameters\u0026gt;\n        is `./data/genome.pairwise_alignment.reads.length.100.fa`, will be\n        called as\n\n        ``` terminal\n        ./benchmarks/pairwise_alignment_global_affine_dna ./data/genome.pairwise_alignment.reads.length.100.fa results/pairwise_alignment_global_affine_dna.single_core.txt -tc 1\n        ```\n\n-   expected\\_runtime: The runtime you measured on your system. This will be\n    used to indicate the progress of the benchmarks to the users.\n-   repeats: The number of repeated measurements. Should be one.\n\n\nThe benchmark must also be added to `resources/config/descriptions.json`.\n\n``` javascript\n{\n    \"categories\": {\n        \"index_search\": {\n            \"title\": \"Index based exact/approximate search\"\n        },\n        \"kmer_counting\": {\n            \"title\": \"Counting k-mers\"\n        },\n        \"pairwise_alignment\": {\n            \"title\": \"Pairwise Alignment\"\n        }\n    },\n\n    \"benchmarks\": {\n        // [...]\n        \"pairwise_alignment_global_affine_dna\": {\n            \"title\": \"DNA global Alignment, affine gap costs\",\n            \"subtitle\": \"match: 2, mismatch: -3, gap_extend: -3, gap_open: -1\",\n            \"base_time\": 245.53,\n            \"category\": \"pairwise_alignment\"\n        }\n    }\n}\n```\n\nIf you created a new category (i.e. created a new subfolder in\n`benchmarks_src/`) add it to `categories`. The `title` in `categories` gives\nthe name of the tab in the app and website.\n\n-   title: is a short description what the benchmark does\n-   subtitle: gives additional information, like the configuration\n-   base\\_time: is the same as the expected\\_runtime above, this value will be\n    used to calculate the score of the generated website. (Should be measured on\n    our test system).\n-   category: is the category\\_id the benchmark belongs to (from `categories`)\n\n### Benchmark validation\n\nThe output of the benchmark needs to be validated. If an existing validator can\nalready validate your output, add it to the validators list, otherwise add a new\nvalidator (see next section).\n\nAdd the benchmark to the validator list in `resources/config/validators.json`\n\n``` javascript\n{\n    // [...]\n    \"pairwise_alignment_validator.py\": [\n        // [...],\n        \"pairwise_alignment_global_affine_dna\"\n    ]\n}\n```\n\nA validator will be called by\n\n-   `./results/\u003cbenchmark_id\u003e[.\u003crepeat\u003e].(single|multi)_core.result.txt` and\n-   `./data/validations/\u003cbenchmark_id\u003e.validate.txt`.\n\nThus `./data/validations/\u003cbenchmark_id\u003e.validate.txt` must be provided. In the\nbest case you have an independent reference implementation that provides you the\nreference output. But you can also copy the result from your benchmark.\n\n\u003e **Note**\n\u003e\n\u003e The source code has a different `./data` content than the one you can\n[download](https://github.com/marehr/bench/releases) (data.tar.gz). You also\nmust provide validation data for the data set on github. In this case, the\n`dataUrl` in `./resources/config/app_config.json` must be updated to the next\nrelease. This url must also be updated, if you add new data for a benchmark.\n\n### Update github wiki\n\nAdditionally, a description/specification of the benchmark should be added to\nthe github wiki where the page should be the benchmark id. As an example,\n\u003chttps://github.com/marehr/bench/wiki/pairwise_alignment_global_affine_dna\u003e.\n\nNew validator\n-------------\n\n### Validator source code\n\nThe current validators are written in python, but can also be written in C++.\nThe source of validators reside in `./validators_src`. A small example for a\nbenchmark and a reference output should be added into `./validators_src`.\n\n### Validator id\n\nThe validator id is simply the name of the executable. As an example, the\nvalidator id of `./validators/index_exact_hamming_distance.py` is\n`index_exact_hamming_distance.py`.\n\n### How a validator will be called\n\nA validator will be called by\n\n-   `./results/\u003cbenchmark_id\u003e[.\u003crepeat\u003e].(single|multi)_core.result.txt` and\n-   `./data/validations/\u003cbenchmark_id\u003e.validate.txt`.\n\nThe first is the output of a benchmark and the second is a reference output.\n\nFor example, after the benchmark `index_exact_search` ran, the validator\n`index_exact_hamming_distance.py` would be called like this:\n\n``` terminal\n./validators/index_exact_hamming_distance.py ./results/index_exact_search.single_core.result.txt ./data/validations/index_exact_search.validate.txt\n```\n\n### Output of a validator\n\nA validator should only give a single number (between 0 and 1) back. If you need\nto give some debug information, output it on stderr.\n\n0 means the result was wrong, 1 means the result was correct.\n\nIn the case of pairwise alignments, either all scores where correct or not, thus\n1 or 0. Since a single wrong result means a defect in the program and therefore\nmakes no sense to say \"15 out of 16 were correct.\".\n\nYou can also give a quality measure. For example in the case of multiple\nalignments, there is no correct and wrong output. But, you can measure the\nquality of such an alignment for example by using\n[BaliBase](http://bioinformatics.oxfordjournals.org/content/15/1/87.abstract).\n\nIf your validator doesn't need a reference output, create an empty reference\noutput. The file is expected to be there, but the content of the file depends on\nthe validator.\n\n### Validator config\n\nAdd the validator to `resources/config/validators.json`. For example adding\n`pairwise_alignment_validator.py` would be:\n\n``` javascript\n{\n    // [...]\n    \"pairwise_alignment_validator.py\": [\n        // [...],\n        // add benchmarks which the validator can handle.\n        \"pairwise_alignment_global_affine_dna\"\n    ]\n}\n```\n\n### Update github wiki\n\nAdditionally, a description/specification of the validator should be added to\nthe github wiki where the page should be the validator id. As an example,\n\u003chttps://github.com/marehr/bench/wiki/pairwise_alignment_validator\u003e.\n\nContributors\n-------\n\nFor questions or comments, feel free to contact:\n\n-   [Chenxu Pan](\n    http://www.mi.fu-berlin.de/en/inf/groups/abi/members/Scientific_Staff/pan.html)\n    \u0026lt;\u003cchenxu.pan[ät]fu-berlin.de\u003e\u0026gt;\n-   [Marcel Ehrhardt](\n    http://www.mi.fu-berlin.de/en/inf/groups/abi/members/Scientific_Staff/ehrhardt.html)\n-   [Svenja Mehringer](\n    http://www.mi.fu-berlin.de/en/inf/groups/abi/members/Scientific_Staff/mehringer.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseqan%2Fbench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fseqan%2Fbench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseqan%2Fbench/lists"}