{"id":24884493,"url":"https://github.com/learnlib/ralib-benchmarking","last_synced_at":"2026-01-06T12:50:49.139Z","repository":{"id":218685818,"uuid":"629917005","full_name":"LearnLib/ralib-benchmarking","owner":"LearnLib","description":null,"archived":false,"fork":false,"pushed_at":"2024-02-07T08:05:53.000Z","size":64076,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-02-01T14:34:11.346Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TeX","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LearnLib.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-19T09:34:39.000Z","updated_at":"2024-02-07T08:06:00.000Z","dependencies_parsed_at":"2024-01-23T10:27:33.114Z","dependency_job_id":"841c05e1-e2b9-4815-9a24-c262f585cb0d","html_url":"https://github.com/LearnLib/ralib-benchmarking","commit_stats":null,"previous_names":["learnlib/ralib-benchmarking"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LearnLib%2Fralib-benchmarking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LearnLib%2Fralib-benchmarking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LearnLib%2Fralib-benchmarking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LearnLib%2Fralib-benchmarking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LearnLib","download_url":"https://codeload.github.com/LearnLib/ralib-benchmarking/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245821548,"owners_count":20678046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-01T14:28:09.486Z","updated_at":"2026-01-06T12:50:49.112Z","avatar_url":"https://github.com/LearnLib.png","language":"TeX","readme":"# RaLib Benchmarking Setup\n\nThis project contains benchmark examples and scripts to \nrun RaLib on these benchmarks. The project is organized\nas follows:\n\n1. **benchmark problems** are located in ```benchmarks/```.\n    Problems are collected from papers on RaLib, from the\n    [automata wiki](https://automata.cs.ru.nl), and some\n    are generated or manually created for this project. \n2. **configurations** of RaLib are located in ```configs/```.\n    The different configurations are used in different \n    series of experiments and specify aspects of the \n    learning experiments (how counterexamples are found,\n    how counterexamples are preprocessed, max. runtimes, \n    teachers/theories for types, etc.) \n3. **configurations** are organized in series in ```experiments/```.\n    Series combine benchmark problems with a configuration.\n    The different series in this project originate from \n    multiple papers on RaLib / register automata learning.\n4. **results** from running experiments will be stored in \n    ```results/```. Results include logs and models from \n    individual learning experiments. Some scripts will \n    use the results to compute plots and tables. \n    Templates and generated documents ce be found \n    in ```plots```.\n5. **RaLib** is a git sub module in ```ralib/```.\n\n\n## RaLib Installation\n\nTo install RaLib, you have to initialize the submodule \nand build RaLib, which requires a JDK and Maven to be \ninstalled.\n\n```\ngit submodule init\ngit submodule update\n./build.sh\n```\n\n## Running experiments \n\nGenerally, experiments are organized in series and \nconfigurations do not specify which learner should \nbe used or how many times experiments should be run.\nShell scripts exist for running individual experiments,\nseries, and complete evaluations.\n\n### Experiments for the evaluation of CT-based learners (TACAS'24)\n\nThis section describes how one can run the experiments \nthat are reported in the TACAS 2024 submission.\nConcrete results may vary a little bit for every \nexecution due to randomization but trends should be\nstable.\n\n1. Running experiments from TACAS 2024 submission\n\n    ```\n    ./run-experiments-model-checker-tacas2024.sh\n    ./run-experiments-with-ce-search-tacas2024.sh\n    ```\n\n    results (logs and models) can be found in ```results/```.\n\n\n2. Generating table in TACAS 2024 submission\n\n    this step requires latexmk and a latex distribution\n    to be installed\n\n    ```\n    ./generate-table-tacas2024.sh\n    ```\n\n    The generated pdf is ```plots/tacas2024-table-results.pdf```\n\n    *Note:* The generated pdf has more rows and columns \n    than the table in the paper.\n\n2. Generating plots\n\n    this step requires latexmk and a latex distribution\n    with pgfplots to be installed\n\n    ```\n    ./generate-plots-tacas2024.sh\n    ```\n\n    The plots in Fig. 4. are:\n\n    ```\n    plots/plot-dtls-resets.pdf\n    plots/plot-dtls-resets-noopt.pdf\n    plots/plot-dtls-counterexamples.pdf\n    plots/plot-dtls-wct.pdf\n    ```\n\n    The plots in Fig. 5 are\n\n    ```\n    plots/plot-gen-transitions.pdf\n    plots/plot-gen-registers.pdf\n    plots/plot-gen-registers-noopt.pdf\n    ```\n\n    *Note:* Some additional plots that are generated\n    were not included in the paper.\n\n### Running individual experiments\n\n```\nrun_experiment.sh [-h] -s series -e experiment -i iterations -l learner\n```\n\nLearn model of `experiments/[series]/[experiment].xml` with config\n`experiments/[series]/config` and specified learner.\n\n### Running a series\n\n```\nrun_series.sh [-h] -s series -i iterations -l learner\n```\n\nRun the series of experiments specified in folder\n`experiments/[series]` with specified learner.\n\n### Running the complete evaluation (2015 RaLib Paper)\n\n```\nrun_evaluation.sh\n```\n\nThis script contains explicit calls to `run_series.sh` for\nall experiments that are to be run as part of the evaluation.\n\n\n### Searching in logs\n\n```\nsearch_logs.sh series experiment learner search_term\n```\n\nSearches for `search_term` in the logs from all iterations.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flearnlib%2Fralib-benchmarking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flearnlib%2Fralib-benchmarking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flearnlib%2Fralib-benchmarking/lists"}