{"id":20441573,"url":"https://github.com/testingautomated-usi/maxitwo","last_synced_at":"2025-08-19T15:09:45.316Z","repository":{"id":102432573,"uuid":"591580047","full_name":"testingautomated-usi/maxitwo","owner":"testingautomated-usi","description":"Replication package of the paper \"Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing\"","archived":false,"fork":false,"pushed_at":"2024-11-17T08:14:45.000Z","size":5531,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-06-20T17:51:12.755Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/testingautomated-usi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-01-21T06:43:25.000Z","updated_at":"2024-11-17T08:14:48.000Z","dependencies_parsed_at":"2024-03-26T11:28:28.761Z","dependency_job_id":"410d5a66-4498-450f-9e75-c1208011519e","html_url":"https://github.com/testingautomated-usi/maxitwo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/testingautomated-usi/maxitwo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testingautomated-usi%2Fmaxitwo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testingautomated-usi%2Fmaxitwo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testingautomated-usi%2Fmaxitwo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testingautomated-usi%2Fmaxitwo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/testingautomated-usi","download_url":"https://codeload.github.com/testingautomated-usi/maxitwo/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testingautomated-usi%2Fmaxitwo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271173376,"owners_count":24711667,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-19T02:00:09.176Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T09:33:43.212Z","updated_at":"2025-08-19T15:09:45.291Z","avatar_url":"https://github.com/testingautomated-usi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Replication Package for the paper: \"Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing\"\n\nThe repository contains the source code to run the experiments and analyze the results.\nThe README only provides instruction on how to replicate the experiments by using pretrained models.\nThe source code of the simulators (i.e., **DonkeyCar** and **Udacity**) as well as the datasets to train the models \n(both cyclegan and dave2) will be provided on request. \n\nAs for the **BeamNG** simulator we used the version _0.23.5.1.1288_. \nA free version of the BeamNG simulator for research purposes can be obtained by registering at [beamng.tech](https://register.beamng.tech) and following the instructions provided by BeamNG.\nIf the above version is not available you can have a look at the code from the [SBST CPS Tool competition](https://github.com/sbft-cps-tool-competition/cps-tool-competition)\nwhich uses a recent version of BeamNG. The code in the package [code_pipeline](https://github.com/testingautomated-usi/maxitwo/tree/master/code_pipeline) and [envs/beamng](https://github.com/testingautomated-usi/maxitwo/tree/master/envs/beamng) have to be updated \nto reflect the changes of the **beamngpy** library that communicates with the simulator. Such library has to be updated in the [requirements.txt](https://github.com/testingautomated-usi/maxitwo/tree/master/requirements.txt) file.\n\nThe link to download the pretrained models as well as the results of the experiment can be downloaded [here](https://drive.switch.ch/index.php/s/fMkAVQSCO5plOBZ).\nThis needs to be done to execute the commands after step 0.\n\n## Step 0. Configure the environment:\n\n1) Install [anaconda](https://docs.anaconda.com/anaconda/install/) for your system;\n\n2) Create the environment: `conda create -n myenv python=3.8`\n\n3) Activate the environment: `conda activate myenv`\n\n4) Install the requirements: `pip install -r requirements.txt`\n\n## Analyzing the results\n\nCopy the `logs` folder in the archive downloaded before and place it at the root of the folder containing this repository.\n\n### 1. Failure probabilities\n\nType:\n\n```commandline\n./compute_correlations_probabilities.sh mapelites\n```\n\nto compute the correlations using the model trained with simulation data and type:\n\n```commandline\n./compute_correlations_probabilities.sh cyclegan\n```\n\nto compute the correlations using the model trained using the cyclegan translation.\n\n### 2. Quality metrics\n\nType:\n\n```commandline\n./compute_correlations_quality_metrics.sh mapelites\n```\n\nto compute the correlations using the model trained with simulation data and type:\n\n```commandline\n./compute_correlations_quality_metrics.sh cyclegan\n```\n\nto compute the correlations using the model trained using the cyclegan translation.\n\n## Replicating the experiments\n\n### 1. Run the MapElites algorithm\n\n#### 1.1. Using the dave2 model trained with simulated images\n\nThe command to type for `Udacity` is:\n\n```commandline\npython run_mapelites.py --env-name udacity \\\n    --udacity-exe-path \u003cpath/to/udacity-simulator/executable\u003e \\\n    --agent-type supervised \\\n    --model-path \u003cpath/to/mixed-dave2.h5\u003e \\\n    --min-angle 100 \\\n    --max-angle 300 \\\n    --population-size 20 \\\n    --iteration-runtime 150 \\\n    --mutation-extent 6 \\\n    --feature-combination turns_count-curvature \\\n    --num-runs 5\n```\n\nThe simulator executable as well as the dave2 model (i.e., `mixed-dave2.h5`) \nare provided in the previously downloaded archive (respectively under `simulators` and `models`).\n\nThe command above runs the MapElites algorithm in the `Udacity` simulator using the\ndave2 model trained with simulated data for 5 runs, each of 150 iterations. \nThe command creates a folder for each run in the `logs` folder named `mapelites_udacity_\u003cdate_str\u003e_i` where `i` is the index of the run. \nMoreover it creates a folder with all the individuals executed in each run, namely `mapelites_udacity_\u003cdate_str\u003e_all`.\n\nThe MapElites algorithm needs to be executed for the same number of runs and iterations using the `BeamNG` simulator. The command is the following:\n\n```commandline\npython run_mapelites.py --env-name beamng \\\n    --beamng-home \u003cpath/to/beamng-simulator/BeamNG.drive-0.23.5.1.12888\u003e \\\n    --beamng-user \u003cpath/to/beamng-simulator\u003e \\\n    --agent-type supervised \\\n    --model-path \u003cpath/to/mixed-dave2.h5\u003e \\\n    --min-angle 100 \\\n    --max-angle 300 \\\n    --population-size 20 \\\n    --iteration-runtime 150 \\\n    --mutation-extent 6 \\\n    --feature-combination turns_count-curvature \\\n    --num-runs 5\n```\n\n#### 1.2. Using the dave2 model trained with pseudo-real images\n\nIn this case we need to use the pretrained cyclegan model to translate the \nimages coming from the simulator to pseudo-real images. \nThe command is similar to the one above except fo the cyclegan related parameters:\n\n```commandline\npython run_mapelites.py --env-name udacity \\\n    --udacity-exe-path \u003cpath/to/udacity-simulator/executable\u003e \\\n    --agent-type supervised \\\n    --model-path \u003cpath/to/mixed-fake-dave2.h5\u003e \\\n    --min-angle 100 \\\n    --max-angle 300 \\\n    --population-size 20 \\\n    --iteration-runtime 150 \\\n    --mutation-extent 6 \\\n    --feature-combination turns_count-curvature \\\n    --num-runs 5 \\\n    --cyclegan-experiment-name udacity \\\n    --gpu-ids 0 \\\n    --cyclegan-checkpoints-dir cyclegan/checkpoints \\\n    --cyclegan-epoch 35\n```\n\nThis is the command for `Udacity`. Similarly for `BeamNG` the \n`--cyclegan-experiment-name` parameter is `beamng` and the \n`--cyclegan-epoch` parameter is 35. For `Donkey` the \n`--cyclegan-experiment-name` parameter is `donkey` and the \n`--cyclegan-epoch` parameter is 40. In the commands below we only provide examples for the model trained using simulated images; \ncyclegan related parameters need to be provided when the model under test is `mixed-fake-dave2.h5` as shown above.\n\nThe command above needs to be executed on a machine with GPU as the inference time needs to match the \nframe rate of the simulator. The `checkpoints` folder of the cyclegan should have already been downloaded; \nthe archive contains a folder named `cyclegan` with a sub-folder named `checkpoints`. Such folder needs to be placed\nin the `cyclegan` folder at the root of the folder containing this repository.\n\n### 2. Migration\n\n#### 2.1. Migrating individuals found by running MapElites on BeamNG on Udacity (Simulated images)\n\n```commandline\npython run_individual_migration.py --env-name udacity \\\n    --udacity-exe-path \u003cpath/to/udacity-simulator/executable\u003e \\\n    --agent-type supervised \\\n    --model-path \u003cpath/to/mixed-dave2.h5\u003e \\\n    --feature-combination turns_count-curvature \\\n    --filepath mapelites_beamng_\u003cdate_str\u003e_all\n```\n\nThis command will create the folder `mapelites_migration_udacity_\u003cdate_str\u003e`.\nIt is better to rename such folder in `mapelites_migration_udacity_beamng_search`. \nIt should be read in this way: migration on `Udacity` of individuals obtained by running the search (i.e., MapElites) on `BeamNG`. The command is for simulated images; for pseudo-real images the three options showed in Section 1.2. should be added.\n\n#### 2.2. Migrating individuals found by running MapElites on Udacity on BeamNG (Simulated images)\n\n```commandline\npython run_individual_migration.py --env-name beamng \\\n    --beamng-home \u003cpath/to/beamng-simulator/BeamNG.drive-0.23.5.1.12888\u003e \\\n    --beamng-user \u003cpath/to/beamng-simulator\u003e \\\n    --agent-type supervised \\\n    --model-path \u003cpath/to/mixed-dave2.h5\u003e \\\n    --feature-combination turns_count-curvature \\\n    --filepath mapelites_udacity_\u003cdate_str\u003e_all\n```\n\nThis command will create the folder `mapelites_migration_beamng_\u003cdate_str\u003e`.\nIt is better to rename such folder in `mapelites_migration_beamng_udacity_search`. \nIt should be read in this way: migration on `BeamNG` of individuals obtained by running the search (i.e., MapElites) on `Udacity`. The command is for simulated images; for pseudo-real images the three options showed in Section 1.2. should be added.\n\n#### 2.3. Migrating individuals found by running MapElites on BeamNG on Donkey (Simulated images)\n\n```commandline\npython run_individual_migration.py --env-name donkey \\\n    --donkey-exe-path \u003cpath/to/donkey-simulator/executable\u003e \\\n    --agent-type supervised \\\n    --model-path \u003cpath/to/mixed-dave2.h5\u003e \\\n    --feature-combination turns_count-curvature \\\n    --filepath mapelites_beamng_\u003cdate_str\u003e_all\n```\n\nThis command will create the folder `mapelites_migration_donkey_\u003cdate_str\u003e`.\nIt is better to rename such folder in `mapelites_migration_donkey_beamng_search`. \nIt should be read in this way: migration on `Donkey` of individuals obtained by running the search (i.e., MapElites) on `BeamNG`. The command is for simulated images; for pseudo-real images the three options showed in Section 1.2. should be added.\n\n#### 2.4. Migrating individuals found by running MapElites on Udacity on Donkey (Simulated images)\n\n```commandline\npython run_individual_migration.py --env-name donkey \\\n    --donkey-exe-path \u003cpath/to/donkey-simulator/executable\u003e \\\n    --agent-type supervised \\\n    --model-path \u003cpath/to/mixed-dave2.h5\u003e \\\n    --feature-combination turns_count-curvature \\\n    --filepath mapelites_udacity_\u003cdate_str\u003e_all\n```\n\nThis command will create the folder `mapelites_migration_donkey_\u003cdate_str\u003e`.\nIt is better to rename such folder in `mapelites_migration_donkey_udacity_search`. \nIt should be read in this way: migration on `Donkey` of individuals obtained by running the search (i.e., MapElites) on `Udacity`. The command is for simulated images; for pseudo-real images the three options showed in Section 1.2. should be added.\n\n### 3. Union\n\nIn this step we merge the maps coming from the same simulators. This step is called Union in the paper.\n\nFirst, copy the `mapelites_run/mapelites_beamng_\u003cdate_str\u003e_all` folder and rename it as `mapelites_beamng_search`;\nin the same way copy the `mapelites_run/mapelites_udacity_\u003cdate_str\u003e_all` folder and rename it as `mapelites_udacity_search`.\n\nThen, create a folder under `logs` grouping the `_all` folders of the MapElites search and migrations on `Udacity`, `BeamNG` and `Donkey`.\nFor the sake of the example let us call the folder `mapelites_run`.\n\nAt this point the `mapelites_run` folder under `logs` should contain the following folders:\n\n```commandline\nmapelites_beamng_\u003cdate_str\u003e_0\nmapelites_beamng_\u003cdate_str\u003e_1\nmapelites_beamng_\u003cdate_str\u003e_2\nmapelites_beamng_\u003cdate_str\u003e_3\nmapelites_beamng_\u003cdate_str\u003e_4\nmapelites_beamng_\u003cdate_str\u003e_all\nmapelites_beamng_search\nmapelites_migration_beamng_udacity_search\nmapelites_migration_donkey_beamng_search\nmapelites_migration_donkey_udacity_search\nmapelites_migration_udacity_beamng_search\nmapelites_udacity_\u003cdate_str\u003e_0\nmapelites_udacity_\u003cdate_str\u003e_1\nmapelites_udacity_\u003cdate_str\u003e_2\nmapelites_udacity_\u003cdate_str\u003e_3\nmapelites_udacity_\u003cdate_str\u003e_4\nmapelites_udacity_\u003cdate_str\u003e_all\nmapelites_udacity_search\n```\n\nwhile the `logs` folder contains:\n\n```commandline\nmapelites_run\n```\n\n### 3.1. Convert Maps\n\nRun the following command:\n\n```commandline\n./convert_probability_maps.sh mapelites_run\n```\n\nto convert the maps from success probability (generated by the search/migration) to failure probability required by the following analyses. \n\n### 3.2. Probability Maps\n\nRun the following command:\n\n```commandline\n./merge_individual_maps_simulators_probabilities.sh mapelites_run\n```\n\nThe command creates the following folders in `mapelites_run`:\n\n```commandline\nmerged_mapelites_beamng_search_mapelites_migration_beamng_udacity_search\nmerged_mapelites_migration_donkey_udacity_search_mapelites_migration_donkey_beamng_search\nmerged_mapelites_udacity_search_mapelites_migration_udacity_beamng_search\n```\n\nAt this point all maps contain the same individuals but executed on different simulators, i.e., \nrespectively `BeamNG`, `Donkey` and `Udacity`.\n\n### 3.3. Quality Metrics Maps\n\nRun the following command:\n\n```commandline\npython plot_quality_metrics_map.py --folder logs \\\n    --filepath mapelites_run \\\n    --datetime-str-beamng \u003cdate_str_beamng\u003e \\\n    --datetime-str-udacity \u003cdate_str_udacity\u003e \n```\n\nwhere `\u003cdate_str_beamng\u003e` and `\u003cdate_str_udacity\u003e` are the datetime strings \ncorresponding to the folders created by the MapElites search respectively for `BeamNG` and `Udacity`.\n\nMove the quality metrics maps from `mapelites_udacity_\u003cdate_str\u003e_all` to `mapelites_run/mapelites_udacity_search` as well as\nmove the quality metrics maps from `mapelites_beamng_\u003cdate_str\u003e_all` to `mapelites_run/mapelites_beamng_search`.\n\nThe maps are the following:\n\n```commandline\nraw_heatmap_max_lateral_positions_turns_count_curvature_iterations_0.json\nraw_heatmap_std_lateral_positions_turns_count_curvature_iterations_0.json\nraw_heatmap_std_speeds_turns_count_curvature_iterations_0.json\nraw_heatmap_std_steering_angles_turns_count_curvature_iterations_0.json\n```\n\nThe `.png` files can be copied as well but it is not mandatory.\n\nThe previous command also prints on the console the bounds for each quality metric. For instance:\n\n```commandline\nINFO:plot_quality_metrics_map:Bounds for the quality metrics: \n{\n    'std_steering_angles': (0.060142596293090676, 0.5921246225056953), \n    'std_lateral_positions': (0.05692786404029756, 0.8862904614683409), \n    'std_speeds': (4.6374119298227985, 12.133422056281063), \n    'max_lateral_positions': (0.20460000000000012, 2.1)\n}\n```\n\nThe first number in each tuple is the lower bound for the respective metric and the second is the upper bound.\n\nThen run the following command:\n\n```commandline\n./merge_individual_maps_simulators_quality_metrics.sh mapelites_run \\\n    0.060142596293090676 0.5921246225056953 \\\n    0.05692786404029756 0.8862904614683409 \\\n    4.6374119298227985 12.133422056281063 \\\n    0.20460000000000012 2.1\n```\n\nThe bounds come from the previous step and need to be provided in the order specified in the previous step.\n\n### 4. Merge\n\nIn this step we merge the maps coming from different simulators.\n\n### 4.1. Probability Maps\n\nRun the following command:\n\n```commandline\n./merge_maps_simulators_probabilities.sh mapelites_run\n```\n\nThe command merges the failure probability maps of `BeamNG` and `Udacity` using the product operator.\nThe command should produce the folder `merged_merged_beamng_udacity` in `mapelites_run`.\n\n### 4.2. Quality Metrics Maps\n\nRun the following command:\n\n```commandline\n./merge_maps_simulators_quality_metrics.sh mapelites_run min \\\n    0.060142596293090676 0.5921246225056953 \\\n    0.05692786404029756 0.8862904614683409 \\\n    4.6374119298227985 12.133422056281063 \\\n    0.20460000000000012 2.1\n```\n\nThe command merges the quality metrics maps of `BeamNG` and `Udacity` using the minimum operator.\nThe command should produce the quality metrics maps inside `mapelites_run/merged_merged_beamng_udacity`.\n\nAt the end of such step the content of the folder `mapelites_run/merged_merged_beamng_udacity` should be the following:\n\n```\nheatmap_failure_probability_multiply_turns_count_curvature_iterations_0.png\nheatmap_max_lateral_positions_min_turns_count_curvature_iterations_0.png\nheatmap_std_lateral_positions_min_turns_count_curvature_iterations_0.png\nheatmap_std_speeds_min_turns_count_curvature_iterations_0.png\nheatmap_std_steering_angles_min_turns_count_curvature_iterations_0.png\nraw_heatmap_failure_probability_multiply_turns_count_curvature_iterations_0.json\nraw_heatmap_max_lateral_positions_min_turns_count_curvature_iterations_0.json\nraw_heatmap_std_lateral_positions_min_turns_count_curvature_iterations_0.json\nraw_heatmap_std_speeds_min_turns_count_curvature_iterations_0.json\nraw_heatmap_std_steering_angles_min_turns_count_curvature_iterations_0.json\n```\n\n### 5. Analysis\n\nIn this step we compute the correlations between the digital siblings (i.e., `Udacity` and `BeamNG`) and the digital twin (i.e., `Donkey`).\n\n### 5.1. Probabilities\n\nRun the following command:\n\n```commandline\n./compute_correlations_probabilities.sh mapelites_run\n```\n\nThe correlations and all the other metrics are printed on console.\n\n### 5.2. Quality Metrics\n\n```commandline\n./compute_correlations_quality_metrics.sh mapelites_run\n```\n\nThe correlations and all the other metrics are printed on console. \nThe command only computes the correlations for the `max_lateral_position` quality metric.\n\n### 6. Offline Evaluation\n\nDownload the offline evaluation datasets from [here](https://drive.switch.ch/index.php/s/4GyponKp6hx1Rs6) and copy them within the `logs` folder of this repository. Let us carry out the offline evaluation for the `dave2` model.\n\n### 5.1. Simulated images\n\nFrom the root of the project type:\n\n```commandline\npython compute_offline_metrics.py --archive-path logs \\\n\t--archive-names offline-evaluation-beamng-dave2 offline-evaluation-donkey-dave2.npz \\\n\t--show-plot\n```\n\nto compute the correlation and the distance between the prediction error distributions of `BeamNG` and `Donkey`. Save the plot where desired.\n\nType:\n\n```commandline\npython compute_offline_metrics.py --archive-path logs \\\n\t--archive-names offline-evaluation-udacity-dave2 offline-evaluation-donkey-dave2.npz \\\n\t--show-plot\n```\n\nto compute the correlation and the distance between the prediction error distributions of `Udacity` and `Donkey`. Save the plot where desired.\n\nType:\n\n```commandline\npython compute_offline_metrics.py --archive-path logs \\\n\t--archive-names offline-evaluation-beamng-dave2 offline-evaluation-udacity-dave2 offline-evaluation-donkey-dave2.npz \\\n\t--show-plot\n```\n\nto merge the prediction error distributions of `BeamNG` and `Udacity` and compute its correlation and distance w.r.t. the prediction error distribution of `Donkey`. Save the plot where desired.\n\n### 5.2. Pseudo-real images\n\nFrom the root of the project type:\n\n```commandline\npython compute_offline_metrics.py --archive-path logs \\\n\t--archive-names offline-evaluation-fake-beamng-dave2 offline-evaluation-fake-donkey-dave2.npz \\\n\t--show-plot\n```\n\nto compute the correlation and the distance between the prediction error distributions of `BeamNG` and `Donkey`. Save the plot where desired.\n\nType:\n\n```commandline\npython compute_offline_metrics.py --archive-path logs \\\n\t--archive-names offline-evaluation-fake-udacity-dave2 offline-evaluation-fake-donkey-dave2.npz \\\n\t--show-plot\n```\n\nto compute the correlation and the distance between the prediction error distributions of `Udacity` and `Donkey`. Save the plot where desired.\n\nType:\n\n```commandline\npython compute_offline_metrics.py --archive-path logs \\\n\t--archive-names offline-evaluation-fake-beamng-dave2 offline-evaluation-fake-udacity-dave2 offline-evaluation-fake-donkey-dave2.npz \\\n\t--show-plot\n```\n\nto merge the prediction error distributions of `BeamNG` and `Udacity` and compute its correlation and distance w.r.t. the prediction error distribution of `Donkey`. Save the plot where desired.\n\n## 6. Citing the Project\n\nTo cite this repository in publications:\n\n```bibtex\n@article{DBLP:journals/ese/BiagiolaSRT24,\n  author       = {Matteo Biagiola and\n                  Andrea Stocco and\n                  Vincenzo Riccio and\n                  Paolo Tonella},\n  title        = {Two is better than one: digital siblings to improve autonomous driving\n                  testing},\n  journal      = {Empir. Softw. Eng.},\n  volume       = {29},\n  number       = {4},\n  pages        = {72},\n  year         = {2024},\n  url          = {https://doi.org/10.1007/s10664-024-10458-4},\n  doi          = {10.1007/S10664-024-10458-4},\n  timestamp    = {Sun, 04 Aug 2024 19:51:03 +0200},\n  biburl       = {https://dblp.org/rec/journals/ese/BiagiolaSRT24.bib},\n  bibsource    = {dblp computer science bibliography, https://dblp.org}\n}\n```\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftestingautomated-usi%2Fmaxitwo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftestingautomated-usi%2Fmaxitwo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftestingautomated-usi%2Fmaxitwo/lists"}