{"id":20852313,"url":"https://github.com/minizinc/mzn-bench","last_synced_at":"2025-07-22T19:04:44.736Z","repository":{"id":40372147,"uuid":"272854584","full_name":"MiniZinc/mzn-bench","owner":"MiniZinc","description":"A framework to performing benchmark testing on MiniZinc models, solvers, and/or the compiler itself.","archived":false,"fork":false,"pushed_at":"2025-05-07T23:33:49.000Z","size":215,"stargazers_count":6,"open_issues_count":2,"forks_count":7,"subscribers_count":4,"default_branch":"develop","last_synced_at":"2025-07-01T16:10:58.868Z","etag":null,"topics":["benchmark","cluster","minizinc","minizinc-benchmarks","minizinc-python","slurm"],"latest_commit_sha":null,"homepage":"https://www.minizinc.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MiniZinc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-06-17T01:58:42.000Z","updated_at":"2025-05-07T23:33:53.000Z","dependencies_parsed_at":"2023-01-21T06:02:05.426Z","dependency_job_id":"f56abd74-6d38-45ce-a8d4-57dfbda34e74","html_url":"https://github.com/MiniZinc/mzn-bench","commit_stats":{"total_commits":70,"total_committers":3,"mean_commits":"23.333333333333332","dds":"0.12857142857142856","last_synced_commit":"d3da5efd9fb2996be8ba1cd7113488faf2532a88"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/MiniZinc/mzn-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MiniZinc%2Fmzn-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MiniZinc%2Fmzn-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MiniZinc%2Fmzn-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MiniZinc%2Fmzn-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MiniZinc","download_url":"https://codeload.github.com/MiniZinc/mzn-bench/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MiniZinc%2Fmzn-bench/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266554213,"owners_count":23947288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","cluster","minizinc","minizinc-benchmarks","minizinc-python","slurm"],"created_at":"2024-11-18T03:17:11.420Z","updated_at":"2025-07-22T19:04:44.717Z","avatar_url":"https://github.com/MiniZinc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MiniZinc Bench\n\nThis is a small collection of scripts that allow you to run benchmarks on a set\nof MiniZinc instance using MiniZinc Python. The process is split into several\nsteps to be easily customisable to different kinds of possible benchmarks.\n\n\u003e Currently, the only supported way of running benchmarks is through\n\u003e [SLURM](https://slurm.schedmd.com/). Other methods may become available in the\n\u003e future.\n\n## Preparation\n\n1. Create a CSV file for the MiniZinc instances containing _problem_, _model_,\n   _data_file_. If you store the instances in the MiniZinc benchmarks\n   repository structure, then you can use the `mzn-bench collect-instances`\n   command:\n   ```bash\n   mzn-bench collect-instances \u003cdirectory\u003e \u003e instances.csv\n   ```\n2. Instantiate a benchmarking environment. This environment should at least\n   contain a Python virtual environment with _mzn-bench_ and your benchmarking\n   scripts, but you can also set up environmental variables, like `PATH`, and\n   load cluster modules. The following script, `bench_env.sh`,\n   provides an example environment that can be loaded using `source bench_env.sh`:\n\n   ```bash\n   if [[ \"${BASH_SOURCE[0]}\" = \"${0}\" ]]; then\n       \u003e\u00262 echo \"Remember: you need to run me as 'source ${0}', not execute it!\"\n       exit\n   fi\n\n   # Create or activate Python virtual environment\n   if [ -d venv ]; then\n       source venv/bin/activate\n   else\n      python3 -m venv venv\n       source venv/bin/activate\n       python3 -m pip install mzn-bench\n   fi\n\n   # Set other environment variables and load cluster modules\n   module load MiniZinc/2.4.3\n   ```\n\n3. Create a benchmarking script. This script will contain the configuration of\n   where the instance file is located, what MiniZinc/Solver configurations to\n   run for every instance, and how the benchmark runner itself should be\n   configured. The script mainly consists of a call to `schedule` in\n   `mzn_bench`. For example a benchmarking script that runs Gecode and\n   Chuffed for 20 minutes might look like this:\n\n   ```python\n   from datetime import timedelta\n   from pathlib import Path\n\n   import minizinc\n\n   from mzn_bench import Configuration, schedule\n\n   schedule(\n       instances=Path(\"./instances.csv\"),\n       timeout=timedelta(minutes=20),\n       configurations=[\n           Configuration(name=\"Gecode\", solver=minizinc.Solver.lookup(\"gecode\")),\n           Configuration(name=\"Chuffed\", solver=minizinc.Solver.lookup(\"chuffed\")),\n       ],\n       nodelist=[\"critical001\"],\n   )\n   ```\n\nThese are all the possible arguments to `schedule`:\n\n- `instances: Path` - The path to the instances file.\n- `timeout: timedelta` - The timeout set for the MiniZinc process.\n- `configurations: Iterable[Configuration]` - MiniZinc solving configurations\n  (see below for details).\n- `nodelist: Optional[Iterable[str]]` - A list of nodes on which SLURM is allowed to\n  schedule the tasks. If `None`, `mzn-bench` will sequentially solve the instances locally.\n- `output_dir: Path = Path.cwd() / \"results\"` - The directory in which the raw\n  results will be placed. This directory will be created if it does not yet\n  exist.\n- `job_name: str = \"MiniZinc Benchmark\"` - The SLURM job name.\n- `cpus_per_task: int = 1` - The number of CPU cores required for each task.\n- `memory: int = 4096` - The maximum memory used for each task.\n- `debug: bool = False` - Directly capture the output of individual jobs\n  and store them in a `./logs/` directory.\n- `wait: bool = False` - The scheduling process will wait for all jobs to\n  finish.\n\nA `Configuration` object has the following attributes:\n\n- `name: str` - Configuration name used in the output.\n- `solver: minizinc.Solver` - MiniZinc Python solver configuration.\n- `minizinc: Optional[Path] = None` - Path to a specific MiniZinc executable.\n  If `None` is provided, then the first `minizinc` executable on the `PATH`\n  will be used.\n- `processes: Optional[int] = None` - Number of processes to be used by the\n  solver.\n- `random_seed: Optional[int] = None` - Random seed to be used by the solver.\n- `free_search: bool = False` - Solver can determine its own search heuristic.\n- `optimisation_level: Optional[int] = None` - MiniZinc compilation\n  optimisation level, e.g., `-O3`.\n- `other_flags: Dict[str, Any] = field(default_factory=dict)` - A mapping of\n  flag name to value of other flags to be provided to the compiler/solver\n- `extra_data: Dict[str, Any] = field(default_factory=dict)` - Extra data to be\n  added when using a specific Configuration. Internally this will be used by\n  MiniZinc Python's `__setitem__` method on the generated instances. If data\n  needs the value of an identifier internal to MiniZinc, then please use an\n  `DZNExpression` object (e.g., `{\"preferred_encoding\": DZNExpression(\"UNARY\")}`).\n\n## Schedule SLURM jobs\n\nThe job now has to be started on the cluster with the right number of tasks\n(one for every instance/solver combination). Luckily, the benchmarking script\nthat you've created in the previous step should take care of all of this.\nSo once we ensure that our environment is ready for ou benchmark, we can\nexecute our script and our job will be scheduled.\n\nFor example, if we had created a script `bench_env.sh` with our benchmarking\nenvironment and a script `start_bench.py` with our `schedule` call, then the\nfollowing code should schedule our job:\n\n```bash\nsource bench_env.sh\npython start_bench.py\n```\n\nYou can keep track of the status of your job using the `squeue` command.\n\n**WARNING:** Once the job has started the CSV file containing the instances and\nthe instance files themselves should not be changed or moved until the full\nbenchmark is finished. This could causes error or, even worse, inconsistent\nresults.\n\n**Note:** If you find a mistake after you have scheduled your job, then you\nshould cancel the job as soon as possible. This can be done by using the\n`scancel` command. This command will take the `job_id`, shown when your job is\nscheduled, as an argument.\n\n## Collect information\n\nOnce the job is finished, it is time to get your data wrangling pants on! This\nrepository contains some scripts that might be helpful in locating and\nformatting the information that you need. Some scripts might be used directly\nwhile other might need some customising to fit your purpose. Note that these\nscripts might require some extra dependencies. For this reason, these scripts\nare not expected to work unless this package is installed as\n`pip install mzn-bench[scripts]`.\nThis allows us to install a minimal version on the running cluster and this\nmore complete version locally while processing the data.\n\n### General aggregation\n\nThe following scripts can help gather the raw `*_stats.yml`/`*_sol.yml` files\nand combine them for further use:\n\n- `mzn-bench collect-objectives \u003cresult_dir\u003e \u003cobjectives.csv\u003e` -\n  This script gathers all objective value information given by MiniZinc and the\n  used solvers and combines it into a single CSV file.\n- `mzn-bench collect-statistics \u003cresult_dir\u003e \u003cstatistics.csv\u003e` -\n  This script gathers all statistical information given by MiniZinc and the used\n  solvers and combines it into a single CSV file.\n\n### Tabulation\n\nThe following scripts filter and tabulate specific statistics.\n\n- `mzn-bench report-status \u003cstatistics.csv\u003e` - This command will report the\n  number of occurrences of the various solving status of your MiniZinc tasks.\n  Note that the number of satisfied instances is reported as `A + B`, where `A`\n  is the number of optimisation instances that reach a solution not proven\n  optimal and `B` is the number of satisfaction instance finding a solution.\n  Please consult the `-h` flag to display all options.\n- `mzn-bench report-mzn-scores \u003cstatistics.csv\u003e` - This command will report the\n  MiniZinc scores computed based on the performance of solvers across instances. \n  These scores are calculated using a weighted formula that considers runtime, \n  solution quality, and solver status. Use the `-h` flag to explore additional \n  options and configurations.\n- `mzn-bench compare-configurations \u003cstatistics.csv\u003e \u003cbefore_conf\u003e \u003cafter_conf\u003e` - This command reports on the differences of the achieved\n  results between two configurations (differences in status, runtime, and\n  objective). You can adjust the changes deemed significant with the\n  `--time-delta` and `--objective-delta` flag. You can use the `--output-mode json` option to ensure the output can be easily parsed by other programs.\n\n### Solution checking\n\nThe `mzn-bench check-solutions` command takes the solutions output during run\nand feeds them back into the model to check that the result is satisfiable.\nIt also stores the objective and satisfiability information to be used when\nchecking statuses. The `-c` option can be used to set how many solutions\nto check (zero to check all solutions).\n\n```bash\n# Check three solutions from each instance\nmzn-bench check-solutions -c 3 ./results\n```\n\nThis requires the problem `.mzn` and `.dzn` files from the benchmark run to be\navailable in order to run the checker. The `--base-dir \u003cDIR\u003e` option can be used\nto specify a root directory relative to which the file names in the\n`*_sol.yml` files are resolved.\n\n### Status checking\n\nThe `mzn-bench check-statuses` command takes the results from `check-solutions`\ncommand above (which must be run first) and then checks for any solvers which\nhave either\n\n- Falsely claimed optimality - where optimality was found by a solver, but a\n  better objective was found elsewhere and verified to be correct.\n- Falsely claimed unsatisfiability - where unsatisfiability was found by a\n  solver, but another solver has given a correct solution for the instance.\n\n### Graph generation\n\nThere are a number of plotting helper functions available in\n`mzn_bench.analysis.plot`. In order to use these, you must enable the\nplotting features with `pip install mzn-bench[plotting]`. These use the\n[Bokeh](https://bokeh.org/) visualisation library to provide interactive plots.\n\nThe `read_csv` function returns a tuple of [pandas](https://pandas.pydata.org/)\ndata frames containing objective and statistics data for plotting or further\ndata analysis.\n\n```py\nfrom mzn_bench.analysis.collect import read_csv\nfrom mzn_bench.analysis.plot import plot_all_instances\nfrom bokeh.plotting import show\n\n# Read CSVs generated by mzn-bench collect-result as pandas dataframes\nobjs, stats = read_csv(\"objectives.csv\", \"statistics.csv\")\n\n# Grid plot giving objective values over time, or time to solve\n# (depending on instance type)\nshow(plot_all_instances(objs, stats))\n```\n\n\n### Testing\nCurrently, this library is tested using a single end-to-end test which runs most of the pipeline locally (without SLURM).\n\n```\npytest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminizinc%2Fmzn-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fminizinc%2Fmzn-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminizinc%2Fmzn-bench/lists"}