{"id":24990253,"url":"https://github.com/arup-group/mc","last_synced_at":"2025-04-12T01:51:59.999Z","repository":{"id":41332295,"uuid":"236709786","full_name":"arup-group/mc","owner":"arup-group","description":"Making MATSim Configuration fun again","archived":false,"fork":false,"pushed_at":"2025-03-25T20:00:45.000Z","size":388,"stargazers_count":8,"open_issues_count":22,"forks_count":1,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-12T01:51:55.588Z","etag":null,"topics":["arup","city-modelling","city-modelling-lab","cml","orchestration","simulation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arup-group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-28T10:31:05.000Z","updated_at":"2025-03-13T21:28:34.000Z","dependencies_parsed_at":"2023-02-17T19:15:37.928Z","dependency_job_id":"a2a5e1ab-92c9-4b0a-9774-3da3ad792498","html_url":"https://github.com/arup-group/mc","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arup-group%2Fmc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arup-group%2Fmc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arup-group%2Fmc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arup-group%2Fmc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arup-group","download_url":"https://codeload.github.com/arup-group/mc/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248505873,"owners_count":21115354,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arup","city-modelling","city-modelling-lab","cml","orchestration","simulation"],"created_at":"2025-02-04T13:35:04.020Z","updated_at":"2025-04-12T01:51:59.971Z","avatar_url":"https://github.com/arup-group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Master of Ceremonies\n\nInterfaces for making MATSim configuration less painful.\n\nModify, generate, diff and debug configs from the command line.\n\n## Contents\n\n* [Installation](#installation)\n* [Usage](#usage)\n* [Command Line](#command-line)\n* [Programming Interface](#programming-interface)\n* [Find](#find)\n* [Validation](#validation)\n* [Updating MC for config changes](#updating-mc-for-config-changes)\n* [Tests](#tests)\n* [Contact](#contacts)\n* [Todo](#todo)\n\n## Installation\n\nMC requires python3.5 or above.\n\n```sh\ngit clone git@github.com:arup-group/mc.git\npip3 install -e .\ncd mc\nvirtualenv -p python3.7 venv\nsource venv/bin/activate\npip3 install -e .\npytest\n```\n\nAlthough we recomment using a virtual environment as shown above (`virtualenv...` \u0026 `source...`), MC requirements are minimal so you may get away with your default environment.\n\n## Usage\n\nMC has a Command Line Interface (CLI), but also a useful pythonic API for bespoke applications.\n\nThe CLI provides a number of features, introduced below. These include some general purpose functionality such as `diff`, as well as some bespoke features required for other tooling, such as `paramreplace`, `step` and `autostep`.\n\nWe are also interested in maintaining a `debug` command to catch common config mistakes. If you have a feature in mind, such as debugging of a common config mistake please add it to the project issues.\n\n## Command Line\n\nThe CLI commands are pretty explorable via help, start with `mc --help`:\n\n```sh\nUsage: mc [OPTIONS] COMMAND [ARGS]...\n\n  Command line interface for MC.\n\nOptions:\n  --help  Show this message and exit.\n\nCommands:\n  autostep      Read config, apply overrides and write out.\n  build         Build a config with defined sub-pops, modes \u0026 activities.\n  convert       Read an existing config and write as xml or json.\n  debug         Debug a config.\n  diff          Simple diff two configs.\n  fill          Read a wildcarded config, apply overrides and write.\n  find          Find and print config components to terminal.\n  gen           Generate a template config: empty|default|test.\n  matchreplace  Read wildcarded config, apply overrides and write.\n  paramreplace  Read config, apply overrides and write out.\n  print         Print a config to terminal.\n  report        Generate a csv report with scoring parameters.\n  step          Read config, apply overrides and write out.\n  summary       Summarise a config.\n```\n\n### Print Example\n\n(from MC root)\n\n```sh\n❯ mc print tests/test_data/test_config.xml\n```\n\n```sh\nmodule {'name': 'global'}\n   param {'name': 'coordinateSystem', 'value': 'EPSG:27700'}\n   param {'name': 'insistingOnDeprecatedConfigVersion', 'value': 'true'}\n   param {'name': 'numberOfThreads', 'value': '32'}\n   param {'name': 'randomSeed', 'value': '4711'}\n module {'name': 'network'}\n   param {'name': 'inputCRS', 'value': 'null'}\n   param {'name': 'inputChangeEventsFile', 'value': 'null'}\n   param {'name': 'inputNetworkFile', 'value': '~/test/network.xml'}\n   param {'name': 'laneDefinitionsFile', 'value': 'null'}\n   param {'name': 'timeVariantNetwork', 'value': 'false'}\n   ...\n```\n\n### Diff Example\n\n(from MC root)\n\n```sh\n❯ mc diff tests/test_data/test_config.xml tests/test_data/test_config_v12.xml\n```\n\n```sh\n- param@plans: insistingOnUsingDeprecatedPersonAttributeFile\n- param@plans: inputPersonAttributesFile\n+/- param@strategy: fractionOfIterationsToDisableInnovation: 0.95 -\u003e 0.9\n```\n\n### Find Example\n\n(from MC root)\n\n```sh\n❯ mc find tests/test_data/test_config.xml coordinateSystem\n```\n\n```sh\nparam {'name': 'coordinateSystem', 'value': 'EPSG:27700'}\n```\n\nYou can read about more complex addressing in [Find](##find).\n\n## Programming Interface\n\n```py\nfrom mc import build\n```\n\nTo facilitate a beautiful future where humans never need edit xml and machines start to explore MATSim simulation config parameter spaces themselves, MC makes a dictionary-like `mc.build.Config` object available. `Config` objects can read and write to MATSim `.xml` config format (and `.json` just in case).\n\n```py\nconfig = build.Config(path='tests/test_data/test_config.xml')\nconfig.write(path='temp.json')\nconfig2 = build.Config(path='temp.json')\nconfig == config2\n```\n\n```py\nTrue\n```\n\n### Quick Reminder\n\nMATSim configs consist of modules, these modules can contain either parameters and/or parameter***sets*** (ie sets of paramaters). Paramater***sets*** can then contain either parameters and/or more parameter***sets***.\n\n### Setting and Getting\n\n`Config` objects consist of nested `Modules`, `ParamSets` and `Params` (just like a MATSim xml formatted config). All of which will behave like a nested set of dicts. Icluding supporting getting and setting methods:\n\n```py\n# get and print module contents:\nconfig['plans'].print()\n```\n\n```py\nmodule {'name': 'plans'}\n    param {'name': 'inputPlansFile', 'value': 'test_inputs/population.xml'}\n    param {'name': 'inputPersonAttributesFile', 'value': 'test_inputs/attributes.xml'}\n    param {'name': 'subpopulationAttributeName', 'value': 'subpopulation'}\n```\n\n```py\n # set and print a single param:\nconfig['plans']['inputPlansFile'] = 'test_inputs/new_population.xml'\nprint(config['plans']['inputPlansFile'])\n```\n```py\ntest_inputs/new_population.xml\n```\n\n### Nested Setting and Getting\n\nNested setting is allowed, for example for an empty `Config` a new module, paramset and param\n can be set together:\n\n```py\nempty_config = build.Config()\nempty_config['global']['coordinateSystem'] = 'EPSG:27700'\n...\nempty_config.print()\n```\n\n```py\nmodule {'name': 'global'}\n    param {'name': 'coordinateSystem', 'value': 'EPSG:27700'}\n```\n\n### Validation of Keys and Values\n\nSetting requires that keys are valid:\n\n```py\nempty_config = build.Config()\nempty_config['NotAModule']['coordinateSystem'] = 'EPSG:27700'\n```\n\n```py\n...\nKeyError: \"key:'NotAModule' not found in modules\"\n```\n\n...and that values are valid:\n\n```py\nempty_config['global']['coordinateSystem'] = {\"crs\": \"27700\"}\n\n```\n\n```py\nINFO creating new empty module: global\nValueError: Please use value of either type ParamSet, Param or str\n```\n\nNested objects can be explicitly accessed via the parent attributes, ie `.modules.values()`, `.parametersets.values()`, `params.values()`. Valid keys (ignoring suffixes) for nestable\nobjects via `.valid_keys`. Note that valid keys and values are hardcoded in `mc.valid`.\n\n### Unique Parameterset Keys\n\nMATSim configurations include parameter***sets*** for which the unique identification (such as the mode or subpopulation) is contained as a parameter. So that we can provide unique keys for the parameter***set***, we therefore suffix parameter***set*** keys as `\u003cparamset_name\u003e:\u003cuid\u003e` where `uid` is the appropriate parameterset\nsubpopulation, mode or activity:\n\n```py\nempty_config.write(path=Path('temp.xml'))\nempty_config['planCalcScore']['scoringParameters:high_income']['modeParams:car']['monetaryDistanceRate'] = '-0.0001'\n```\n\n```py\nINFO creating new empty module: planCalcScore\nINFO creating new empty parameterset: scoringParameters:high_income\nINFO creating new empty parameterset: modeParams:car\n```\n\nIn this case we have created a scoring parameterset for the `high_income` subpopulation and a mode parameterset for `car`.\n\nWhen writing to `.xml` these suffixes are ignored, writing a clean usable config. When reading in\nan existing config, these suffixes are automatically generated.\n\n**WARNING: Not specifying a suffix will generally be assumed as using `\u003cparamset_name\u003e:default`.**\n\n## Find\n\nBoth the CLI and API support searches for config components using an addressing system. The addressing system uses a string format with `/` to denote a nested parameterset or parameter. This allows changes to be easilly passed as commands via common web interfaces for example.\n\n```py\nconfig = build.Config(path='tests/test_data/test_config.xml')\nsearch = config.find(\"plans/inputPlansFile\")\nfor i in search:\n  i.print()\n```\n\n```py\nparam {'name': 'inputPlansFile', 'value': '~/test/population.xml.gz'}\n```\n\nFind is returning a list because it supports partial addresses which result in multiple finds:\n\n```py\nsearch = config.find(\"modeParams:car/monetaryDistanceRate\")\nfor i in search:\n  i.print()\n```\n\n```py\nparam {'name': 'monetaryDistanceRate', 'value': '-0.0001'}  # eg subpopulation A\nparam {'name': 'monetaryDistanceRate', 'value': '-0.0001'}  # eg subpopulation B\n```\n\nAddresses can omit components, for example if we want to look at all `monetaryDistanceRates` for the default subpopulation (ie from `scoringParameters:default`):\n\n```py\nsearch = config.find(\"scoringParameters:default/monetaryDistanceRate\")\n# this is equivalent to \"*/scoringParameters:default/*/monetaryDistanceRate\"\nfor i in search:\n  i.print()\n```\n\n```py\nparam {'name': 'monetaryDistanceRate', 'value': '-0.0'}  # eg walk\nparam {'name': 'monetaryDistanceRate', 'value': '-0.0'}  # eg bike\nparam {'name': 'monetaryDistanceRate', 'value': '-0.001'}  # eg pt\nparam {'name': 'monetaryDistanceRate', 'value': '-0.0001'}  # eg car\n```\n\nOr more simply we can get all `monetaryDistanceRates`:\n\n```py\nsearch = config.find(\"monetaryDistanceRate\")\n# this is equivalent to \"*/*/*/monetaryDistanceRate\"\n# this is equivalnet to \"*/scoringParameters:*/modeParams:*/monetaryDistanceRate\"\nfor i in search:\n  i.print()\n```\n\n```py\nparam {'name': 'monetaryDistanceRate', 'value': '-0.0'}  # eg subpop A walk\nparam {'name': 'monetaryDistanceRate', 'value': '-0.0'}  # eg subpop A bike\nparam {'name': 'monetaryDistanceRate', 'value': '-0.001'}  # subpop A eg pt\nparam {'name': 'monetaryDistanceRate', 'value': '-0.0001'}  # subpop A eg car\nparam {'name': 'monetaryDistanceRate', 'value': '-0.0'}  # eg subpop B walk\nparam {'name': 'monetaryDistanceRate', 'value': '-0.0'}  # eg subpop B bike\nparam {'name': 'monetaryDistanceRate', 'value': '-0.001'}  # eg subpop B pt\nparam {'name': 'monetaryDistanceRate', 'value': '-0.0001'}  # eg subpop B car\n```\n\nIn the examples above, you can see that wildcarding with `*` can be used to return 'all' config elements. The `*` operator tells the find method to search all at a given level. As shown above, it is useful for returning all elements within a parameterset or explicitly describing levels to search.\n\n### Find and Set\n\nNote that the `find` method is returning a reference to the found config objects, which can then be set:\n\n```py\nconfig.find(\"plans/inputPlansFile\")[0] = \"NEW/PATH\"\n```\n\n## Validation\n\nMC has a built-in representation of a valid config structure, specifically the viable names of modules, parametersets and parameters. When reading in an existing config or adding new components, MC will throw validation errors if the valid config structure is not maintained.\n\n```py\nempty_config = build.Config()\nempty_config['NotAModule']['coordinateSystem'] = 'EPSG:27700'\n```\n\n```py\n...\nKeyError: \"key:'NotAModule' not found in modules\"\n```\n\nThis system is useful for preventing typos, but has to be\n[maintained and updated for changes to valid configs](#updating-mc-for-config-changes). The valid mapping is described in the `mc.valid` module.\n\n## Report\nThe CLI supports to generate a CSV report for summarising the scoring parameters for different modes and subpopulations in a tabular format from a given MATSim config.\n```py\n mc report \u003cMATSIM_CONFIF\u003e\u003cOUTPUT_DIR\u003e\n mc report tests/test_data/test_config.xml tests/test_data    \n```\n\n```\nmode:car,bus,train,walk,bike\n\n|        subpopulation         |           default            |           unknown            |\n---------------------------------------------------------------------------------------------\n|             car              |                              |                              |\n---------------------------------------------------------------------------------------------\n|    marginalUtilityOfMoney    |             0.0              |             0.0              |\n---------------------------------------------------------------------------------------------\n|          performing          |             6.0              |             6.0              |\n---------------------------------------------------------------------------------------------\n|     utilityOfLineSwitch      |             -1.0             |             -1.0             |\n---------------------------------------------------------------------------------------------\n|    mode_specific_constant    |             0.0              |             0.0              |\n---------------------------------------------------------------------------------------------\n| marginal_utility_of_distance |             0.0              |             0.0              |\n---------------------------------------------------------------------------------------------\n|marginal_utility_of_traveling |             -6.0             |             -6.0             |\n---------------------------------------------------------------------------------------------\n|    monetary_distance_rate    |             -0.0             |             -0.0             |\n---------------------------------------------------------------------------------------------\n```\n\n\n\n## Updating MC for Config Changes\n\nAn example of how to update the validation mapping can be seen with the addition of a new `hermes` module:\n\n```xml\n  \u003cmodule name=\"hermes\" \u003e\n      \u003cparam name=\"endTime\" value=\"32:00:00\" /\u003e\n      \u003cparam name=\"flowCapacityFactor\" value=\"0.01\" /\u003e\n      \u003cparam name=\"mainMode\" value=\"car\" /\u003e\n      ...\n  \u003c/module\u003e\n```\n\nIn order to make this hermes module available to MC's validation, the following is added to `mc/valid.py`:\n\n```json\n  \"hermes\": {\n      \"params\": {\n            \"mainMode\": \"car\",\n            \"endTime\": \"36:00:00\",\n            \"flowCapacityFactor\": \"0.01\",\n            ...\n      }\n  },\n```\n\n## Tests\n\n```sh\n    python -m pytest -vv tests\n```\n\nTo generate XML \u0026 HTML coverage reports to `reports/coverage`:\n\n```sh\n    ./scripts/code-coverage.sh\n```\n\n## Contact\n\nfred.shone@arup.com\n\n## Further Work\n\n* Add more and better debugging\n* Update default configurations\n* Print a high level summary to terminal (scaling/subpops, modes etc)\n* Add detailed value validation, for example acceptable integer ranges\n* Add further debugging by comparing config to other inputs such as transit schedules\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farup-group%2Fmc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farup-group%2Fmc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farup-group%2Fmc/lists"}