{"id":21843998,"url":"https://github.com/dkauf42/bayota","last_synced_at":"2026-05-13T07:31:43.108Z","repository":{"id":46051710,"uuid":"180859770","full_name":"dkauf42/bayota","owner":"dkauf42","description":"Framework for identifying cost-effective, optimal strategies for watershed pollutant reduction in the Chesapeake Bay Program's Phase 6 Watershed Model","archived":false,"fork":false,"pushed_at":"2021-11-17T15:54:48.000Z","size":44984,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-26T11:42:04.641Z","etag":null,"topics":["decision-support","docker","optimization-framework","parallel-processing","python","water-quality"],"latest_commit_sha":null,"homepage":"https://doi.org/10.1016/j.envsoft.2021.105141","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dkauf42.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-11T19:03:09.000Z","updated_at":"2022-08-30T14:51:29.000Z","dependencies_parsed_at":"2022-08-30T20:11:42.082Z","dependency_job_id":null,"html_url":"https://github.com/dkauf42/bayota","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkauf42%2Fbayota","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkauf42%2Fbayota/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkauf42%2Fbayota/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkauf42%2Fbayota/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dkauf42","download_url":"https://codeload.github.com/dkauf42/bayota/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244825657,"owners_count":20516592,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["decision-support","docker","optimization-framework","parallel-processing","python","water-quality"],"created_at":"2024-11-27T22:18:05.701Z","updated_at":"2026-05-13T07:31:38.082Z","avatar_url":"https://github.com/dkauf42.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BayOTA\n\nBayOTA (Bay Optimization Tools for Analysis) is designed for use by the partners of the\nChesapeake Bay Program (CBP) as well as the general public as part of the Optimization Tool\nDevelopment Project (EPA-R3-CBP-16-03).\nSpecifically, this extends the functionality of -\nand helps users of - CAST (the Chesapeake Bay Assessement Scenario Tool),\nwhich is the CBP Phase 6 time-averaged watershed model.\n\n\u003cstrong\u003eTable of Contents\u003c/strong\u003e\n\n* [How do I get set up?](#-how-do-i-get-set-up)\n    1. [Ensure the IPOPT solver is installed and in $PATH](#1-ensure-the-ipopt-solver-is-installed-and-in-path)\n    2. [Clone the repository](#2-clone-the-repository)\n    3. [Configure the workspace before installing](#3-configure-the-workspace-before-installing)\n    4. [Install packages](#4-install-packages)\n    5. [Double-check the local paths](#5-double-check-the-local-paths)\n    6. [Test the installation](#6-test-the-installation)\n* [Usage](#-usage)\n    - [Specification Files](#specification-files)\n    - [Batch Sequence](#batch-sequence)\n    - [Running from the command line](#1-running-from-the-command-line)\n    - [Running from the python prompt](#2-running-from-the-python-prompt)\n    - [Running from a jupyter notebook](#3-running-from-a-jupyter-notebook)\n    - [Cleaning up after installation and runs](#-cleaning-up-after-intallation-and-runs)\n    - [Debugging or troubleshooting](#-debugging-or-troubleshooting)\n* [Project structure](#-project-structure)\n    - [Components](#components)\n    - [Directory Tree](#directory-tree)\n* [Updating](#-updating)\n* [Uninstalling](#-uninstalling)\n* [Other Notes](#-other-notes)\n* [Credits](#-credits)\n* [Disclaimer](#-disclaimer)\n* [License](#-license)\n* [Who do I talk to?](#-who-do-i-talk-to)\n\n# ⚙ How do I get set up?\n\n##### 1📉 Ensure the IPOPT solver is installed and in $PATH\n\n-- The Ipopt solver must be installed separately in order to solve Efficiency BMP optimization problems.\n- Option 1: compile/install using instructions found at https://www.coin-or.org/Ipopt/documentation/node14.html\n- Option 2: install using conda: `conda install ipopt`\n\n-- After installation, the Ipopt executable location must be added to the environment $PATH variable\n\n\n##### 2👥 Clone the repository\n\n-- Check out a clone of this repository to a location of your choice, e.g.\n```\ngit clone https://gitlab.com/daka42/bayota.git ~/bayota\n```\n\n-- From the project directory, get the latest version:\n\n```\ncd bayota/\n\ngit pull\n```\n\n-- Most work will be done in the 'develop' branch, so switch to it with git checkout\n\n```\ngit checkout develop\n```\n\n##### 3🏡 Configure the workspace before installing\n\nSet up where you would like to have a bayota 'workspace'.  \\\nSpecify this before installing, by doing one of the following...\n\n- Set the following ENV variables:\n    - `BAYOTA_HOME` (should point to the directory where the bayota repo is located), and \n    - `BAYOTA_WORKSPACE_HOME` (should point to the directory where workspace will be located)\n  \nOR\n- Customize the following values in `bayota_settings/install_config.ini`:\n    - `repo` (should point to the directory where the bayota repo is located)\n    - `local_workspace_stem`\n\n***Note:*** *Important filepaths are set (during install) by the `bayota_settings` package.\\\nThese paths include general output, logging, temporary files, etc., and are defined in the following three config files:*\n\n- `bash_config.con` *specifies the path of the project directory.*\n- `logging_config.yaml` *specifies the format and targets of log messages.*\n- `user_config.ini` *specifies output path stems (for stdout, graphics, and logs)*\n\n*These three config files are required for conducting BayOTA optimization studies, and will be copied into `~/bayota_ws_{version}/config/` during the first install (or first test run).*\n\n***Note:*** *A subset of downloaded CAST source data is included in the 'source_data_subset' folder in this repository. \\\nThe path to these data must be specific in the user_config.ini file, as \\\nfull installation and usage is not possible otherwise.*\n\n*These files will not be programmatically changed by subsequent code executions after being generated.*\\\n*The example/default config files can be found in the `bayota_settings` package.*\n\n\n##### 4💾 Install packages\n\n-- From the project dir (`bayota/`), enter:\n\n```\npip install .\n```\n\n\n##### 5🛣️ Double check the local paths\n\nDuring the first install (or first test run), default configuration files are generated.\\\n-- In `bayota_ws_{version}/config/`, customize values within:\n\n- `user_config.ini` to direct output to the desired directories.\\\n- `bash_config.con` to specify the project home.\n\n\n##### 6✅ Test the installation\n\n-- From the project directory, run the automated test suites:\n\n```\npython setup.py test\n```\n\nOr, if you want to run it from the installed package:\n```\npytest --pyargs bayota\n```\n\nOr, for one package at a time:\n```\npytest --pyargs bayota.castjeeves\npytest --pyargs bayota.bayom_e\n```\n\n***Note:*** *To remove the test files after running the tests, use `python setup.py clean`.*\n\n-- **If the tests pass, you should be good to go!**\n\n# ▶ Usage\n\nOptimization studies can be conducted in BayOTA in multiple ways:\n1) Command-line: batch mode or single run\n2) Python prompt: batch or single run\n3) Jupyter notebook: batch or single run\n\n#### Specification Files\nTypical usage requires setting up model and run configurations in 'specification files'. \nThese are defined (by default) in a workspace subdirectory, and there are 3 types:\n* **batch specs** - These set up one or more studies, by specifying:\n    - geographies\n    - names of model and experiment spec files to use\n    - other options, such as moving/translating solution files after solving\n* **model specs** - These set up the model objectives/constraints\n* **experiment specs** - These allow modification of the model for particular experiments and running through different values of constraints.\n\n#### Batch Sequence\n\n\u003cimg src=\"./.images/code_organization_specification_files.png\" alt=\"batch_sequence\" width=\"295\" height=\"377\"/\u003e\n\n#### 1⌨ Running from the command line\n-- First, change directory to the project root (`cd bayota/`).\n\nTwo methods of running are provided. \\\nThey each provide the ability to run a batch of optimization studies automatically. The options are to use either the \nSlurm job manager or AWS Batch in docker containers.  To execute, one would run either:\n`bin/run_scripts/docker_batch_runner.py`\nor\n`bin/run_scripts/slurm_batch_runner.py`\n\nNote: When either of these are run, they will automatically call the following run_steps during execution:\n1) `bin/run_steps/step1_generatemodel.py`\n2) `bin/run_steps/step2_modifymodel.py`\n3) `bin/run_steps/step3_solveonetrial.py`\n\n###### Batch runs are set up using 'specification files'. These will be in one's workspace.\n\n-- Example command for a batch of studies:\n* `-d` (or `--dryrun`) argument can be included to only print the commands that would be submitted without running them\n* `--no_slurm` argument needed when not using the SLURM job manager\n* `--help` for command syntax\n\n```\n./bin/run_scripts/run_step0_batch_of_studies.py -n calvertMD_cost_and_load_objective_experiments.yaml --dryrun\n```\n\n\n#### 2🐍 Running from the python prompt\n```python\nfrom bayom_e.model_handling.interface import build_model\nfrom bayom_e.solver_handling import solvehandler\n\n# Create a model instance\nmodel_spec_name = 'costmin_total_Npercentreduction'\nmodel, dataplate = build_model(model_spec_name=model_spec_name,geoscale='county',geoentities='Perry, PA',baseloadingfilename='2010NoActionLoads_updated.csv',savedata2file=False,log_level='INFO')\n\n# Set a constraint level\nmodel.percent_reduction_minimum['N'] = 20\n\n# Solve the instance and get results\n\nsolvername = 'ipopt'\ninstance, results, feasible = solvehandler.solve(localsolver=True, solvername=solvername,\n                                                 instance=model)\n```\n\n#### 3📓 Running from a jupyter notebook\nThe approach to use in a notebook is the same as the python prompt.\n\n#### 🧹 Cleaning up after intallation and runs\n\n--- To remove build files created by \"`python setup.py install`\" or to remove temporary files created in the project home during a run:\n\n```\npython setup.py clean\n```\n\n#### 🐛 Debugging or troubleshooting\n\n* Add `--log_level=DEBUG` argument when running from the command line to output the most verbose logging messages.\n\n# 📁 Project Structure\n\n#### Components\n\n\u003cimg src=\"./.images/components_20200325.png\" alt=\"components\" width=\"489\" height=\"311\"/\u003e\n\n#### Directory Tree\n```\nbayota\n│\n├── README.md              \u003c- Top-level README for users/developers of this project.\n├── CHANGELOG.md           \u003c- Documentation of notable changes to this project\n│\n├── bin                    \u003c- scripts for running from the command-line and performing analyses\n│   ├── aws_scripts/\n│   ├── run_scripts/\n│   ├── run_steps/\n│   ├── python_scripts/\n│   └── my_test_scripts/\n│\n├── bayom_e                \u003c- *Python package* to solve optimization problem involving 'Efficiency' Best Management Practices (BMPs) of CAST\n│   ├── config.py\n│   ├── data_handling/\n│   ├── model_handling/\n│   ├── solver_handling/\n│   ├── solution_handling/\n│   └── ...\n|\n├── castjeeves             \u003c- *Python package* to access, query, and parse source data from the Chesapeake Bay Assessement Scenario Tool (CAST)\n│   ├── jeeves.py\n│   ├── sourcehooks/\n│   ├── sqltables/\n│   └── ...\n│\n├── bayota_settings        \u003c- *Python package* that configures directory paths (output, graphics, \u0026 logging). Contains example config files.\n│   ├── base.py\n│   ├── log_setup.py\n│   └── ...\n│\n├── bayota_util            \u003c- *Python package* for utility methods that haven't yet found a home elsewhere\n│   ├── s3_operations.py\n│   ├── spec_and_control_handler.py\n│   ├── str_manip.py\n│   └── ...\n│\n├── Dockerfile\n├── LICENSE\n├── MANIFEST.in\n├── setup.py\n├── VERSION\n```\n\n# 🚮 Updating\n\n--- To use the latest version of this repository:\n- Enter the `bayota/` directory\n- Activate your desired environment\n- Run the commands:\n\n   ```\n   git pull\n   git checkout master\n   pip install . --upgrade\n   ```\n\n# 🚮 Uninstalling\n\n--- To uninstall the python packages from your environment (site-packages):\n\n```\npip uninstall bayota\n```\n\n--- *(less common) To remove development version of package (i.e., remove it from easy-install.pth and delete the .egg-link)*\n\n```\npython setup.py develop --uninstall\n```\n\n--- To remove everything (uninstall the python packages and then delete the source directory):\n\n```\npip uninstall bayota\nrm -r bayota/\n```\n\n\n# 📔 Other Notes\n\nTo use pynumero package from Pyomo:\n- `scipy` is required\n- may need to run `conda install -c conda-forge pynumero_libraries` for ASL library\n\n\n## 💕 Credits\n\nMajor dependencies:\n\n* [Pyomo](https://www.pyomo.org/)\n* [IPOPT solver](https://projects.coin-or.org/Ipopt)\n* [Pandas](https://pandas.pydata.org/)\n* [NumPy](https://www.numpy.org/)\n\nFunding Acknowledgment:\n\n* U.S. EPA cooperative agreement under federal grant EPA-R3-CBP-16-03 - \"Chesapeake Bay Optimization Tool Development\"\n\n## ❗ Disclaimer\n\nThis is a beta version of the Bay Optimization Tools for Analysis (BayOTA), in the process of being tested. It is provided on an “as is” and “as available” basis and is believed to contain defects. A primary purpose of this beta testing release is to solicit feedback on performance and defects. The Chesapeake Bay Program Office (CBPO) does not give any express or implied warranties of any kind, including warranties of suitability or usability of the website, its software, or any of its content, or warranties of fitness for any particular purpose.\nAll users of BayOTA are advised to safeguard important data, to use caution, and not to rely in any way on correct functioning or performance of the beta release and/or accompanying materials. CBPO will not be liable for any loss (including direct, indirect, special, or consequential losses) suffered by any party as a result of the use of or inability to use BayOTA, its software, or its content, even if CBPO has been advised of the possibility of such loss.\nShould you encounter any bugs, glitches, lack of functionality, or other problems on the website, please let us know. We can be reached at \u003chttps://www.chesapeakebay.net/who/group/modeling_team\u003e. Your help in this regard is greatly appreciated!\n\nThe contents of this document do not necessarily reflect the views and policies of the Environmental Protection Agency, nor does the EPA endorse trade names or recommend the use of commercial products mentioned in this document.\n\n## 🎓 License\nThe 3-Clause BSD License\n\n## 📧 Who do I talk to?\n\nThe U.S. EPA Chesapeake Bay Program, Modeling Team (https://www.chesapeakebay.net/who/group/modeling_team)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkauf42%2Fbayota","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdkauf42%2Fbayota","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkauf42%2Fbayota/lists"}