{"id":18088455,"url":"https://github.com/prakaa/mms-monthly-cli","last_synced_at":"2025-10-30T00:15:19.997Z","repository":{"id":180382987,"uuid":"665049646","full_name":"prakaa/mms-monthly-cli","owner":"prakaa","description":"Source code and CLI tool to query and download data from the Australian Energy Market Operator's Monthly Data Archive","archived":false,"fork":false,"pushed_at":"2025-03-31T17:47:16.000Z","size":512,"stargazers_count":6,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-13T03:13:27.951Z","etag":null,"topics":["aemo","australia","data","energy","national-electricity-market","nem","nemweb","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prakaa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-11T10:20:04.000Z","updated_at":"2025-01-04T01:53:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"8954a453-27d8-451e-a366-222162a6f9f7","html_url":"https://github.com/prakaa/mms-monthly-cli","commit_stats":{"total_commits":49,"total_committers":2,"mean_commits":24.5,"dds":"0.16326530612244894","last_synced_commit":"abc1313f8675a752975a3ec07b92e42454ec91a1"},"previous_names":["prakaa/mms-monthly-cli"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakaa%2Fmms-monthly-cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakaa%2Fmms-monthly-cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakaa%2Fmms-monthly-cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakaa%2Fmms-monthly-cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prakaa","download_url":"https://codeload.github.com/prakaa/mms-monthly-cli/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248657923,"owners_count":21140846,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aemo","australia","data","energy","national-electricity-market","nem","nemweb","python"],"created_at":"2024-10-31T17:13:06.607Z","updated_at":"2025-10-30T00:15:19.917Z","avatar_url":"https://github.com/prakaa.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AEMO Monthly Data Archive Tool\n[![PyPI version](https://badge.fury.io/py/mms-monthly-cli.svg)](https://badge.fury.io/py/mms-monthly-cli)\n[![Continuous Integration and Deployment](https://github.com/prakaa/mms-monthly-cli/actions/workflows/cicd.yml/badge.svg)](https://github.com/prakaa/mms-monthly-cli/actions/workflows/cicd.yml)\n[![codecov](https://codecov.io/gh/prakaa/mms-monthly-cli/branch/master/graph/badge.svg?token=WL7DH013Q7)](https://codecov.io/gh/prakaa/mms-monthly-cli)\n[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/prakaa/mms-monthly-cli/master.svg)](https://results.pre-commit.ci/latest/github/prakaa/mms-monthly-cli/master)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\nA CLI utility to find and obtain data made available through AEMO's [MMS Monthly Data Archive](https://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/MMSDM/).\n\n\u003e **Note**\n\u003e\n\u003e This package and its CLI have some overlap with [NEMOSIS](https://github.com/UNSW-CEEM/NEMOSIS)\n\u003e and [NEMSEER](https://github.com/UNSW-CEEM/NEMSEER).\n\u003e However:\n\u003e - NEMOSIS does not provide access to the monthly data archive, and does not allow a user to download certain\n \u003e   tables such as bid data tables (`BIDPEROFFER`).\n\u003e - NEMSEER only handles PASA and pre-dispatch tables available in the monthly data archive, though it\n\u003e   provides a lot of useful utilities for handling these tables\n\n## Installation\n\nRecommended that you use [`pipx`](https://github.com/pypa/pipx) to install the CLI as it [prevents dependency conflicts](https://github.com/pypa/pipx#overview-what-is-pipx).\n\n```bash\npipx install mms-monthly-cli\n```\n\nHowever, you can still install via `pip` if you are only using the Python package:\n```bash\npip install mms-monthly-cli\n```\n\n## Usage\n\nYou can use this package as a module (source code functions within [`mms_monthly.py`](./mms_monthly_cli/mms_monthly.py)) or as a CLI tool to:\n\n1. Inspect what data is available\n2. Download and unzip a table CSV for a particular period (i.e. a month within a year)\n\n### Python module\n\nSimply import `mms_monthly_cli` as follows:\n\n```python\nimport mms_monthly_cli.mms_monthly as mms_monthly\n```\n\nThis will expose the functions listed below (accessed using `mms_monthly.\u003cfunc_name\u003e`).\n\n\u003e [!NOTE]\n\u003e The `data_dir` argument requires you to specify which folder within `MMSDM_YYYY_MM` you are interested in.\n\u003e Most users will be interested in tables within `DATA` (i.e. `data_dir=\"DATA\"`).\n\u003e\n\u003e You can also access `PREDISP_ALL_DATA` (pre-dispatch data)\n\u003e and `P5MIN_ALL_DATA` (5-minute pre-dispatch data). Data tables in the `PREDISP_ALL_DATA` folders\n\u003e contain *all* forecasts for an interval of interest, whereas `PREDISPATCH` data tables in the `DATA` folder only contain the latest interval\n\u003e (refer to [this part of the NEMSEER docs for more detail](https://nemseer.readthedocs.io/en/latest/quick_start.html#predispatch-tables)).\n\u003e It is unclear if the same is true for `P5MIN_ALL_DATA`; `P5MIN` data tables in `DATA` appear have the same data with some minor differences\n\u003e (e.g. one column name is different in the constraint files).\n\u003e\n\u003e If you are accessing pre-dispatch data, consider using [NEMSEER](https://github.com/UNSW-CEEM/NEMSEER).\n\n---\n```python\nget_years_and_months() -\u003e Dict[int, List[int]]\n```\n```md\nYears and months with data on NEMWeb MMSDM Historical Data Archive\nReturns:\n    Months mapped to each year. Data is available for each of these months.\n```\n---\n```python\nget_available_tables(year: int, month: int, data_dir: str) -\u003e List[str]\n```\n```md\nTables that can be requested from MMSDM Historical Data Archive for a\n   particular month and year.\nArgs:\n    year: Year\n    month: Month\n    data_dir: Directory within monthly archives\n\nReturns:\n    List of tables associated with that forecast type for that period\n```\n---\n```python\nget_table_names_and_sizes(year: int, month: int, data_dir: str) -\u003e Dict\n```\n```md\nReturns table names and sizes from MMSDM Historical Data Archive page\n\nFor a year and month in the MMSDM Historical Data Archive, returns a list of\ntuples each consisting of:\n- A table name (obtained via captured regex group)\n- The size of the associated zip file\n\nArgs:\n    year: Year\n    month: Month\n    data_dir: Directory within monthly archives\nReturns:\n    Tuple of table names and file sizes\n```\n---\n```python\nget_and_unzip_table_csv(year: int, month: int, data_dir: str, table: str, cache: pathlib.Path) -\u003e None\n```\n```md\nUnzipped (single) csv file downloaded from `url` to `cache`\n\nThis function:\n\n1. Downloads zip file in chunks to limit memory use and enable progress bar\n2. Validates that the zip contains a single file that has the same name as the zip\n\nArgs:\n    year: Year\n    month: Month\n    data_dir: Directory within monthly archives\n    table: Table name\n    cache: Path to save zip.\nReturns:\n    None. Extracts csv to `cache`\n```\n---\n### CLI tool\n\nThe CLI tool uses [Typer](https://typer.tiangolo.com/).\n\n![CLI usage](./mms-monthly-cli.gif)\n\u003csub\u003e\u003csup\u003eGIF produced using [asciinema](https://github.com/asciinema/asciinema) and [agg](https://github.com/asciinema/agg)\u003c/sup\u003e\u003c/sub\u003e\n\nUsage is documented within the tool:\n```bash\nmms-monthly-cli --help\n```\n...which returns:\n```bash\nUsage: mms-monthly-cli [OPTIONS] COMMAND [ARGS]...\n\n A CLI utility to find and obtain data made available through AEMO's MMS Monthly Data\n Archive: http://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/MMSDM/\n\n╭─ Options ───────────────────────────────────────────────────────────────────────────────╮\n│ --install-completion          Install completion for the current shell.                 │\n│ --show-completion             Show completion for the current shell, to copy it or      │\n│                               customize the installation.                               │\n│ --help                        Show this message and exit.                               │\n╰─────────────────────────────────────────────────────────────────────────────────────────╯\n╭─ Commands ──────────────────────────────────────────────────────────────────────────────╮\n│ available-periods  Displays years and the months within them for which data is          │\n│                    available                                                            │\n│ available-tables   Displays available tables for a period (i.e. supplied month and      │\n│                    year)                                                                │\n│ get-table          Download and unzip monthly data zip file to get data table CSV in    │\n│                    cache. To see available periods, use the `available_periods` command │\n│                    To see available tables for a given period, use the                  │\n│                    `available_tables` command                                           │\n╰─────────────────────────────────────────────────────────────────────────────────────────╯\n```\n\n## License\n\nThis tool and associated source code (reused from [`nemseer`](https://github.com/UNSW-CEEM/NEMSEER), which is licensed under GNU GPL-3.0-or-later) was created by Abhijith Prakash with contributions from Matthew Davis.\n\nThis tool and its source code is licensed under the terms of [GNU GPL-3.0-or-later licences](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprakaa%2Fmms-monthly-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprakaa%2Fmms-monthly-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprakaa%2Fmms-monthly-cli/lists"}