https://github.com/prakaa/mms-monthly-cli
Source code and CLI tool to query and download data from the Australian Energy Market Operator's Monthly Data Archive
https://github.com/prakaa/mms-monthly-cli
aemo australia data energy national-electricity-market nem nemweb python
Last synced: about 1 month ago
JSON representation
Source code and CLI tool to query and download data from the Australian Energy Market Operator's Monthly Data Archive
- Host: GitHub
- URL: https://github.com/prakaa/mms-monthly-cli
- Owner: prakaa
- License: gpl-3.0
- Created: 2023-07-11T10:20:04.000Z (almost 2 years ago)
- Default Branch: master
- Last Pushed: 2025-03-31T17:47:16.000Z (about 2 months ago)
- Last Synced: 2025-04-13T03:13:27.951Z (about 1 month ago)
- Topics: aemo, australia, data, energy, national-electricity-market, nem, nemweb, python
- Language: Python
- Homepage:
- Size: 500 KB
- Stars: 6
- Watchers: 1
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# AEMO Monthly Data Archive Tool
[](https://badge.fury.io/py/mms-monthly-cli)
[](https://github.com/prakaa/mms-monthly-cli/actions/workflows/cicd.yml)
[](https://codecov.io/gh/prakaa/mms-monthly-cli)
[](https://results.pre-commit.ci/latest/github/prakaa/mms-monthly-cli/master)
[](https://github.com/psf/black)
A CLI utility to find and obtain data made available through AEMO's [MMS Monthly Data Archive](https://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/MMSDM/).
> **Note**
>
> This package and its CLI have some overlap with [NEMOSIS](https://github.com/UNSW-CEEM/NEMOSIS)
> and [NEMSEER](https://github.com/UNSW-CEEM/NEMSEER).
> However:
> - NEMOSIS does not provide access to the monthly data archive, and does not allow a user to download certain
> tables such as bid data tables (`BIDPEROFFER`).
> - NEMSEER only handles PASA and pre-dispatch tables available in the monthly data archive, though it
> provides a lot of useful utilities for handling these tables
## Installation
Recommended that you use [`pipx`](https://github.com/pypa/pipx) to install the CLI as it [prevents dependency conflicts](https://github.com/pypa/pipx#overview-what-is-pipx).
```bash
pipx install mms-monthly-cli
```
However, you can still install via `pip` if you are only using the Python package:
```bash
pip install mms-monthly-cli
```
## Usage
You can use this package as a module (source code functions within [`mms_monthly.py`](./mms_monthly_cli/mms_monthly.py)) or as a CLI tool to:
1. Inspect what data is available
2. Download and unzip a table CSV for a particular period (i.e. a month within a year)
### Python module
Simply import `mms_monthly_cli` as follows:
```python
import mms_monthly_cli.mms_monthly as mms_monthly
```
This will expose the functions listed below (accessed using `mms_monthly.`).
> [!NOTE]
> The `data_dir` argument requires you to specify which folder within `MMSDM_YYYY_MM` you are interested in.
> Most users will be interested in tables within `DATA` (i.e. `data_dir="DATA"`).
>
> You can also access `PREDISP_ALL_DATA` (pre-dispatch data)
> and `P5MIN_ALL_DATA` (5-minute pre-dispatch data). Data tables in the `PREDISP_ALL_DATA` folders
> contain *all* forecasts for an interval of interest, whereas `PREDISPATCH` data tables in the `DATA` folder only contain the latest interval
> (refer to [this part of the NEMSEER docs for more detail](https://nemseer.readthedocs.io/en/latest/quick_start.html#predispatch-tables)).
> It is unclear if the same is true for `P5MIN_ALL_DATA`; `P5MIN` data tables in `DATA` appear have the same data with some minor differences
> (e.g. one column name is different in the constraint files).
>
> If you are accessing pre-dispatch data, consider using [NEMSEER](https://github.com/UNSW-CEEM/NEMSEER).
---
```python
get_years_and_months() -> Dict[int, List[int]]
```
```md
Years and months with data on NEMWeb MMSDM Historical Data Archive
Returns:
Months mapped to each year. Data is available for each of these months.
```
---
```python
get_available_tables(year: int, month: int, data_dir: str) -> List[str]
```
```md
Tables that can be requested from MMSDM Historical Data Archive for a
particular month and year.
Args:
year: Year
month: Month
data_dir: Directory within monthly archives
Returns:
List of tables associated with that forecast type for that period
```
---
```python
get_table_names_and_sizes(year: int, month: int, data_dir: str) -> Dict
```
```md
Returns table names and sizes from MMSDM Historical Data Archive page
For a year and month in the MMSDM Historical Data Archive, returns a list of
tuples each consisting of:
- A table name (obtained via captured regex group)
- The size of the associated zip file
Args:
year: Year
month: Month
data_dir: Directory within monthly archives
Returns:
Tuple of table names and file sizes
```
---
```python
get_and_unzip_table_csv(year: int, month: int, data_dir: str, table: str, cache: pathlib.Path) -> None
```
```md
Unzipped (single) csv file downloaded from `url` to `cache`
This function:
1. Downloads zip file in chunks to limit memory use and enable progress bar
2. Validates that the zip contains a single file that has the same name as the zip
Args:
year: Year
month: Month
data_dir: Directory within monthly archives
table: Table name
cache: Path to save zip.
Returns:
None. Extracts csv to `cache`
```
---
### CLI tool
The CLI tool uses [Typer](https://typer.tiangolo.com/).
GIF produced using [asciinema](https://github.com/asciinema/asciinema) and [agg](https://github.com/asciinema/agg)
Usage is documented within the tool:
```bash
mms-monthly-cli --help
```
...which returns:
```bash
Usage: mms-monthly-cli [OPTIONS] COMMAND [ARGS]...
A CLI utility to find and obtain data made available through AEMO's MMS Monthly Data
Archive: http://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/MMSDM/
╭─ Options ───────────────────────────────────────────────────────────────────────────────╮
│ --install-completion Install completion for the current shell. │
│ --show-completion Show completion for the current shell, to copy it or │
│ customize the installation. │
│ --help Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────────────────────────────────────────────────────────────────────╮
│ available-periods Displays years and the months within them for which data is │
│ available │
│ available-tables Displays available tables for a period (i.e. supplied month and │
│ year) │
│ get-table Download and unzip monthly data zip file to get data table CSV in │
│ cache. To see available periods, use the `available_periods` command │
│ To see available tables for a given period, use the │
│ `available_tables` command │
╰─────────────────────────────────────────────────────────────────────────────────────────╯
```
## License
This tool and associated source code (reused from [`nemseer`](https://github.com/UNSW-CEEM/NEMSEER), which is licensed under GNU GPL-3.0-or-later) was created by Abhijith Prakash with contributions from Matthew Davis.
This tool and its source code is licensed under the terms of [GNU GPL-3.0-or-later licences](./LICENSE).