https://github.com/mousebrains/q2netcdf
Convert Rockland Q-files into NetCDF
https://github.com/mousebrains/q2netcdf
Last synced: 3 months ago
JSON representation
Convert Rockland Q-files into NetCDF
- Host: GitHub
- URL: https://github.com/mousebrains/q2netcdf
- Owner: mousebrains
- License: gpl-3.0
- Created: 2024-11-05T05:01:49.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2026-03-22T02:06:37.000Z (3 months ago)
- Last Synced: 2026-03-22T11:06:57.389Z (3 months ago)
- Language: Python
- Size: 459 KB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: docs/CONTRIBUTING.md
- License: LICENSE
- Security: docs/SECURITY.md
Awesome Lists containing this project
README
# q2netcdf
[](https://www.python.org/downloads/)
[](https://github.com/mousebrains/q2netcdf/actions/workflows/ci.yml)
[](https://codecov.io/gh/mousebrains/q2netcdf)
[](https://pypi.org/project/q2netcdf/)
> **Note**: The package requires Python 3.10+, but the standalone `mergeqfiles` script is compatible with Python 3.7+ for deployment on legacy systems (e.g., MicroRider instruments).
Python tools for manipulating [Rockland Scientific's](https://rocklandscientific.com) Q-files as generated by the [ISDP data logger](https://rocklandscientific.com/news/rockland-data-logger/).
## Features
- **Q-file to NetCDF Conversion**: Convert binary Q-files to CF-1.8 compliant NetCDF format
- **Header Parsing**: Extract metadata, sensor configurations, and timing information
- **Data Extraction**: Read scalar channels and frequency spectra from binary records
- **File Manipulation**: Merge, reduce, and inspect Q-files
- **Configuration Generation**: Create syntactically correct `isdp.cfg` files
- **Format Support**: Handles Q-file versions 1.2 and 1.3
- **Sensor Mapping**: Built-in support for 200+ sensor identifier codes
## Command-Line Tools
- **`q2netcdf`** - Translate Q-files into CF-1.8 compliant NetCDF files
- **`QFile`** - Dump header and data records from Q-files
- **`QHeader`** - Display header record information
- **`QReduce`** - Reduce Q-file size by removing records
- **`mergeqfiles`** - Merge multiple Q-files (designed for MicroRider/Slocum Glider integration)
- **`mkISDPcfg`** - Generate syntactically correct `isdp.cfg` configuration files
## Quick Start
```bash
# Display Q-file header information
QHeader data.q
# Convert Q-file to NetCDF
q2netcdf data.q --nc output.nc
# Dump all records from Q-file
QFile data.q --logLevel INFO
```
## Installation
### From PyPI
```bash
pip install q2netcdf
```
Or using [pipx](https://pipx.pypa.io/stable/installation/) for the command-line tools:
```bash
pipx install q2netcdf
```
### From Source
```bash
git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e .
```
### For Development
```bash
git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e ".[dev]"
pytest # Run tests
```
## Installation on the MicroRider
For deployment on Rockland Scientific MicroRider instruments:
1. Transfer [src/q2netcdf/mergeqfiles.py](src/q2netcdf/mergeqfiles.py) and [src/q2netcdf/mergeqfiles.cfg](src/q2netcdf/mergeqfiles.cfg) to the MicroRider's data directory.
2. Dismount the data directory on your host computer.
3. Connect to the MicroRider via a serial connection using your favorite terminal emulator at 115,200 baud.
4. `ls data` to verify the files are present. This will cause the MicroRider to unmount data from being read-only to the MicroRider and then remount it as read-write. This will typically cause the serial connection to drop then be reestablished. You might have to restart your terminal emulator's connection.
5. `mv data/mergeqfiles.py /usr/local/bin/mergeqfiles` to install the script as mergeqfiles.
6. Test with `mergeqfiles --help` to make sure the installation was successful.
7. You can disconnect the MicroRider's serial connection.
8. You should now run a simulation on the Slocum glider to verify Q-files are being uploaded to the science computer's storage as .mri files.
9. Diagnostic information is written into the data/mergeqfiles.log file on the MicroRider.
10. You can modify data/mergeqfiles.cfg to adjust the information included in the .mri file sent to the glider.
This is designed for integration with [TWR's Slocum Glider uRider proglet](https://www.teledynemarine.com/brands/webb-research/slocum-glider).
## Python API
```python
from q2netcdf import QFile
# Stream records from a Q-file
with QFile("data.q") as qf:
header = qf.header()
print(header.channels(), header.spectra())
for record in qf.data():
print(record.stime, record.channels)
# Convert a Q-file to an xarray Dataset
from q2netcdf.q2netcdf import loadQfile
ds = loadQfile("data.q")
print(ds)
ds.to_netcdf("output.nc")
```
## Usage Examples
### Converting Q-files to NetCDF
```bash
q2netcdf data.q --nc output.nc
```
### Inspecting Q-file Headers
```bash
# Show full header with channel and spectra names
QHeader data.q
# Show condensed header
QHeader data.q --nothing
# Show only configuration
QHeader data.q --channels --spectra --frequencies
```
### Examining Q-file Data Records
```bash
# Dump all records
QFile data.q
# Limit number of records displayed
QFile data.q --n 10
```
### Merging Q-files
```bash
mergeqfiles -o merged.q file1.q file2.q file3.q
```
### Generating ISDP Configuration Files
```bash
mkISDPcfg --help
```
## File Format
Q-files are binary files containing oceanographic data from Rockland Scientific instruments. The format is documented in Rockland Technical Note TN-054.
### Q-file Structure
- **Header Record** (0x1729): File version, timestamp, channel/spectra identifiers, frequencies, configuration
- **Data Records** (0x1657): Timestamped measurements with scalar channels and frequency spectra
### Supported Versions
- **v1.2**: Original format with full metadata in each data record
- **v1.3**: Optimized format with reduced redundancy
## Contributing
Contributions are welcome! Please see [CONTRIBUTING.md](docs/CONTRIBUTING.md) for guidelines.
### Development Setup
```bash
git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e ".[dev]"
pytest # Run tests
ruff check src/ tests/ # Lint
mypy src/ # Type check
```
To run a module directly: `python3 -m q2netcdf.QHeader --help`
## Additional Documentation
- [CHANGELOG.md](docs/CHANGELOG.md) - Version history and changes
- [CONTRIBUTING.md](docs/CONTRIBUTING.md) - Contributor guidelines and development setup
- [SECURITY.md](docs/SECURITY.md) - Security policy and vulnerability reporting
- [QFILE_FORMAT.md](docs/QFILE_FORMAT.md) - Q-file binary format specification
## License
This project is licensed under the [GNU General Public License v3.0 or later](LICENSE).
## Author
**Pat Welch**
Email: pat@mousebrains.com
## Acknowledgments
Developed for use with [Rockland Scientific](https://rocklandscientific.com) oceanographic instruments and the ISDP data logger.