Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/jonnor/micropython-npyfile
Numpy .npy file support for MicroPython - read/write/streaming
https://github.com/jonnor/micropython-npyfile
Last synced: 3 months ago
JSON representation
Numpy .npy file support for MicroPython - read/write/streaming
- Host: GitHub
- URL: https://github.com/jonnor/micropython-npyfile
- Owner: jonnor
- License: mit
- Created: 2024-08-11T14:03:18.000Z (5 months ago)
- Default Branch: master
- Last Pushed: 2024-10-02T20:16:57.000Z (3 months ago)
- Last Synced: 2024-10-17T13:17:47.379Z (3 months ago)
- Language: Python
- Size: 192 KB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
- awesome-micropython - micropython-npyfile - Numpy .npy file support for MicroPython, supports read/write/streaming. (Libraries / Mathematics)
README
[](https://github.com/jonnor/micropython-npyfile/actions/workflows/tests.yaml)
# micropython-npyfile
Support for [Numpy files (.npy)](https://numpy.org/doc/stable/reference/generated/numpy.lib.format.html) for [MicroPython](https://micropython.org/).
Simple persistence of multi-dimensional numeric array data, and interoperability with Numpy/CPython et.c.
Was initially written to be used with [emlearn-micropython](https://github.com/emlearn/emlearn-micropython),
a Machine Learning and Digital Signal Processing library for MicroPython.
#### Features
- Reading & writing .npy files with numeric data (see below for Limitations)
- Streaming/chunked reading & writing
- No external dependencies. Uses [array.array](https://docs.micropython.org/en/latest/library/array.html)
- Written in pure Python. Compatible with CPython, CircuitPython, et.c.
## Installing
This package can be installed using [mip](https://docs.micropython.org/en/latest/reference/packages.html#installing-packages-with-mip).
For example:
```bash
mpremote mip install github:jonnor/micropython-npyfile
```
Or just copy the `npyfile.py` file to your MicroPython device.
## Usage
#### Save a file (simple)
```python
import array
import npyfile
shape = (10, 4)
data = array.array('f', (1.0 for _ in range(shape[0]*shape[1])))
npyfile.save('mydata.npy', data, shape)
```
#### Load a file (simple)
```python
import npyfile
shape, data = npyfile.load('mydata.npy')
print(shape)
print(data)
```
#### Streaming read
Streaming/chunked reading can be used to keep memory usage low.
```python
import npyfile
with npyfile.Reader('mydata.npy') as reader:
# Metadata available on the reader object
print(reader.shape, reader.typecode, reader.itemsize)
# NOTE: assumes input is 2d. Pick chunksize in another way if not
chunksize = reader.shape[1]
for chunk in reader.read_data_chunks(chunksize):
print(len(chunk), chunk)
```
More examples:
- Streaming matching data from two files: [two_streams.py](./examples/digits/two_streams.py)
#### Streaming write
Streaming/chunked writing can be used to keep memory usage low.
See implementation of `npyfile.save()`, in [npyfile.py](./npyfile.py)
#### Reading .npz files
.npz files are ZIP archives.
They can be read using an implementation of the zipfile module,
such as [jonnor/micropython-zipfile](https://github.com/jonnor/micropython-zipfile).
Both uncompressed and DEFLATE compressed files are supported.
For example code, see [read_npz.py](./examples/read_npz.py).
#### Saving portable files with numpy
numpy.save defaults to allowing pickle support.
This makes is very easy to accidentially save numpy.array
objects that use pickle for serialization, instead of just numeric data.
Such files will not be loadable by `npyfile` (or most other .npy readers).
To avoid this, when saving numeric arrays, always use:
```python
numpy.save(PATH, DATA, allow_pickle=False)
```
## Limitations
- Only little-endian is supported, not big-endian
- Only C data order is supported, not Fortran
- Strings are not supported
- Complex numbers not supported
- Pickled data is not supported
NOT TESTED on unsupported/malformed/malicious inputs.
## TODO
Contributions welcomed!
TODO:
- Add tests for unsupported/malformed inputs
## Developing
#### Running tests on host
Install the Unix/Window port of MicroPython. Then run:
```
MICROPYPATH=./ micropython tests/test_npyfile.py
```
The tests can also be ran under CPython
```
PYTHONPATH=./ python tests/test_npyfile.py
```
#### Running tests on device
Connect a MicroPython device via USB.
Copy over the data
```
mpremote cp npyfile.py :
mpremote -r cp tests/ :
mpremote run tests/test_npyfile.py
```