https://github.com/datalab-org/datalab-app-plugin-insitu
datalab block plugin for in situ measurements (NMR, UV-Vis, XRD)
https://github.com/datalab-org/datalab-app-plugin-insitu
chemistry data-management datalab-plugin experimental-data materials-science
Last synced: 3 months ago
JSON representation
datalab block plugin for in situ measurements (NMR, UV-Vis, XRD)
- Host: GitHub
- URL: https://github.com/datalab-org/datalab-app-plugin-insitu
- Owner: datalab-org
- License: mit
- Created: 2024-11-01T15:11:24.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2026-02-26T22:49:48.000Z (4 months ago)
- Last Synced: 2026-02-28T18:58:50.262Z (4 months ago)
- Topics: chemistry, data-management, datalab-plugin, experimental-data, materials-science
- Language: Python
- Homepage: https://datalab-app-plugin-insitu.readthedocs.io
- Size: 145 MB
- Stars: 4
- Watchers: 1
- Forks: 1
- Open Issues: 22
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
#
datalab *in situ* plugin
A Python plugin for processing and visualizing *in situ* (NMR, UV-Vis, XRD) data within [_datalab_](https://github.com/datalab-org/datalab) instances.
The project was originally developed in the [Grey Group](https://www.ch.cam.ac.uk/group/grey/) in the Department of Chemistry at the University of Cambridge, with support and ongoing maintenance from [*datalab industries ltd.*](https://datalab.industries).
## Features
- Support for NMR, XRD and UV-Vis data, recorded alongside electrochemical experiments, temperature ramps or other time series data (support varies across data types)
- Integration with electrochemical data formats for combined analysis using [navani](https://github.com/be-smith/navani)
- Interactive visualization using Bokeh, with heatmaps and spectra selection
- Support for both local files, as well as running as a plugin directly on a *datalab* instance
## Roadmap
There are still several planned features and improvements for this plugin,
including:
- Supporting all formats for electrochemistry, NMR, XRD and UV-Vis that are supported by the corresponding non-*in situ* blocks available in *datalab*.
- More flexible specification of other time series data across all data types, e.g., temperature or pressure ramps.
- Improved performance and caching of data, especially for large datasets.
# Installation
The `datalab-app-plugin-insitu` package is currently a "core plugin" of *datalab* (since *datalab* v0.5.3), i.e., it will be installed on *datalab* instances by default.
## Development installation
We recommend you use [`uv`](https://astral.sh/uv) for managing virtual environments and Python versions.
Once you have `uv` installed, you can clone this repository and install the package in a fresh virtual environment with:
```shell
git clone git@github.com:datalab-org/datalab-app-plugin-insitu
cd datalab-app-plugin-insitu
uv sync --all-extras --dev
```
You can activate `pre-commit` in your cloned repository with `uv run pre-commit install`.
You can run the tests using `pytest`, optionaly with the marker `show_plots` to display the plots generated by the tests:
```shell
uv run pytest -m show_plots
```
## Usage
Full reference API documentation can be found on [ReadTheDocs](https://datalab-app-plugin-insitu.readthedocs.io/).
### Data Structure Requirements
#### NMR
Your *in situ* NMR data should be organized as follows:
```
data_folder.zip/
├── /
│ ├── 1/
│ │ ├── acqus
│ │ └── pdata/
│ │ └── 1/
│ │ └── ascii-spec.txt
│ ├── 2/
│ │ └── ...
│ └── ...
└── /
└── *.MPR
```
with the `` and `` names specified at runtime (or via the *datalab* UI).
#### UV-Vis
```
data_folder.zip/
├── /
│ ├── scan_0000.txt
│ ├── scan_0001.txt
│ ├── scan_0002.txt
│ ├── scan_0003.txt
│ ├── scan_0004.txt
│ ├── scan_0005.txt
│ ├── ...
│ └── scan_.txt
├── /
│ └── reference.txt
└── /
├──
├──
└── ...
```
where the `` contains the UV-Vis scans (named as `*_.txt`, which will be used for sorting), the `` contains the reference spectrum, and the `` contains one or more electrochemical data files loadable by [navani](https://github.com/be-smith/navani). Note if multiple files are present then navani will attempt to stitch them together, so do not upload duplicates (e.g processed and unprocessed versions of the same file).
#### XRD
There are two XRD plotting modes, against Temperature and against Electrochemistry, both pathways require a folder containing the xrd data and the log data as shown below, with the electrochemistry pathway requiring a third directory for the electrochemistry data.
```
data_folder.zip/
├── /
│ ├── 0000-scan.dat
│ ├── 0001-scan.dat
│ ├── 0002-scan.dat
│ ├── 0003-scan.dat
│ ├── 0004-scan.dat
│ ├── 0005-scan.dat
│ ├── ...
│ └── -scan.dat
└── /
└── time_series_log.csv
└── / (optional)
├──
├──
└── ...
```
where the `` contains the XRD scans (named as `0000-scan.dat`, `0001-scan.dat`, etc.), and the `` contains a time series log file in CSV format, mapping scan number to the time series data, e.g. for the Temperature mode:
```csv
scan_number,Temp
0000, 25.0
0001, 30.0
0002, 35.0
0003, 40.0
```
And for the electrochemistry mode:
```csv
scan_number,start_time,end_time
0000, 2025-07-02 19:05:59.614000, 2025-07-02 19:06:56.167000
0001, 2025-07-02 19:15:03.582000, 2025-07-02 19:15:59.280000
0002, 2025-07-02 19:24:07.586000, 2025-07-02 19:25:04.074000
0003, 2025-07-02 19:33:12.198000, 2025-07-02 19:34:08.917000
```
The `` contains one or more electrochemistry files readable by [navani](https://github.com/be-smith/navani), Note if multiple files are present then navani will attempt to stitch them together, so do not upload duplicates (e.g processed and unprocessed versions of the same file). For the XRD usecase it's currently expected that the echem file has a `Timestamp` column containing date-time information for matching to the log file (e.g Neware files).
### Front-end Usage
The following configuration parameters are available in the *datalab* UI:
#### Required Parameters
- **`scan_time`** (UV-Vis only): The time in seconds between each scan, including any rest time. This is used to calculate the time of each scan when no log file is available.
#### Optional Parameters
##### Performance Optimization
- **`data_granularity`**: Controls sampling frequency along the x-axis (wavelength/2θ) of the heatmap to reduce rendering time. Default: automatically calculated to target ~1000 data points.
- XRD: Uses max-pooling interpolation
- UV-Vis: Uses linear interpolation
- **`sample_granularity`**: Controls sampling frequency along the y-axis (time/experiment number) of the heatmap. Uses linear interpolation. Default: automatically calculated to target ~1000 samples.
##### File Selection
- **`glob_str`** (XRD only): A pattern to match specific files in the XRD data folder. Examples:
- `"summed"` → matches files containing "summed" (automatically converted to `*summed*`)
- `"*summed*"` → matches files containing "summed" (explicit wildcards)
- `"*.xy"` → matches all `.xy` files
- Leave empty to process all files in the folder (default behavior)
## License
This project is released under the conditions of the MIT license. Please see [LICENSE](https://github.com/datalab-org/datalab-app-plugin-insitu/blob/main/LICENSE) for the full text of the license.
## Contact
For questions and support, please [open an issue](https://github.com/datalab-org/datalab-app-plugin-insitu/issues) on the GitHub repository or join the [public datalab Slack workspace](https://join.slack.com/t/datalab-world/shared_invite/zt-2h58ev3pc-VV496~5je~QoT2TgFIwn4g).