https://github.com/BAMresearch/openbis-upload-helper
Upload Helper Tool and Parsers for OpenBIS in the BAM Data Store
https://github.com/BAMresearch/openbis-upload-helper
bam-data-store metadata-extraction openbis parser uploader
Last synced: 23 days ago
JSON representation
Upload Helper Tool and Parsers for OpenBIS in the BAM Data Store
- Host: GitHub
- URL: https://github.com/BAMresearch/openbis-upload-helper
- Owner: BAMresearch
- License: mit
- Created: 2025-04-11T16:30:40.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2026-01-12T13:58:08.000Z (27 days ago)
- Last Synced: 2026-01-12T18:56:12.576Z (27 days ago)
- Topics: bam-data-store, metadata-extraction, openbis, parser, uploader
- Language: Python
- Homepage:
- Size: 561 KB
- Stars: 3
- Watchers: 3
- Forks: 2
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-bamresearch - openbis-upload-helper
README
# OpenBIS Upload Helper
The openBIS Upload Helper is a tool to parse files and extract their relevant metadata into objects and vocabularies in openBIS.
This tool works using the package [`bam-masterdata`](https://github.com/BAMresearch/bam-masterdata) to grep the object types definitions. Each parser is loaded via entry points from individual GitHub repositories (see e.g., [`masterdata-parser-example`](https://github.dev/BAMresearch/masterdata-parser-example)).
## Usage
If you want to use this tool for your own openBIS instance, we recommend:
1. Fork the `bam-masterdata` repository.
2. Fork this `openbis-upload-helper` repository.
3. Adapt both forks to your specific openBIS instance.
4. Create your own GitHub repositories for your parsers.
5. Deploy the app at your convenience.
## Development
If you want to develop locally this package, clone the project and enter in the workspace folder:
```sh
git clone https://github.com/BAMresearch/openbis-upload-helper.git
cd openbis-upload-helper
```
Create a virtual environment (you can use Python>3.10) in your workspace:
```sh
python3 -m venv .venv
source .venv/bin/activate
```
Run the following commands to pip install the dependencies:
```sh
pip install --upgrade pip
pip install uv
uv pip install -e '.[dev,parsers]'
```
**Note**: The parsers are loaded as entry points via the optional dependencies in the `pyproject.toml` of this repository. More info under Parsers and Dynamic Plugins.
### Configuration: settings.ini
A sample settings file is provided at:
`openbis_upload_helper\uploader\settings.ini.example`
Copy this file to the project root and rename it to **`settings.ini`**. **Do not commit or push this file** to the repository because it contains secrets.
What to set inside
- **DEBUG**: set to `True` for local **development** so the Django development server will serve static files automatically.
- **SECRET_KEY**: Django secret key for cryptographic signing.
- **SECRET_ENCRYPTION_KEY**: application-specific encryption key used by this project.
- **CSRF_TRUSTED_ORIGINS**: include your local host entries (for example `http://127.0.0.1:8000`). Use full scheme + host entries.
- **OPENBIS_URL**: openBIS instance URL for uploading the data.
- **SPACE_FILTER**: List of space codes to exclude from display (e.g. ["DEFAULT", "SETTINGS"])
- **UPLOAD_SIZE_LIMIT**: Maximum total size of the uploaded files in bytes. Example: `UPLOAD_SIZE_LIMIT=10000000000` for 10 GiB. The value must be an integer (bytes).
- **UPLOAD_TIMEOUT_SECONDS**: Maximum time allowed for the file upload and extraction phase (in seconds). If the loader spends longer than this value while receiving/writing/extracting files, the upload will be aborted. Example: `UPLOAD_TIMEOUT_SECONDS=3000`.
Static files note
- With **DEBUG=True**, the development server serves static files automatically.
- With **DEBUG=False** you must collect static files and serve them with a proper web server.
Database and admin steps
- Run migrations before starting the server:
`python openbis_upload_helper\manage.py migrate`
- If you need the Django admin site, create a superuser after migrating:
`python openbis_upload_helper\manage.py createsuperuser`
Order recommendation (development)
1. Copy and configure **`settings.ini`** or **`.env`**.
2. `python openbis_upload_helper\manage.py migrate`
3. (Optional) `python openbis_upload_helper\manage.py createsuperuser`
### Run the app
You can locally deploy the app for development by running:
```sh
python openbis_upload_helper/manage.py runserver
```
This will run the Django app server:
```sh
Performing system checks...
System check identified no issues (0 silenced).
June 05, 2025 - 06:24:20
Django version 5.2.1, using settings 'uploader.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
```
Simply click on the localhost address, `http://127.0.0.1:8000/`, to launch the app locally.
# Parsers and Dynamic Plugins
This project discovers parser plugins at runtime using Python entry points.
## How it works
1. **Include the parser dependency**
Add the parser to your `pyproject.toml` under `[project.optional-dependencies]` in the `parsers` section.
2. **Install the parser**
After adding the dependency, install it with:
```sh
uv pip install -e .[parsers]
```
3. **Load the parser at runtime**
The application uses the loader function [get_entry_point_parsers]
It scans the configured entry point group and returns the available parser plugins dynamically.