https://github.com/valiantljk/sci-swift
Scientific Object Store based on Openstack Swift
https://github.com/valiantljk/sci-swift
openstack-swift sci-swift scientific
Last synced: 6 months ago
JSON representation
Scientific Object Store based on Openstack Swift
- Host: GitHub
- URL: https://github.com/valiantljk/sci-swift
- Owner: valiantljk
- License: other
- Created: 2017-10-12T18:49:22.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2018-09-06T20:24:59.000Z (about 7 years ago)
- Last Synced: 2025-03-26T20:02:29.515Z (7 months ago)
- Topics: openstack-swift, sci-swift, scientific
- Language: C
- Size: 3.27 MB
- Stars: 3
- Watchers: 2
- Forks: 3
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## Sci-Swift
Scientific Object Store based on Openstack Swift
## Why Sci-Swift
* **_No code change_** required for existing HDF5 applications to use object store (A promising HPC storage for replacing current PFS)
* Enable HDF5 to work with **_any kind of formats/storage_** using your favourite **_python_** library.
e.g, FITs, Parquet, ROOT, TensorFlow, etc
## Design
## QuickStart ( May skip step 1-3, with ./quickstart)
(Step 1-3 building time ~ 2mins 35 sec)
Python 2.7.13 :: Anaconda 2.3.0 (64-bit)
**Step 1**. Create a data only docker container
```
docker run -v /srv --name SWIFT_DATA busybox
```
**Step 2**. Create a openstack object store (with volume from the existing data container, binding host port 12345 with container port 8080)
```
docker run -d --security-opt seccomp=unconfined -p 12345:8080 --volumes-from SWIFT_DATA -t valiantljk/sciswift:v3 # need 'docker login' first
```
**Step 3**. Start openstack swift
```
docker ps # to get the containerID of the current running sciswift container
docker exec -it containerID bash # launch a shell for mornitoring purpose, i.e., you will see the openstack swift logs
./usr/local/bin/startmain.sh # start the openstack swift server, i.e., launching account/object/proxy server process
```
**Step 4**. Test openstack swift and sciswift
* Test connection to openstack swift from host
```
swift -A http://127.0.0.1:12345/auth/v1.0 -U test:tester -K testing stat # can observe the previous mornitoring shell for logs
```
* Test connection to openstack swift within sciswift container
```
docker exec -it containerID bash # launch an interactive shell
swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat
```
* Test with python-swiftclient
```
cd /build/sci-swift/examples # within the same container launched in last step
source sourceme
python upload.py swift 1.h5 wind 100 100 # create a 100*100 array with h5py and upload this file to swift store
python list.py swift # list all objects in 'swift' container
```
* Test with HDF5 Swift VOL, the real SciSwift feature
```
cd /build/sci-swift/vol_plugins/test
make
./sciswift swift
```
## Code Structure
```
.
|-- LICENSE
|-- README.md
|-- examples
| |-- list.py
| |-- sourceme
| `-- upload.py
|-- src
| |-- Makefile
| |-- __init__.py
| |-- swift.c
| |-- swift.h
| |-- swift.py
| |-- swift_vol.c
| `-- swift_vol.h
`-- test
|-- Makefile
|-- __init__.py
|-- readme
|-- test.py
|-- test_swift.c
|-- test_swift.py
`-- test_swift_vol.c
```