{"id":20450749,"url":"https://github.com/valiantljk/sci-swift","last_synced_at":"2025-04-13T02:30:34.761Z","repository":{"id":92382979,"uuid":"106732572","full_name":"valiantljk/sci-swift","owner":"valiantljk","description":"Scientific Object Store based on Openstack Swift","archived":false,"fork":false,"pushed_at":"2018-09-06T20:24:59.000Z","size":3433,"stargazers_count":3,"open_issues_count":4,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-26T20:02:29.515Z","etag":null,"topics":["openstack-swift","sci-swift","scientific"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/valiantljk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-12T18:49:22.000Z","updated_at":"2023-03-05T00:51:50.000Z","dependencies_parsed_at":"2023-05-17T02:00:21.355Z","dependency_job_id":null,"html_url":"https://github.com/valiantljk/sci-swift","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valiantljk%2Fsci-swift","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valiantljk%2Fsci-swift/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valiantljk%2Fsci-swift/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valiantljk%2Fsci-swift/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/valiantljk","download_url":"https://codeload.github.com/valiantljk/sci-swift/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248657796,"owners_count":21140841,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["openstack-swift","sci-swift","scientific"],"created_at":"2024-11-15T10:54:57.505Z","updated_at":"2025-04-13T02:30:34.749Z","avatar_url":"https://github.com/valiantljk.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Sci-Swift\nScientific Object Store based on Openstack Swift\n\n## Why Sci-Swift\n\n* **_No code change_** required for existing HDF5 applications to use object store (A promising HPC storage for replacing current PFS) \n* Enable HDF5 to work with **_any kind of formats/storage_** using your favourite **_python_** library.\n  e.g, FITs, Parquet, ROOT, TensorFlow, etc\n\n## Design\n\n![sciswift2](https://user-images.githubusercontent.com/1396867/33396383-2f32fdd0-d4fd-11e7-9038-a87dcc501d2d.png)\n\n## QuickStart ( May skip step 1-3, with ./quickstart) \n(Step 1-3 building time ~ 2mins 35 sec)\n\nPython 2.7.13 :: Anaconda 2.3.0 (64-bit)\n\n**Step 1**. Create a data only docker container\n ```\n  docker run -v /srv --name SWIFT_DATA busybox\n```\n**Step 2**. Create a openstack object store (with volume from the existing data container, binding host port 12345 with container port 8080)\n```\n  docker run -d --security-opt seccomp=unconfined -p 12345:8080 --volumes-from SWIFT_DATA -t valiantljk/sciswift:v3 # need 'docker login' first\n```\n\n**Step 3**. Start openstack swift\n```\n  docker ps # to get the containerID of the current running sciswift container \n  docker exec -it containerID bash # launch a shell for mornitoring purpose, i.e., you will see the openstack swift logs\n  ./usr/local/bin/startmain.sh      # start the openstack swift server, i.e., launching account/object/proxy server process \n```\n**Step 4**. Test openstack swift and sciswift\n\n   * Test connection to openstack swift from host\n```\n  swift -A http://127.0.0.1:12345/auth/v1.0 -U test:tester -K testing stat # can observe the previous mornitoring shell for logs\n```\n   * Test connection to openstack swift within sciswift container \n```\n  docker exec -it containerID bash # launch an interactive shell \n  swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat \n```\n   * Test with python-swiftclient\n```\n  cd /build/sci-swift/examples   # within the same container launched in last step\n  source sourceme\n  python upload.py swift 1.h5 wind 100 100  # create a 100*100 array with h5py and upload this file to swift store\n  python list.py swift # list all objects in 'swift' container\n```\n   * Test with HDF5 Swift VOL, the real SciSwift feature \n```\n  cd /build/sci-swift/vol_plugins/test\n  make\n  ./sciswift swift\n```\n\n\n## Code Structure\n```\n.\n|-- LICENSE\n|-- README.md\n|-- examples\n|   |-- list.py\n|   |-- sourceme\n|   `-- upload.py\n|-- src\n|   |-- Makefile\n|   |-- __init__.py\n|   |-- swift.c\n|   |-- swift.h\n|   |-- swift.py\n|   |-- swift_vol.c\n|   `-- swift_vol.h\n`-- test\n    |-- Makefile\n    |-- __init__.py\n    |-- readme\n    |-- test.py\n    |-- test_swift.c\n    |-- test_swift.py\n    `-- test_swift_vol.c\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvaliantljk%2Fsci-swift","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvaliantljk%2Fsci-swift","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvaliantljk%2Fsci-swift/lists"}