https://github.com/streamingfast/dstore
Blob stores abstractions. Supports AWS S3, Google Storage, Azure Blob File Storage, and local FS
https://github.com/streamingfast/dstore
Last synced: 5 months ago
JSON representation
Blob stores abstractions. Supports AWS S3, Google Storage, Azure Blob File Storage, and local FS
- Host: GitHub
- URL: https://github.com/streamingfast/dstore
- Owner: streamingfast
- License: apache-2.0
- Created: 2020-04-06T20:07:47.000Z (about 5 years ago)
- Default Branch: develop
- Last Pushed: 2024-03-25T19:34:38.000Z (about 1 year ago)
- Last Synced: 2024-04-01T15:10:30.349Z (about 1 year ago)
- Language: Go
- Homepage: https://dfuse.io
- Size: 208 KB
- Stars: 10
- Watchers: 7
- Forks: 10
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
- jimsghstars - streamingfast/dstore - Blob stores abstractions. Supports AWS S3, Google Storage, Azure Blob File Storage, and local FS (Go)
README
# StreamingFast Storage Abstraction
[](https://pkg.go.dev/github.com/streamingfast/dstore)
[](https://opensource.org/licenses/Apache-2.0)`dstore` is a simple abstraction on top of Local storage and Cloud
storage. It handles commonly used functions to store things (locally,
or on cloud storage providers), list files, delete, etc..It is used by **[StreamingFast](https://github.com/streamingfast/streamingfast)**.
## Features
It currently supports:
* AWS S3 (`s3://[bucket]/path?region=us-east-1`, with [AWS-specific env vars](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html))
* Minio (through the S3 interface)
* Google Storage (`gs://[bucket]/path`, with `GOOGLE_APPLICATION_CREDENTIALS` env var set)
* Azure Blob Storage (`az://[account].[container]/path`, with `AZURE_STORAGE_KEY` env var set)
* Local file systems (including virtual of fused-based) (`file:///` prefix)### Testing
The `storetests` package contains all our integration tests we perform on our store implementation.
Some of the store implementations can be tested directly while few others, from Cloud Providers
essentially, requires some extra environment variables to run. They are skip if the correct
environment variables for the provider is not set.To run the full test suite, you will need to perform the following steps.
First, you will need to have locally a few dependencies:
- [minio](https://github.com/minio/minio)Then, start `minio` server:
```bash
mkdir -p /tmp/minio-tests/store-tests
cd /tmp/minio-tests
minio server .
```Ensure you have access to GCP Storage Bucket, S3 Bucket, then run the full test suite:
```bash
STORETESTS_GS_STORE_URL="gs://streamingfast-developement-random/store-tests"\
STORETESTS_S3_STORE_URL="s3://streamingfast-customer-outbox/store-tests?region=us-east-2"\
STORETESTS_S3_MINIO_STORE_URL="s3://localhost:9000/store-tests?region=none&insecure=true&access_key_id=minioadmin&secret_access_key=minioadmin"\
go test ./...
```> **Note** S3 bucket is not existing anymore, you need to provide your own.
## Contributing
**Issues and PR in this repo related strictly to the dstore library.**
Report any protocol-specific issues in their
[respective repositories](https://github.com/streamingfast/streamingfast#protocols)**Please first refer to the general
[StreamingFast contribution guide](https://github.com/streamingfast/streamingfast/blob/master/CONTRIBUTING.md)**,
if you wish to contribute to this code base.## License
[Apache 2.0](LICENSE)