Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/digital-blueprint/relay-blob-bundle
The blob bundle provides an API for abstracting different shared storage systems.
https://github.com/digital-blueprint/relay-blob-bundle
dbp-relay storage-api
Last synced: about 2 months ago
JSON representation
The blob bundle provides an API for abstracting different shared storage systems.
- Host: GitHub
- URL: https://github.com/digital-blueprint/relay-blob-bundle
- Owner: digital-blueprint
- License: agpl-3.0
- Created: 2023-05-03T13:49:19.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-04-11T14:31:24.000Z (9 months ago)
- Last Synced: 2024-04-12T00:10:43.155Z (8 months ago)
- Topics: dbp-relay, storage-api
- Language: PHP
- Homepage:
- Size: 793 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
Relay-Blob Bundle README
================================
# DbpRelayBlobBundle
[GitHub](https://github.com/digital-blueprint/relay-blob-bundle) |
[Packagist](https://packagist.org/packages/dbp/relay-blob-bundle) |
[Changelog](https://github.com/digital-blueprint/relay-blob-bundle/blob/main/CHANGELOG.md)
The blob bundle provides an API for abstracting different shared storage systems.
You can upload a file unauthorized via the API to a configured bucket and gets a short ephemeral link.
Authentication takes place via signed requests.
The file is attached to the bucket, not to an owner.
A bucket can be an application or an application space. For example, you can have two buckets with a different target group for one application.
A bucket is configured in the config file.
## Requirements
You need a DbpRelayBlobConnector bundle installed to make this bundle working. E.g. [DbpRelayBlobConnectorFilesystemBundle](https://github.com/digital-blueprint/relay-blob-connector-filesystem-bundle)
## Bundle installation
You can install the bundle directly from [packagist.org](https://packagist.org/packages/dbp/relay-blob-bundle).
```bash
composer require dbp/relay-blob-bundle
```
## Integration into the Relay API Server
* Add the bundle to your `config/bundles.php` in front of `DbpRelayCoreBundle`:
```php
...
Dbp\Relay\BlobBundle\DbpRelayBlobBundle::class => ['all' => true],
Dbp\Relay\CoreBundle\DbpRelayCoreBundle::class => ['all' => true],
];
```
If you were using the [DBP API Server Template](https://github.com/digital-blueprint/relay-server-template)
as template for your Symfony application, then this should have already been generated for you.
* Run `composer install` to clear caches
## Configuration
The bundle has multiple configuration values that you can specify in your
app, either by hard-coding it, or by referencing an environment variable.
For this create `config/packages/dbp_relay_blob.yaml` in the app with the following
content:
```yaml
dbp_relay_blob:
database_url: %env(resolve:DATABASE_URL)%'
reporting_interval: "0 11 * * MON" # when notification cronjob should run
cleanup_interval: "*/5 * * * *" # when cleanup cronjob should run
file_integrity_checks: true # if file integrity checks should be performed periodically
integrity_check_interval: "0 0 1 * *" # when integrity check cronjob should run
bucket_size_check_interval: "0 2 * * 1" # when bucket size check cronjob should run
quota_warning_interval: "0 7 * * *" # when bucket quota should be checked and if needed warning emails should be sent
additional_auth: true # enable client credential flow
buckets:
test_bucket:
service: 'Dbp\Relay\BlobConnectorFilesystemBundle\Service\FilesystemService' # The path to a dbp relay blob connector service
internal_bucket_id: '019072b9-7736-7430-aabd-ad1bbeeebacf' # A given internal id for a bucket
bucket_id: 'test-bucket' # friendly name of the bucket thats also used for the request
key: '12345' # public key for signed request
quota: 500 # Max quota in MB
notify_when_quota_over: 70 # percent of quota when the bucket owner should be notified that the storage is running out
report_when_expiry_in: 'P30D' # duration of how much in advance a bucket owner or user should be warned about the deletion of files
bucket_owner: '[email protected]' # Email who will be notified when quota is reached
link_expire_time: 'P7D' # Max expire time of sharelinks in ISO 8601
warn_quota: # Notification configuration how emails are sent when the quota is about to be reached
dsn: '%env(TUGRAZ_MAILER_TRANSPORT_DSN)%'
from: '[email protected]'
to: '[email protected]'
subject: 'Blob Bucket Quota Warning'
html_template: 'emails/warn-quota.html.twig'
reporting: # Reporting configuration how emails are sent when files are about to expire
dsn: '%env(TUGRAZ_MAILER_TRANSPORT_DSN)%'
from: '[email protected]'
to: '[email protected]' # this email is a fallback, if no email field of a file is set
subject: 'Blob file Deletion Report'
html_template: 'emails/reporting.html.twig'
integrity: # Integrity check configuration how emails are sent when file hash or metadata hash do not match with the saved file or metadata
dsn: '%env(TUGRAZ_MAILER_TRANSPORT_DSN)%'
from: '[email protected]'
to: '[email protected]'
subject: 'Blob File Integrity Check Report'
html_template: 'emails/integrity.html.twig'
additional_types:
- generic_id_card: '%kernel.project_dir%/config/packages/schemas/relay-blob-bundle/test-bucket/generic_id_card.json'
```
For more info on bundle configuration see .
## Development & Testing
* Install dependencies: `composer install`
* Run tests: `composer test`
* Run linters: `composer run lint`
* Run cs-fixer: `composer run cs-fix`
## Bundle dependencies
Don't forget you need to pull down your dependencies in your main application if you are installing packages in a bundle.
```bash
# updates and installs dependencies of dbp/relay-blob-bundle
composer update dbp/relay-blob-bundle
```
## Scripts
### Database migration
Run this script to migrate the database. Run this script after installation of the bundle and
after every update to adapt the database to the new source code.
```bash
php bin/console doctrine:migrations:migrate --em=dbp_relay_blob_bundle
```
## Error codes
See the [API documentation](doc/api.md).
## CronJobs
### Cleanup Cronjob
This cronjob is for cleanup purposes. It starts every hour and deletes old files.
### Send Report Cronjob
This cronjob sends reports to given email addresses, or the bucket owner.
In these reports there are all files which are going to be deleted in the timeframe specified in the config.
The email address are attached to these files or there is a default in the config.
This cronjob starts every Monday at 9 o'clock in the Morning (UTC).
### Quota Warning
This cronjob sends reports to the in `warn_quota` configured email.
This email lets the configured person know before the bucket space is used up.
An email gets sent after the bucket is filled to a certain percentage, configured with the option `notify_when_quota_over`.
The option `quota_warning_interval` indicates in which interval this should get checked.
### Bucket Size Check
This cronjob sends reports to the in `bucket_size` configured email.
This email lets the configured person know if a bucket is inconsistent, either because the number of entries in the table is different than in the storage backend or because the sum of filesizes stored in the database is different than the sum of filesize of the actually stored files.
An email is sent if some inconsistencies are detected.
The option `bucket_size_check_interval` indicates in which interval this should get checked.
### Integrity Check
This cronjobs sends reports to the in `integrity` configured email.
This emails lets the configured person know if the metadata or file integrity checks fail. This means, that the metadata or file hashes, which are stored upon upload, are different then the hashes of the metadata or file that are currently stored. This could indicate bitrot or manual changes in the database or file.
An email is sent if some inconsistencies are detected.
The option `integrity_check_interval` indicates in which interval this should get checked.
The option `file_integrity_checks` can be used to enable or disable the integrity checks.