https://github.com/pyronear/pyro-storage
Data curation for wildfire detection
https://github.com/pyronear/pyro-storage
backend-api docker
Last synced: 5 months ago
JSON representation
Data curation for wildfire detection
- Host: GitHub
- URL: https://github.com/pyronear/pyro-storage
- Owner: pyronear
- License: apache-2.0
- Created: 2022-05-19T11:30:06.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2024-07-06T08:53:17.000Z (almost 2 years ago)
- Last Synced: 2024-10-29T13:27:53.210Z (over 1 year ago)
- Topics: backend-api, docker
- Language: Python
- Homepage: https://pyronear.org/pyro-storage
- Size: 774 KB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Data curation API
The building blocks of our data curation API.
## Quick Tour
### Running/stopping the service
You can run the API containers using this command:
```shell
make run
```
You can now navigate to `http://localhost:8080/docs` to interact with the API (or do it through HTTP requests) and explore the documentation.
In order to stop the service, run:
```shell
make stop
```
### How is the database organized
The back-end core feature is to interact with the metadata tables. For the service to be useful for data curation, multiple tables/object types are introduced and described as follows:
#### Access-related tables
- Accesses: stores the hashed credentials and access level for users & devices.
#### Core data curation worklow tables
- Media: metadata of a picture and its storage bucket key.
- Annotations: metadata of an annotation file and its storage bucket key.
### What is the full data curation workflow through the API
The API has been designed to provide, for each data entry:
- timestamp
- the picture that was uploaded
- the annotation file associated with that picture
With the previously described tables, here are all the steps to upload a data entry:
- Prerequisites (ask the instance administrator): register user
- Create a media object & upload content: save the picture metadata and upload the image content.
- Create an annotation object & upload content: save the annotation metadata and upload the annotation content.
## Installation
### Prerequisites
- [Docker](https://docs.docker.com/engine/install/)
- [Docker compose](https://docs.docker.com/compose/)
- [Make](https://www.gnu.org/software/make/) (optional)
The project was designed so that everything runs with Docker orchestration (standalone virtual environment), so you won't need to install any additional libraries.
## Configuration
In order to run the project, you will need to specific some information, which can be done using a `.env` file.
This file will have to hold the following information:
- `S3_ACCESS_KEY`: public key to access to the [S3 storage service](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html)
- `S3_SECRET_KEY`: private key to access the resource.
- `S3_REGION`: your S3 bucket is geographically identified by its location's region
- `S3_ENDPOINT_URL`: the URL providing a S3 endpoint by your cloud provider
- `BUCKET_NAME`: the name of the storage bucket
Optionally, the following information can be added:
- `SENTRY_DSN`: the URL of the [Sentry](https://sentry.io/) project, which monitors back-end errors and report them back.
- `SERVER_NAME`: the server tag to apply to events.
- `CORS_ORIGIN`: comma-separated list of allowed origins
So your `.env` file should look like something similar to:
```
S3_ACCESS_KEY=YOUR_ACCESS_KEY
S3_SECRET_KEY=YOUR_SECRET_KEY
S3_REGION=bucket-region
S3_ENDPOINT_URL='https://s3.mydomain.com/'
BUCKET_NAME=my_storage_bucket_name
SENTRY_DSN='https://replace.with.you.sentry.dsn/'
SERVER_NAME=my_storage_bucket_name
```
The file should be placed at the root folder of your local copy of the project.
## More goodies
### Documentation
The full package documentation is available [here](https://pyronear.org/pyro-storage) for detailed specifications.
### Python client
This project is a REST-API, and you can interact with the service through HTTP requests. However, if you want to ease the integration into a Python project, take a look at our [Python client](client).
## Contributing
Any sort of contribution is greatly appreciated!
You can find a short guide in [`CONTRIBUTING`](CONTRIBUTING.md) to help grow this project!
## License
Distributed under the Apache 2.0 License. See [`LICENSE`](LICENSE) for more information.