https://github.com/unytics/data-api
Serve Data with Low-Latency
https://github.com/unytics/data-api
alloydb api bigquery bigtable cloudsql data datastore firestore spanner
Last synced: 5 months ago
JSON representation
Serve Data with Low-Latency
- Host: GitHub
- URL: https://github.com/unytics/data-api
- Owner: unytics
- License: mit
- Created: 2024-11-26T15:40:17.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-13T16:51:30.000Z (10 months ago)
- Last Synced: 2025-07-19T18:47:52.275Z (5 months ago)
- Topics: alloydb, api, bigquery, bigtable, cloudsql, data, datastore, firestore, spanner
- Language: Python
- Homepage:
- Size: 98.6 KB
- Stars: 5
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Serve Data with Low-Latency
---
## 1. Features 🎍
**`data-api` discovers your datastore data & exposes them as a REST read-only API.**
> 💡 By datastore we mean [firestore in datastore mode](https://cloud.google.com/datastore/docs/concepts/overview).
Features include:
- generate open-api spec definition
- generate Swagger-UI documentation page
- expose datastore data via key-value or property-filtering.
- cache responses with configurable duration
- manage permissions with api-keys or openID tokens.
> 💡 `data-api` is great to create a low-latency API for your **BigQuery** data
> (that you can export to datastore with one sql query. More below).
## 2. Deploy 🚀
Run
```bash
gcloud run deploy data-api \
--source . \
--set-env-vars "PROJECT=$PROJECT,DATABASE=$DATABASE"
```
with:
- `$PROJECT` the Google Cloud Project where you datastore database resides
- `$DATABASE` the name of your datastore database (if not given, it will use the default database).
You can then get the url of your `data-api` Cloud Run Service by running:
```bash
gcloud run services describe --format "value(status.url)"
```
## 3. Exposed Routes 🚚
> 💡 **Datastore Reminder**:
>
> - Data in datastore is organized in a hierarchy: `database/namespace/kind` (like the `database/schema/table` hierarchy in relational databases).
> - Inside a `kind` (think `table`), the `entities` (think `rows`) have a `key` (a string or integer) and a `value` (a dict).
`data-api` Cloud Run service:
- exposes the data of the `database` defined as environment variable at deploy time (see above).
- considers each `namespace` as a different api (which has its own open-api spec definition and Swagger UI).
- exposes the following routes (with `GET` method):
| url | Description |
|------------------------------------|-------------------------------------------------------------------------------------- |
| `/` | Redirects to `/api/` |
| `/api/` | Returns the list of namespaces in `database` |
| `/api//` | Returns details on `namespace` api including its `kinds` and urls |
| `/api//openapi.json` | Returns the openapi spec definition of the `namespace` api |
| `/api//swagger-ui.html` | Returns the Swagger UI (documentation portal) of the `namespace` api |
| `/api///` | Returns a list of entity values of `kind` |
| `/api///` | Returns the entity value of `key` |
| `/api///?foo=bar` | Returns a list of entity values of `kind` for which `foo` property is equal to `bar` |
## 4. [BONUS] Export BigQuery table to datastore ✍️
`data-api` is great to create a low-latency API for your **BigQuery** data.
You can export a BigQuery table into datastore with one sql command to make it available to `data-api`:
```sql
call bigfunctions.eu.export_table_to_datastore(
'your-project.dataset.table',
'user_id',
'your-project/your-database/default/users'
);
```
> 💡 If you don't want to use the public bigfunction, you can deploy the function in your own BigQuery project. Check the [function documentation](https://unytics.io/bigfunctions/bigfunctions/export_table_to_datastore/).
## 5. Contribute 👋
Any contribution is more than welcome 🤗!
- Add a ⭐ on the repo to show your support
- [Join our Slack](https://join.slack.com/t/unytics/shared_invite/zt-1gbv491mu-cs03EJbQ1fsHdQMcFN7E1Q) and talk with us
- Raise an issue to raise a bug or suggest improvements
- Open a PR!
## 6. TODO 📝
- manage permissions with a metadata key-value
- make queries work when filtering with integer (using the datastore schema)
- extend this concept to other backends