{"id":22338778,"url":"https://github.com/unytics/data-api","last_synced_at":"2025-07-30T00:31:02.495Z","repository":{"id":264882198,"uuid":"894573667","full_name":"unytics/data-api","owner":"unytics","description":"Serve Data with Low-Latency","archived":false,"fork":false,"pushed_at":"2025-03-13T16:51:30.000Z","size":101,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-19T18:47:52.275Z","etag":null,"topics":["alloydb","api","bigquery","bigtable","cloudsql","data","datastore","firestore","spanner"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/unytics.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-26T15:40:17.000Z","updated_at":"2025-03-30T09:54:39.000Z","dependencies_parsed_at":"2024-12-11T16:32:22.353Z","dependency_job_id":"17451350-0180-4557-bd29-465c915c81c3","html_url":"https://github.com/unytics/data-api","commit_stats":null,"previous_names":["unytics/data-api"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/unytics/data-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unytics%2Fdata-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unytics%2Fdata-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unytics%2Fdata-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unytics%2Fdata-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/unytics","download_url":"https://codeload.github.com/unytics/data-api/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unytics%2Fdata-api/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267785723,"owners_count":24144117,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-29T02:00:12.549Z","response_time":2574,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alloydb","api","bigquery","bigtable","cloudsql","data","datastore","firestore","spanner"],"created_at":"2024-12-04T07:05:12.497Z","updated_at":"2025-07-30T00:31:02.435Z","avatar_url":"https://github.com/unytics.png","language":"Python","readme":"\u003cp align=\"center\"\u003e\n  \u003cimg class=\"hero-image\" src=\"https://github.com/user-attachments/assets/15d2f888-d0e2-493a-adaf-5338fd6dcf53\" alt=\"logo_and_name\" style=\"width: 100%; max-width: 300px\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003cem\u003e\u003cstrong\u003eServe Data with Low-Latency\u003c/strong\u003e\u003c/em\u003e\n\u003c/p\u003e\n\n\n---\n\n\u003cbr\u003e\n\n## 1. Features 🎍\n\n**`data-api` discovers your datastore data \u0026 exposes them as a REST read-only API.**\n\n\u003e 💡 By datastore we mean [firestore in datastore mode](https://cloud.google.com/datastore/docs/concepts/overview).\n\nFeatures include:\n\n- generate open-api spec definition\n- generate Swagger-UI documentation page\n- expose datastore data via key-value or property-filtering.\n- cache responses with configurable duration\n- manage permissions with api-keys or openID tokens.\n\n\u003e 💡 `data-api` is great to create a low-latency API for your **BigQuery** data\n\u003e (that you can export to datastore with one sql query. More below).\n\n\n\u003cbr\u003e\n\n## 2. Deploy 🚀\n\nRun\n\n```bash\ngcloud run deploy data-api \\\n  --source . \\\n  --set-env-vars \"PROJECT=$PROJECT,DATABASE=$DATABASE\"\n```\n\nwith:\n\n- `$PROJECT` the Google Cloud Project where you datastore database resides\n- `$DATABASE` the name of your datastore database (if not given, it will use the default database).\n\n\nYou can then get the url of your `data-api` Cloud Run Service by running:\n\n```bash\ngcloud run services describe --format \"value(status.url)\"\n```\n\n\u003cbr\u003e\n\n## 3. Exposed Routes 🚚\n\n\u003e 💡 **Datastore Reminder**:\n\u003e\n\u003e - Data in datastore is organized in a hierarchy: `database/namespace/kind` (like the `database/schema/table` hierarchy in relational databases).\n\u003e - Inside a `kind` (think `table`), the `entities` (think `rows`) have a `key` (a string or integer) and a `value` (a dict).\n\n\u003cbr\u003e\n\n`data-api` Cloud Run service:\n\n- exposes the data of the `database` defined as environment variable at deploy time (see above).\n- considers each `namespace` as a different api (which has its own open-api spec definition and Swagger UI).\n- exposes the following routes (with `GET` method):\n\n| url                                | Description                                                                                      |\n|------------------------------------|--------------------------------------------------------------------------------------            |\n| `/`                                | Redirects to `/api/`                                                                             |\n| `/api/`                            | Returns the list of namespaces in `database`                                                     |\n| `/api/\u003cnamespace\u003e/`                | Returns details on `namespace` api including its `kinds` and urls                                |\n| `/api/\u003cnamespace\u003e/openapi.json`    | Returns the openapi spec definition of the `namespace` api                                       |\n| `/api/\u003cnamespace\u003e/swagger-ui.html` | Returns the Swagger UI (documentation portal) of the `namespace` api                             |\n| `/api/\u003cnamespace\u003e/\u003ckind\u003e/`         | Returns a list of entity values of `kind`                                                        |\n| `/api/\u003cnamespace\u003e/\u003ckind\u003e/\u003ckey\u003e`    | Returns the entity value of `key`                                                                |\n| `/api/\u003cnamespace\u003e/\u003ckind\u003e/?foo=bar` | Returns a list of entity values of `kind` for which `foo` property is equal to `bar`             |\n\n\u003cbr\u003e\n\n\n\n## 4. [BONUS] Export BigQuery table to datastore ✍️\n\n`data-api` is great to create a low-latency API for your **BigQuery** data.\n\nYou can export a BigQuery table into datastore with one sql command to make it available to `data-api`:\n\n```sql\ncall bigfunctions.eu.export_table_to_datastore(\n  'your-project.dataset.table',\n  'user_id',\n  'your-project/your-database/default/users'\n);\n```\n\n\u003e 💡 If you don't want to use the public bigfunction, you can deploy the function in your own BigQuery project. Check the [function documentation](https://unytics.io/bigfunctions/bigfunctions/export_table_to_datastore/).\n\n\n\n\n\u003cbr\u003e\n\n\n## 5. Contribute 👋\n\nAny contribution is more than welcome 🤗!\n\n- Add a ⭐ on the repo to show your support\n- [Join our Slack](https://join.slack.com/t/unytics/shared_invite/zt-1gbv491mu-cs03EJbQ1fsHdQMcFN7E1Q) and talk with us\n- Raise an issue to raise a bug or suggest improvements\n- Open a PR!\n\n\u003cbr\u003e\n\n\n## 6. TODO 📝\n\n- manage permissions with a metadata key-value\n- make queries work when filtering with integer (using the datastore schema)\n- extend this concept to other backends\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funytics%2Fdata-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funytics%2Fdata-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funytics%2Fdata-api/lists"}