{"id":32565836,"url":"https://github.com/stackql/stackql-provider-template","last_synced_at":"2025-10-29T04:53:47.310Z","repository":{"id":314347652,"uuid":"1053004481","full_name":"stackql/stackql-provider-TEMPLATE","owner":"stackql","description":"Template for developing StackQL providers that map cloud APIs to SQL interfaces, enabling developers to query and manage cloud resources using SQL.","archived":false,"fork":false,"pushed_at":"2025-09-11T22:06:23.000Z","size":1038,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-12T00:57:56.775Z","etag":null,"topics":["stackql","stackql-provider"],"latest_commit_sha":null,"homepage":"https://stackql.io/docs","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stackql.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-08T21:15:51.000Z","updated_at":"2025-09-11T22:06:26.000Z","dependencies_parsed_at":"2025-09-12T00:57:59.766Z","dependency_job_id":"3d966bfd-71b8-490c-ad89-0954b75fa563","html_url":"https://github.com/stackql/stackql-provider-TEMPLATE","commit_stats":null,"previous_names":["stackql/stackql-provider-template"],"tags_count":null,"template":true,"template_full_name":null,"purl":"pkg:github/stackql/stackql-provider-TEMPLATE","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackql%2Fstackql-provider-TEMPLATE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackql%2Fstackql-provider-TEMPLATE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackql%2Fstackql-provider-TEMPLATE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackql%2Fstackql-provider-TEMPLATE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stackql","download_url":"https://codeload.github.com/stackql/stackql-provider-TEMPLATE/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackql%2Fstackql-provider-TEMPLATE/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281563798,"owners_count":26522704,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-29T02:00:06.901Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["stackql","stackql-provider"],"created_at":"2025-10-29T04:53:40.331Z","updated_at":"2025-10-29T04:53:47.303Z","avatar_url":"https://github.com/stackql.png","language":"JavaScript","readme":"# StackQL Provider Template\r\n\r\nThis repository serves as a template for developing StackQL providers. It provides a structured workflow and tools to generate, test, and document StackQL providers for various cloud services and APIs.\r\n\r\n## What is StackQL?\r\n\r\n[StackQL](https://github.com/stackql/stackql) is an open-source SQL interface for cloud APIs that allows you to query and manipulate cloud resources using SQL-like syntax. With StackQL, you can:\r\n\r\n- Query cloud resources across multiple providers using familiar SQL syntax\r\n- Join data from different services and providers\r\n- Execute CRUDL operations (`SELECT`, `INSERT`, `UPDATE`, `REPLACE`, `DELETE`) on cloud resources\r\n- Execute lifecycle operations (like starting or stopping vms) using `EXEC`\r\n- Build custom dashboards and reports\r\n- Automate infrastructure operations using [`stackql-deploy`](https://stackql-deploy.io/)\r\n\r\n## What are StackQL Providers?\r\n\r\nStackQL providers are extensions that connect StackQL to specific cloud services or APIs. Each provider:\r\n\r\n1. Defines a schema that maps API endpoints to SQL-like resources and methods\r\n2. Implements authentication mechanisms for the target API\r\n3. Translates SQL operations into API calls\r\n4. Transforms API responses into tabular data that can be queried with SQL\r\n\r\nThis template repository helps you build StackQL providers by converting OpenAPI specifications into StackQL-compatible provider schemas using the `@stackql/provider-utils` package.\r\n\r\n## How StackQL Providers Work\r\n\r\nStackQL providers bridge the gap between SQL queries and REST APIs:\r\n\r\n1. **Resource Mapping**: API endpoints are mapped to SQL-like tables and views\r\n2. **Method Mapping**: API operations are mapped to SQL verbs (`SELECT`, `INSERT`, `UPDATE`, `REPLACE`, `DELETE` and `EXEC`)\r\n3. **Parameter Mapping**: SQL query conditions are translated to API parameters\r\n4. **Response Transformation**: API responses are converted to tabular results\r\n\r\n## Prerequisites\r\n\r\nTo use this template for developing a StackQL provider, you'll need:\r\n\r\n1. An OpenAPI specification for the target API\r\n2. Node.js and `npm` installed on your system\r\n3. StackQL CLI installed (see [StackQL Installation](https://stackql.io/docs/installing-stackql))\r\n4. API credentials for testing your provider\r\n\r\n## Development Workflow\r\n\r\n### 1. Clone this Template\r\n\r\nStart by cloning this template repository and installing dependencies:\r\n\r\n```bash\r\ngit clone https://github.com/stackql/stackql-provider-template.git stackql-provider-myprovider\r\ncd stackql-provider-myprovider\r\nnpm install\r\n```\r\n\r\n### 2. Download the OpenAPI Specification\r\n\r\nObtain the OpenAPI specification for your target API. You can typically find this in the API documentation or developer portal.\r\n\r\n```bash\r\nmkdir -p provider-dev/downloaded\r\ncurl -L https://api-url.example.com/openapi.yaml -o provider-dev/downloaded/provider-name.yaml\r\n```\r\n\r\n\u003e recommended to automate this by creating a script in the `provider-dev/scripts` folder\r\n\r\n### 3. Split the OpenAPI Spec into Service Specs\r\n\r\nBreak down the OpenAPI specification into smaller, service-specific files:\r\n\r\n```bash\r\nnpm run split -- \\\r\n  --provider-name your-provider-name \\\r\n  --api-doc provider-dev/downloaded/provider-name.yaml \\\r\n  --svc-discriminator tag \\\r\n  --output-dir provider-dev/source \\\r\n  --overwrite \\\r\n  --svc-name-overrides \"$(cat \u003c\u003cEOF\r\n{\r\n  \"service_tag_1\": \"service_name_1\",\r\n  \"service_tag_2\": \"service_name_2\"\r\n  # Add more mappings as needed\r\n}\r\nEOF\r\n)\"\r\n```\r\n\r\nThis step organizes the API endpoints into logical services based on OpenAPI tags. You can customize the service names using the `--svc-name-overrides` parameter.  \r\n\r\n`svc-discriminator` can be based upon `tags` in each operation or based upon the path for each operation.\r\n\r\n### 4. Generate Mappings\r\n\r\nGenerate the mapping configuration that connects OpenAPI operations to StackQL resources:\r\n\r\n```bash\r\nnpm run generate-mappings -- \\\r\n  --provider-name your-provider-name \\\r\n  --input-dir provider-dev/source \\\r\n  --output-dir provider-dev/config\r\n```\r\n\r\nThis creates a CSV mapping file that you'll need to edit to define how OpenAPI operations translate to StackQL resources, methods, and SQL verbs.\r\n\r\n### 5. Edit the Mapping File\r\n\r\nEdit the generated `provider-dev/config/all_services.csv` file to add:\r\n- `stackql_resource_name`: The name of the StackQL resource (table/view)\r\n- `stackql_method_name`: The name of the StackQL method\r\n- `stackql_verb`: The SQL verb (`SELECT`, `INSERT`, `UPDATE`, `REPLACE`, `DELETE`, `EXEC`)\r\n\r\nFor example:\r\n```csv\r\nservice,operationId,summary,stackql_resource_name,stackql_method_name,stackql_verb\r\ncompute,listDroplets,List all Droplets,droplets,list,SELECT\r\ncompute,createDroplet,Create a new Droplet,droplets,insert,INSERT\r\ncompute,getDroplet,Retrieve an existing Droplet,droplets,get,SELECT\r\ncompute,deleteDroplet,Delete a Droplet,droplets,delete,DELETE\r\n```\r\n\r\n### 6. Generate the Provider\r\n\r\nTransform the OpenAPI service specs into a StackQL provider:\r\n\r\n```bash\r\nnpm run generate-provider -- \\\r\n  --provider-name your-provider-name \\\r\n  --input-dir provider-dev/source \\\r\n  --output-dir provider-dev/openapi/src/your-provider-name \\\r\n  --config-path provider-dev/config/all_services.csv \\\r\n  --servers '[{\"url\": \"https://api.example.com/v1\"}]' \\\r\n  --provider-config '{\"auth\": {\"credentialsenvvar\": \"PROVIDER_API_KEY\",\"type\": \"header\", \"headerName\": \"Authorization\"}}' \\\r\n  --overwrite\r\n```\r\n\r\nMake any necessary post-processing updates to the generated files, for example:\r\n\r\n```bash\r\nnode provider-dev/scripts/flatten_allOf.cjs\r\nsh provider-dev/scripts/fix_broken_links.sh\r\n```\r\n\r\n\u003e this will vary by provider and may not be necessary in many cases\r\n\r\n### 7. Test the Provider\r\n\r\n#### Start the StackQL Server\r\n\r\n```bash\r\nPROVIDER_REGISTRY_ROOT_DIR=\"$(pwd)/provider-dev/openapi\"\r\nnpm run start-server -- --provider your-provider-name --registry $PROVIDER_REGISTRY_ROOT_DIR\r\n```\r\n\r\n#### Test Metadata Routes\r\n\r\n```bash\r\nnpm run test-meta-routes -- your-provider-name --verbose\r\n```\r\n\r\n#### Run Test Queries\r\n\r\n```bash\r\nPROVIDER_REGISTRY_ROOT_DIR=\"$(pwd)/provider-dev/openapi\"\r\nREG_STR='{\"url\": \"file://'${PROVIDER_REGISTRY_ROOT_DIR}'\", \"localDocRoot\": \"'${PROVIDER_REGISTRY_ROOT_DIR}'\", \"verifyConfig\": {\"nopVerify\": true}}'\r\n./stackql shell --registry=\"${REG_STR}\"\r\n```\r\n\r\nExample test query:\r\n```sql\r\nSELECT * FROM your-provider-name.service_name.resource_name LIMIT 10;\r\n```\r\n\r\nWhen you're done testing, stop the StackQL server:\r\n```bash\r\nnpm run stop-server\r\n```\r\n\r\n### 8. Publish the Provider\r\n\r\nTo publish your provider:\r\n\r\n1. Fork the [stackql-provider-registry](https://github.com/stackql/stackql-provider-registry) repository\r\n2. Copy your provider directory to `providers/src` in a feature branch\r\n3. Follow the [registry release flow](https://github.com/stackql/stackql-provider-registry/blob/dev/docs/build-and-deployment.md)\r\n\r\nTest your published provider in the `dev` registry:\r\n```bash\r\nexport DEV_REG=\"{ \\\"url\\\": \\\"https://registry-dev.stackql.app/providers\\\" }\"\r\n./stackql --registry=\"${DEV_REG}\" shell\r\n```\r\n\r\nPull and verify your provider:\r\n```sql\r\nregistry pull your-provider-name;\r\n-- Run test queries\r\n```\r\n\r\n### 9. Generate Documentation\r\n\r\nProvider doc microsites are built using Docusaurus and published using GitHub Pages.  To genarate and publish comprehensive user docs for your provider, do the following:  \r\n\r\na. Upodate `headerContent1.txt` and `headerContent2.txt` accordingly in `provider-dev/docgen/provider-data/`  \r\n\r\nb. Update the following in `website/docusaurus.config.js`:  \r\n\r\n```js\r\n// Provider configuration - change these for different providers\r\nconst providerName = \"yourprovidername\";\r\nconst providerTitle = \"Your Provider Title\";\r\n```\r\n\r\nc. Then generate docs using...\r\n\r\n```bash\r\nnpm run generate-docs -- \\\r\n  --provider-name your-provider-name \\\r\n  --provider-dir ./provider-dev/openapi/src/your-provider-name/v00.00.00000 \\\r\n  --output-dir ./website \\\r\n  --provider-data-dir ./provider-dev/docgen/provider-data\r\n```  \r\n\r\nd. Test the documentation locally:\r\n```bash\r\ncd website\r\nyarn build\r\nyarn start\r\n```\r\n\r\n### 10. Publish Documentation\r\n\r\nRemove the `.disabled` extension from `.github/workflows/test-web-deploy.yml.disabled` and `.github/workflows/prod-web-deploy.yml.disabled`  \r\n\r\nSet up GitHub Pages in your repository settings, and configure DNS if needed:\r\n\r\n| Source Domain | Record Type | Target |\r\n|---------------|-------------|--------|\r\n| your-provider-name-provider.stackql.io | CNAME | stackql.github.io. |\r\n\r\n## Authentication Configuration\r\n\r\nDifferent APIs require different authentication methods. Here are common authentication configurations:\r\n\r\n### API Key in Header\r\n```json\r\n{\r\n  \"auth\": {\r\n    \"credentialsenvvar\": \"PROVIDER_API_KEY\",\r\n    \"type\": \"header\",\r\n    \"headerName\": \"X-API-Key\"\r\n  }\r\n}\r\n```\r\n\r\n### Bearer Token\r\n```json\r\n{\r\n  \"auth\": {\r\n    \"credentialsenvvar\": \"PROVIDER_TOKEN\",\r\n    \"type\": \"bearer\"\r\n  }\r\n}\r\n```\r\n\r\n### Basic Authentication\r\n```json\r\n{\r\n  \"auth\": {\r\n    \"credentialsenvvar\": \"PROVIDER_BASIC_AUTH\",\r\n    \"type\": \"basic\"\r\n  }\r\n}\r\n```\r\n\r\n### OAuth (Client Credentials Flow)\r\n```json\r\n{\r\n  \"auth\": {\r\n    \"credentialsenvvar\": \"PROVIDER_OAUTH_CONFIG\",\r\n    \"type\": \"oauth-client-credentials\",\r\n    \"tokenUrl\": \"https://auth.example.com/token\"\r\n  }\r\n}\r\n```\r\n\r\n## Contributing\r\n\r\nContributions are welcome! Please feel free to submit a Pull Request.\r\n\r\n## License\r\n\r\nMIT","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackql%2Fstackql-provider-template","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstackql%2Fstackql-provider-template","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackql%2Fstackql-provider-template/lists"}