{"id":20405730,"url":"https://github.com/cloudquery/python-plugin-template","last_synced_at":"2025-04-12T15:05:58.802Z","repository":{"id":219249689,"uuid":"704562929","full_name":"cloudquery/python-plugin-template","owner":"cloudquery","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-08T08:47:55.000Z","size":807,"stargazers_count":6,"open_issues_count":1,"forks_count":0,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-12T15:05:40.453Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudquery.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-10-13T14:23:36.000Z","updated_at":"2025-04-08T08:47:59.000Z","dependencies_parsed_at":"2024-01-26T11:29:48.516Z","dependency_job_id":"edfacf95-d98f-469c-bf38-1eed99585095","html_url":"https://github.com/cloudquery/python-plugin-template","commit_stats":null,"previous_names":["cloudquery/python-plugin-template"],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudquery%2Fpython-plugin-template","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudquery%2Fpython-plugin-template/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudquery%2Fpython-plugin-template/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudquery%2Fpython-plugin-template/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudquery","download_url":"https://codeload.github.com/cloudquery/python-plugin-template/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248586235,"owners_count":21128997,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T05:12:47.355Z","updated_at":"2025-04-12T15:05:58.764Z","avatar_url":"https://github.com/cloudquery.png","language":"Python","readme":"# Python Source Plugin Template\n\nThis repo contains everything you need to get started with building a new plugin.\nTo get started all you need to do is change a few names, define some tables, and write an API Client to populate the tables.\n\n[![Mastering CloudQuery: How to build a Source Plugin in Python](https://i.ytimg.com/vi/TSbGHz5Z09M/maxresdefault.jpg)](https://youtu.be/TSbGHz5Z09M \"Mastering CloudQuery: How to build a Source Plugin in Python\")\n\n## Key files \u0026 classes\n\n- [plugin/tables/items.py](plugin/tables/items.py)\n  - `Items` - A boilerplate table definition\n  - `ItemResolver` - A boilerplate table resolver\n- [plugin/example/client.py](plugin/example/client.py)\n  - `ExampleClient` - A boilerplate API Client\n- [plugin/client/client.py]\n  - `Spec` - Defines the CloudQuery Config\n  - `Client` (uses: `ExampleClient`) - Wraps your API Client\n- [plugin/plugin.py](plugin/plugin.py)\n  - `ExamplePlugin` - The plugin registration / how CloudQuery knows what tables your plugin exposes.\n\n## Getting started\n\n### Defining your tables\n\nThe first thing you need to do is identify the tables you want to create with your plugin.\nConventionally, CloudQuery plugins have a direct relationship between tables and API responses.\n\nFor example:\nIf you had an API endpoint https://api.example.com/items/{num} and for each value of `num` it provided an object\n\n```json\n{\n   \"num\": {{num}},\n   \"date\": \"2023-10-12\",\n   \"title\": \"A simple example\"\n}\n```\n\nThen you would design the table class as\n\n```python\nclass Items(Table):\n    def __init__(self) -\u003e None:\n        super().__init__(\n            name=\"item\",\n            title=\"Item\",\n            columns=[\n                Column(\"num\", pa.uint64(), primary_key=True),\n                Column(\"date\", pa.date64()),\n                Column(\"title\", pa.string()),\n            ],\n        )\n    ...\n```\n\nCreating one table for each endpoint that you want to capture.\n\n### API Client\n\nNext you'll need to define how the tables are retrieved, it's recommended to implement this as a generator, as per the example in `plugin/example/client.py`.\n\n### Spec\n\nHaving written your API Client you will have, identified the authentication and/or operational variables needed.\nAdding these to the CloudQuery config spec can be done by editing the `Spec` `dataclass` using standard python, and adding validation where needed.\n\n### Plugin\n\nFinally, you need to edit the `plugin.py` file to set the plugin name and version, and add the `Tables` to the `get_tables` function.\n\n### Test run\n\nTo test your plugin you can run it locally.\n\nTo automatically manage your virtual environment and install the dependencies listed in the `pyproject.toml` you can use `poetry`.\nPoetry is an improved package \u0026 environment manager for Python that uses the standardised `pyproject.toml`, if you don't have it installed you can pull it with `pip install poetry`.\n\nTo install the dependencies into a new virtual environment run `poetry install`.\nIf you have additional dependencies you can add them with `poetry add {package_name}` which will add them to the `pyproject.toml` and install them into the virtual environment.\n\nThen to run the plugin `poetry run main serve`, which will launch the plugin manually as a GRPC service.\n\nWith that running you can adjust the `TestConfig.yaml` to match your plugin and run `cloudquery sync`.\nThis should result in the creation of a sqlite database `db.sqlite` where you can validate your tables are as expected.\n\n### Publishing\n\n1. Update the plugin metadata in [plugin/plugin.py](plugin/plugin.py#L13) to match your team and plugin name.\n2. Run `python main.py package -m \"Initial release\" \"v0.0.1\" .`. `-m` specifies changelog and `v0.0.1` is the version.\n3. Run `cloudquery plugin publish -f` to publish the plugin to the CloudQuery registry.\n\n\u003e More about publishing plugins [here](https://docs.cloudquery.io/docs/developers/publishing-an-addon-to-the-hub)\n\n## Links\n\n- [Architecture](https://www.cloudquery.io/docs/developers/architecture)\n- [Concepts](https://www.cloudquery.io/docs/developers/creating-new-plugin/python-source)\n- [Video tutorial](https://youtu.be/TSbGHz5Z09M)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudquery%2Fpython-plugin-template","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudquery%2Fpython-plugin-template","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudquery%2Fpython-plugin-template/lists"}