https://github.com/slok/terraform-provider-dataprocessor
Terraform provider for easy and clean data processing (JQ, YQ, Go plugins...).
https://github.com/slok/terraform-provider-dataprocessor
data-processing go-plugin golang infrastructure jq terraform terraform-provider yaegi yq
Last synced: 7 months ago
JSON representation
Terraform provider for easy and clean data processing (JQ, YQ, Go plugins...).
- Host: GitHub
- URL: https://github.com/slok/terraform-provider-dataprocessor
- Owner: slok
- License: apache-2.0
- Created: 2022-08-05T18:11:13.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-03-02T09:07:44.000Z (over 2 years ago)
- Last Synced: 2025-03-19T08:39:19.952Z (7 months ago)
- Topics: data-processing, go-plugin, golang, infrastructure, jq, terraform, terraform-provider, yaegi, yq
- Language: Go
- Homepage:
- Size: 172 KB
- Stars: 13
- Watchers: 2
- Forks: 0
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# terraform-provider-dataprocessor
[](https://github.com/slok/terraform-provider-dataprocessor/actions/workflows/ci.yml)
[](https://goreportcard.com/report/github.com/slok/terraform-provider-dataprocessor)
[](https://raw.githubusercontent.com/slok/terraform-provider-dataprocessor/master/LICENSE)
[](https://github.com/slok/terraform-provider-dataprocessor/releases/latest)
[](https://registry.terraform.io/providers/slok/dataprocessor/latest/docs)
Avoid ugly terraform logic and code to transform data. This Terraform provider helps you with the data processing in a clean and easy way by using tools like [JQ], [YQ] and Go plugins.
## Processors
### JQ
The famous and well known [JQ] processor for your JSON inputs.
### YQ
The famous and well known [YQ] processor for your YAML inputs.
### Go plugins v1
Check [examples](examples/plugins):
- [FS check](examples/plugins/check_fs/): Checks files exist on disk. Shows how you can access the FS outside the plugin.
- [Complex validation](examples/plugins/complex_validation): Validate Prometheus Rules. Shows how to create advanced logic plugins.
- [Data structure transformation](examples/plugins/data_structure_transformation/): Transforms a data structure into another. Shows how to transform data for easier consumption by different terraform providers.
- [Filtering](examples/plugins/filtering/): Filters a list of usernames based on a regex. Shows how to filter terraform data to avoid HCL complex logic.
- [Remote plugin](examples/plugins/remote_plugin/): Uses a plugin that is hosted in github. Shows how plugins can be shared and create plugin repos.
- [Simple validation](examples/plugins/simple_validation/): Validates the length of a string. Shows that simple validation plugins can be powerful (like small functions), perfect to be used as a remote plugin.
The processor for everything :tada:, is the most powerful of all. You can use _almost_ (e.g `unsafe` package is banned) all the Go standard library. These are the requirements to create a plugin:
- Written in Go.
- No external dependencies, only Go standard library.
- Implemented in a single file (or string block).
- Implement the plugin API (Check the examples to know how to do it).
- The Filter function should be called:`ProcessorPluginV1`.
- The Filter function should have this signature: `ProcessorPluginV1(ctx context.Context, inputData string, vars map[string]string) (result string, error error)`.
This is the simplest plugin that you could create, a noop:
```go
package tfplugin
import "context"
func ProcessorPluginV1(ctx context.Context, inputData string, vars map[string]string) (string, error) {
return inputData, nil
}
```
However you can do complex things like loading JSON, HTTP requests, using timers, complex regex validations, templating...
Go plugins are implemented with [Yaegi], so they are portable and can run anywhere terraform can run.
## Use cases
- Generate, filter, mutate... JSON data.
- Retrieve JSON data from APIs, process and use it on other Terraform providers resources.
- Remove ugly HCL code in favor of a more clean and powerful data processing approach.
- Data validation (including complex cases).
## Requirements
- Terraform `>=1.x`.
## Terraform cloud
The provider its compatible with Terraform cloud workers, its focused on portability and thats why it doesn't require any binary CLI.
## Development
To install your plugin locally you can do `make install`, it will build and install in your `${HOME}/.terraform/plugins/...`
Note: The installation is ready for `OS_ARCH=linux_amd64`, so you make need to change the [`Makefile`](./Makefile) if using other OS.
Example:
```bash
cd ./examples/local
rm -rf ./.terraform ./.terraform.lock.hcl
cd -
make install
cd -
terraform plan
```
[JQ]: https://stedolan.github.io/jq/
[YQ]: https://github.com/mikefarah/yq
[Yaegi]: https://github.com/traefik/yaegi