https://github.com/agile-lab-dev/witboost-terraform-scaffold
https://github.com/agile-lab-dev/witboost-terraform-scaffold
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/agile-lab-dev/witboost-terraform-scaffold
- Owner: agile-lab-dev
- License: apache-2.0
- Created: 2023-08-03T11:07:05.000Z (almost 3 years ago)
- Default Branch: master
- Last Pushed: 2024-04-03T09:38:47.000Z (about 2 years ago)
- Last Synced: 2024-04-03T10:39:15.078Z (about 2 years ago)
- Language: Scala
- Size: 2 MB
- Stars: 0
- Watchers: 6
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Designed by [Agile Lab](https://www.agilelab.it/), Witboost is a versatile platform that addresses a wide range of sophisticated data engineering challenges. It enables businesses to discover, enhance, and productize their data, fostering the creation of automated data platforms that adhere to the highest standards of data governance. Want to know more about Witboost? Check it out [here](https://www.witboost.com/) or [contact us!](https://witboost.com/contact-us)
This repository is part of our [Starter Kit](https://github.com/agile-lab-dev/witboost-starter-kit) meant to showcase Witboost's integration capabilities and provide a "batteries-included" product.
# Terraform Specific Provisioner
- [Overview](#overview)
- [Building](#building)
- [Running](#running)
- [Configuring](#configuring)
- [Deploying](#deploying)
- [HLD](docs/HLD.md)
- [API specification](docs/API.md)
## Overview
This project implements a simple Specific Provisioner that uses [Terraform](https://www.terraform.io/) to provision infrastructure as code. After deploying this microservice and configuring Witboost to use it, the platform can deploy components using the Terraform capabilities.
### What's a Specific Provisioner?
A Specific Provisioner is a microservice which is in charge of deploying components that use a specific technology. When the deployment of a Data Product is triggered, the platform generates it descriptor and orchestrates the deployment of every component contained in the Data Product. For every such component the platform knows which Specific Provisioner is responsible for its deployment, and can thus send a provisioning request with the descriptor to it so that the Specific Provisioner can perform whatever operation is required to fulfill this request and report back the outcome to the platform.
You can learn more about how the Specific Provisioners fit in the broader picture [here](https://docs.witboost.agilelab.it/docs/p2_arch/p1_intro/#deploy-flow).
### Terraform
Terraform is an open-source infrastructure as code (IaC) tool developed by HashiCorp. It allows users to define and manage their cloud infrastructure in a declarative manner. With Terraform, you can define the desired state of your infrastructure using configuration files, typically written in HashiCorp Configuration Language (HCL) or JSON. These configuration files describe the various cloud resources and their configurations, such as virtual machines, storage, networks, and other services.
When you apply the Terraform configuration, it compares the current state of your infrastructure with the desired state defined in the configuration files. Terraform then automatically creates, updates, or deletes resources to align the infrastructure with the specified state. This enables consistent and reproducible deployments, making it easier to manage complex cloud environments and ensuring that the infrastructure stays in the desired state over time.
Terraform supports multiple cloud providers, including AWS, Microsoft Azure, Google Cloud Platform, and others, as well as various on-premises infrastructure providers. It has gained popularity among developers, sysadmins, and DevOps engineers for its simplicity, flexibility, and ability to automate the provisioning and management of cloud resources. You can find more information on it [here](https://developer.hashicorp.com/terraform).
### Software stack
This microservice is written in Scala 2.13, using HTTP4S for the HTTP layer. Project is built with SBT and supports packaging as JAR, fat-JAR and Docker image, ideal for Kubernetes deployments (which is the preferred option).
## Building
**Requirements:**
- Java 11
- SBT
- Terraform v1.5.7 or later (for running tests and for local usage)
- Docker (for building images only)
**Generating sources:** this project uses OpenAPI as standard API specification and the [sbt-guardrail](https://github.com/guardrail-dev/sbt-guardrail) plugin to generate server code from the [specification](./framework/src/main/openapi/interface-specification.yml).
To work with the codebase in a local environment, it is necessary to export the following environment variable:
```bash
export CI_JOB_TOKEN=""
```
The code generation is done automatically in the compile phase:
```bash
sbt compile
```
**Tests:** are handled by the standard task as well:
```bash
sbt test
```
**Artifacts & Docker image:** the project uses SBT Native Packager for packaging. Build artifacts with:
```
sbt package
```
The Docker image can be built with the standard docker CLI:
```
docker build -t .
```
*Note:* the version for the project is automatically computed using information gathered from Git, using branch name and tags. Unless you are on a release branch `1.2.x` or a tag `v1.2.3` it will end up being `0.0.0`. You can follow this branch/tag convention or update the version computation to match your preferred strategy.
**CI/CD:** the pipeline is based on GitLab CI as that's what we use internally. It's configured by the `.gitlab-ci.yaml` file in the root of the repository. You can use that as a starting point for your customizations.
## Running locally
To run the server locally, use:
```bash
sbt compile run
```
By default, the server binds to port 8080 on localhost. After it's up and running you can make provisioning requests to this address.
## Running on k8s
This repo provides a helm chart used to deploy the Terraform provisioner on a Kubernetes cluster.
More details about how to configure and run it are in the helm [README.md](helm/README.md).
## Configuring
Most application configurations are handled with the Typesafe Config library. You can find the default settings in the `reference.conf`. Customize them and use the `config.file` system property or the other options provided by Typesafe Config according to your needs. The provided docker image expects the config file mounted at path `/config/application.conf`. See [Configuration](docs/Configuration.md) for more information about this file.
## Deploying
This microservice is meant to be deployed to a Kubernetes cluster with the included Helm chart and the scripts that can be found in the `helm` subdirectory. You can find more details [here](helm/README.md).
## License
This project is available under the [Apache License, Version 2.0](https://opensource.org/licenses/Apache-2.0); see [LICENSE](LICENSE) for full details.
## About Witboost
[Witboost](https://witboost.com/) is a cutting-edge Data Experience platform, that streamlines complex data projects across various platforms, enabling seamless data production and consumption. This unified approach empowers you to fully utilize your data without platform-specific hurdles, fostering smoother collaboration across teams.
It seamlessly blends business-relevant information, data governance processes, and IT delivery, ensuring technically sound data projects aligned with strategic objectives. Witboost facilitates data-driven decision-making while maintaining data security, ethics, and regulatory compliance.
Moreover, Witboost maximizes data potential through automation, freeing resources for strategic initiatives. Apply your data for growth, innovation and competitive advantage.
[Contact us](https://witboost.com/contact-us) or follow us on:
- [LinkedIn](https://www.linkedin.com/showcase/witboost/)
- [YouTube](https://www.youtube.com/@witboost-platform)