Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/observeinc/terraform-google-collection

Terraform module which streamlines collection from multiple sources within GCP
https://github.com/observeinc/terraform-google-collection

gcp google-cloud integrationname-gcp integrations observe terraform-module

Last synced: 3 months ago
JSON representation

Terraform module which streamlines collection from multiple sources within GCP

Awesome Lists containing this project

README 

          
# Observe Google Collection



This module creates a log sink, pub/sub topic, and pub/sub subscription needed to

facilitate the collection of asset inventory records, metrics and logs from GCP for a

given project.



This module also creates a Cloud Function to fetch some data through the GCP REST API.



## Usage



Here is an example manifest for collecting data from a Google Cloud organization.



After running `terraform apply`, data should start flowing into Pub/Sub. In the Observe

UI, one would set up the GCP app. The info from the `terraform output` and `terraform output -raw service_account_private_key` are needed to set up the GCP App pollers.



```hcl

provider "google" {

  project = "YOUR_PROJECT_ID"

  region  = "YOUR_DEFAULT_REGION"

}



module "observe_gcp_collection" {

  source  = "observeinc/collection/google"

  name    = "observe"



  resource = "projects/YOUR_PROJECT_ID"

}



output "project" {

  description = "The Pub/Sub project of the subcription (to be passed to the Pub/Sub poller)"

  value       = module.observe_gcp_collection.project

}



# To extract correct value - terraform output -json | jq -r '.subscription.value.name' 

output "subscription" {

  description = "The Pub/Sub subscription created by this module (to be passed to the Pub/Sub poller)"

  value       = module.observe_gcp_collection.subscription

}



# To extract properly formatted string - terraform output -json | jq -r '.service_account_private_key.value'

output "service_account_private_key" {

  description = "A service account key to be passed to the pollers for Pub/Sub and Cloud Monitoring"

  value       = base64decode(module.observe_gcp_collection.service_account_key.private_key)

  sensitive   = true

}



```



## Requirements



| Name | Version |

|------|---------|

|  [terraform](#requirement\_terraform) | >= 0.12.21 |

|  [google](#requirement\_google) | >= 4.15 |

|  [random](#requirement\_random) | ~> 3.0 |



## Providers



| Name | Version |

|------|---------|

|  [google](#provider\_google) | >= 4.15 |

|  [random](#provider\_random) | ~> 3.0 |



## Modules



No modules.



## Resources



| Name | Type |

|------|------|

| [google_cloud_asset_folder_feed.folder_feed](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_asset_folder_feed) | resource |

| [google_cloud_asset_project_feed.project_feed](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_asset_project_feed) | resource |

| [google_cloud_scheduler_job.rest_of_assets](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_scheduler_job) | resource |

| [google_cloud_scheduler_job.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_scheduler_job) | resource |

| [google_cloud_tasks_queue.task_queue](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_tasks_queue) | resource |

| [google_cloudfunctions_function.gcs_function](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloudfunctions_function) | resource |

| [google_cloudfunctions_function.rest_of_assets](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloudfunctions_function) | resource |

| [google_cloudfunctions_function.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloudfunctions_function) | resource |

| [google_cloudfunctions_function_iam_member.cloud_scheduler](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloudfunctions_function_iam_member) | resource |

| [google_cloudfunctions_function_iam_member.cloud_scheduler_rest_of_assets](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloudfunctions_function_iam_member) | resource |

| [google_folder_iam_member.cloudfunction](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/folder_iam_member) | resource |

| [google_logging_folder_sink.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_folder_sink) | resource |

| [google_logging_organization_sink.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_organization_sink) | resource |

| [google_logging_project_sink.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_project_sink) | resource |

| [google_organization_iam_member.cloudfunction](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/organization_iam_member) | resource |

| [google_project_iam_member.cloudfunction](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_iam_member) | resource |

| [google_project_iam_member.poller](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_iam_member) | resource |

| [google_pubsub_subscription.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_subscription) | resource |

| [google_pubsub_subscription_iam_member.poller_pubsub](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_subscription_iam_member) | resource |

| [google_pubsub_topic.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic) | resource |

| [google_pubsub_topic_iam_member.cloudfunction_pubsub](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic_iam_member) | resource |

| [google_pubsub_topic_iam_member.sink_pubsub](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic_iam_member) | resource |

| [google_service_account.cloud_scheduler](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |

| [google_service_account.cloudfunction](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |

| [google_service_account.poller](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |

| [google_service_account_key.poller](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account_key) | resource |

| [google_storage_bucket.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket) | resource |

| [google_storage_bucket_iam_member.bucket_iam](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_member) | resource |

| [google_storage_bucket_iam_member.gcs_function_bucket_iam](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_member) | resource |

| [random_id.cloudtasks_queue](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/id) | resource |

| [google_folder.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/folder) | data source |

| [google_project.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/project) | data source |



## Inputs



| Name | Description | Type | Default | Required |

|------|-------------|------|---------|:--------:|

|  [bucket\_lifecycle\_abort\_upload\_days](#input\_bucket\_lifecycle\_abort\_upload\_days) | The number of days to wait before deleting AbortIncompleteMultipartUpload. | `number` | `7` | no |

|  [bucket\_lifecycle\_delete\_days](#input\_bucket\_lifecycle\_delete\_days) | The number of days to wait before Delete of temporary bucket files. | `number` | `14` | no |

|  [cloud\_function\_debug\_level](#input\_cloud\_function\_debug\_level) | The debug level for the GCP cloud functions | `string` | `"WARNING"` | no |

|  [enable\_asset\_tracking](#input\_enable\_asset\_tracking) | Whether to enable the Cloud function that tracks GCP assets. | `bool` | `true` | no |

|  [enable\_function](#input\_enable\_function) | DEPRECATED: This variable has been renamed to 'enable\_asset\_tracking'. Please update your configuration to use 'enable\_asset\_tracking' instead. | `bool` | `null` | no |

|  [folder\_include\_children](#input\_folder\_include\_children) | Whether to include all children Projects of a Folder when collecting logs | `bool` | `true` | no |

|  [function\_available\_memory\_mb](#input\_function\_available\_memory\_mb) | Memory (in MB), available to the function. Default value is 512. Possible values include 128, 256, 512, 1024, etc. | `number` | `4096` | no |

|  [function\_bucket](#input\_function\_bucket) | GCS bucket containing the Cloud Function source code | `string` | `"observeinc"` | no |

|  [function\_disable\_logging](#input\_function\_disable\_logging) | Whether to disable function logging. | `bool` | `false` | no |

|  [function\_max\_instances](#input\_function\_max\_instances) | The limit on the maximum number of function instances that may coexist at a given time. | `number` | `100` | no |

|  [function\_object](#input\_function\_object) | GCS object key of the Cloud Function source code zip file. Will use the latest release unless modified. | `string` | `"google-cloud-functions-latest.zip"` | no |

|  [function\_roles](#input\_function\_roles) | A list of IAM roles to give the Cloud Function. | `set(string)` | 
[
  "roles/compute.viewer",
  "roles/iam.serviceAccountViewer",
  "roles/cloudscheduler.viewer",
  "roles/cloudasset.viewer",
  "roles/browser",
  "roles/logging.viewer",
  "roles/monitoring.viewer",
  "roles/storage.objectCreator",
  "roles/storage.objectViewer",
  "roles/storage.objectAdmin",
  "roles/storage.admin",
  "roles/cloudfunctions.invoker",
  "roles/cloudtasks.enqueuer",
  "roles/cloudtasks.viewer",
  "roles/cloudtasks.taskDeleter",
  "roles/iam.serviceAccountUser"
]
 | no |

|  [function\_schedule\_frequency](#input\_function\_schedule\_frequency) | Cron schedule for the job | `string` | `"0 * * * *"` | no |

|  [function\_schedule\_frequency\_rest\_of\_assets](#input\_function\_schedule\_frequency\_rest\_of\_assets) | Cron schedule for the job | `string` | `"*/5 * * * *"` | no |

|  [function\_timeout](#input\_function\_timeout) | Timeout (in seconds) for the function. Default value is 300 seconds. Cannot be more than 540 seconds. | `number` | `300` | no |

|  [gcp\_region](#input\_gcp\_region) | The location where the Task Queue will be created. | `string` | `"us-central1"` | no |

|  [labels](#input\_labels) | A map of labels to add to resources (https://cloud.google.com/resource-manager/docs/creating-managing-labels)"

Note: Many, but not all, Google Cloud SDK resources support labels. | `map(string)` | `{}` | no |

|  [logging\_exclusions](#input\_logging\_exclusions) | Log entries that match any of these exclusion filters will not be exported.

If a log entry is matched by both logging\_filter and one of logging\_exclusions it will not be exported.

Relevant docs: https://cloud.google.com/logging/docs/reference/v2/rest/v2/billingAccounts.exclusions#LogExclusion | 
list(object({
    name        = string
    description = string
    filter      = string
    disabled    = string
  }))
 | `[]` | no |

|  [logging\_filter](#input\_logging\_filter) | An advanced logs filter. The only exported log entries are those that are
in the resource owning the sink and that match the filter.

Relevant docs: https://cloud.google.com/logging/docs/view/building-queries | `string` | `""` | no |

|  [max\_attempts](#input\_max\_attempts) | The maximum number of retry attempts for a task in case of failure. | `number` | `-1` | no |

|  [max\_concurrent\_dispatches](#input\_max\_concurrent\_dispatches) | The maximum number of tasks that can be dispatched concurrently. | `number` | `2` | no |

|  [max\_dispatches\_per\_second](#input\_max\_dispatches\_per\_second) | The maximum rate at which tasks can be dispatched per second. | `number` | `2` | no |

|  [max\_retry\_duration](#input\_max\_retry\_duration) | The time limit for retrying a task in seconds | `string` | `"7200s"` | no |

|  [min\_backoff](#input\_min\_backoff) | The minimum amount of time to wait between retries in seconds | `string` | `"30s"` | no |

|  [name](#input\_name) | Module name. Used as a name prefix. | `string` | `"obs"` | no |

|  [poller\_roles](#input\_poller\_roles) | A list of IAM roles to give the Observe poller (through the service account key output). | `set(string)` | 
[
  "roles/monitoring.viewer"
]
 | no |

|  [project\_id](#input\_project\_id) | Billing Project ID needed for asset feed. | `string` | `null` | no |

|  [pubsub\_ack\_deadline\_seconds](#input\_pubsub\_ack\_deadline\_seconds) | Ack deadline for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) | `number` | `60` | no |

|  [pubsub\_maximum\_backoff](#input\_pubsub\_maximum\_backoff) | Retry policy maximum backoff for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) | `string` | `"600s"` | no |

|  [pubsub\_message\_retention\_duration](#input\_pubsub\_message\_retention\_duration) | Message retention for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) | `string` | `"86400s"` | no |

|  [pubsub\_minimum\_backoff](#input\_pubsub\_minimum\_backoff) | Retry policy minimum backoff for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) | `string` | `"10s"` | no |

|  [resource](#input\_resource) | The identifier of the GCP Resource to monitor.

The resource can be a project, folder, or organization.

Examples: "projects/my\_project-123", "folders/1234567899", "organizations/34739118321" | `string` | n/a | yes |



## Outputs



| Name | Description |

|------|-------------|

|  [project](#output\_project) | The ID of the Project in which resources were created |

|  [service\_account\_key](#output\_service\_account\_key) | A service account key to be passed to the pollers for Pub/Sub and Cloud Monitoring |

|  [subscription](#output\_subscription) | The Pub/Sub subscription created by this module. |

|  [topic](#output\_topic) | The Pub/Sub topic created by this module. |