{"id":48299057,"url":"https://github.com/observeinc/terraform-google-collection","last_synced_at":"2026-04-04T23:42:47.224Z","repository":{"id":76081874,"uuid":"475186987","full_name":"observeinc/terraform-google-collection","owner":"observeinc","description":"Terraform module which streamlines collection from multiple sources within GCP","archived":false,"fork":false,"pushed_at":"2025-09-16T20:52:19.000Z","size":1885,"stargazers_count":2,"open_issues_count":7,"forks_count":2,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-09-16T21:31:05.542Z","etag":null,"topics":["gcp","google-cloud","integrationname-gcp","integrations","observe","terraform-module"],"latest_commit_sha":null,"homepage":"https://docs.observeinc.com/en/latest/content/integrations/gcp/gcp.html#installation","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/observeinc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-03-28T21:38:04.000Z","updated_at":"2025-08-11T22:55:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"fc8bd5ee-896d-4837-95a2-378b77c48b4a","html_url":"https://github.com/observeinc/terraform-google-collection","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/observeinc/terraform-google-collection","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/observeinc%2Fterraform-google-collection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/observeinc%2Fterraform-google-collection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/observeinc%2Fterraform-google-collection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/observeinc%2Fterraform-google-collection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/observeinc","download_url":"https://codeload.github.com/observeinc/terraform-google-collection/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/observeinc%2Fterraform-google-collection/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31419542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T20:09:54.854Z","status":"ssl_error","status_checked_at":"2026-04-04T20:09:44.350Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gcp","google-cloud","integrationname-gcp","integrations","observe","terraform-module"],"created_at":"2026-04-04T23:42:46.694Z","updated_at":"2026-04-04T23:42:47.216Z","avatar_url":"https://github.com/observeinc.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Observe Google Collection\n\nThis module creates a log sink, pub/sub topic, and pub/sub subscription needed to\nfacilitate the collection of asset inventory records, metrics and logs from GCP for a\ngiven project.\n\nThis module also creates a Cloud Function to fetch some data through the GCP REST API.\n\n## Usage\n\nHere is an example manifest for collecting data from a Google Cloud organization.\n\nAfter running `terraform apply`, data should start flowing into Pub/Sub. In the Observe\nUI, one would set up the GCP app. The info from the `terraform output` and `terraform output -raw service_account_private_key` are needed to set up the GCP App pollers.\n\n```hcl\nprovider \"google\" {\n  project = \"YOUR_PROJECT_ID\"\n  region  = \"YOUR_DEFAULT_REGION\"\n}\n\nmodule \"observe_gcp_collection\" {\n  source  = \"observeinc/collection/google\"\n  name    = \"observe\"\n\n  resource = \"projects/YOUR_PROJECT_ID\"\n}\n\noutput \"project\" {\n  description = \"The Pub/Sub project of the subcription (to be passed to the Pub/Sub poller)\"\n  value       = module.observe_gcp_collection.project\n}\n\n# To extract correct value - terraform output -json | jq -r '.subscription.value.name' \noutput \"subscription\" {\n  description = \"The Pub/Sub subscription created by this module (to be passed to the Pub/Sub poller)\"\n  value       = module.observe_gcp_collection.subscription\n}\n\n# To extract properly formatted string - terraform output -json | jq -r '.service_account_private_key.value'\noutput \"service_account_private_key\" {\n  description = \"A service account key to be passed to the pollers for Pub/Sub and Cloud Monitoring\"\n  value       = base64decode(module.observe_gcp_collection.service_account_key.private_key)\n  sensitive   = true\n}\n\n```\n\u003c!-- BEGIN_TF_DOCS --\u003e\n## Requirements\n\n| Name | Version |\n|------|---------|\n| \u003ca name=\"requirement_terraform\"\u003e\u003c/a\u003e [terraform](#requirement\\_terraform) | \u003e= 0.12.21 |\n| \u003ca name=\"requirement_google\"\u003e\u003c/a\u003e [google](#requirement\\_google) | \u003e= 4.15 |\n| \u003ca name=\"requirement_random\"\u003e\u003c/a\u003e [random](#requirement\\_random) | ~\u003e 3.0 |\n\n## Providers\n\n| Name | Version |\n|------|---------|\n| \u003ca name=\"provider_google\"\u003e\u003c/a\u003e [google](#provider\\_google) | \u003e= 4.15 |\n| \u003ca name=\"provider_random\"\u003e\u003c/a\u003e [random](#provider\\_random) | ~\u003e 3.0 |\n\n## Modules\n\nNo modules.\n\n## Resources\n\n| Name | Type |\n|------|------|\n| [google_cloud_asset_folder_feed.folder_feed](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_asset_folder_feed) | resource |\n| [google_cloud_asset_project_feed.project_feed](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_asset_project_feed) | resource |\n| [google_cloud_scheduler_job.rest_of_assets](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_scheduler_job) | resource |\n| [google_cloud_scheduler_job.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_scheduler_job) | resource |\n| [google_cloud_tasks_queue.task_queue](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_tasks_queue) | resource |\n| [google_cloudfunctions_function.gcs_function](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloudfunctions_function) | resource |\n| [google_cloudfunctions_function.rest_of_assets](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloudfunctions_function) | resource |\n| [google_cloudfunctions_function.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloudfunctions_function) | resource |\n| [google_cloudfunctions_function_iam_member.cloud_scheduler](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloudfunctions_function_iam_member) | resource |\n| [google_cloudfunctions_function_iam_member.cloud_scheduler_rest_of_assets](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloudfunctions_function_iam_member) | resource |\n| [google_folder_iam_member.cloudfunction](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/folder_iam_member) | resource |\n| [google_logging_folder_sink.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_folder_sink) | resource |\n| [google_logging_organization_sink.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_organization_sink) | resource |\n| [google_logging_project_sink.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_project_sink) | resource |\n| [google_organization_iam_member.cloudfunction](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/organization_iam_member) | resource |\n| [google_project_iam_member.cloudfunction](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_iam_member) | resource |\n| [google_project_iam_member.poller](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_iam_member) | resource |\n| [google_pubsub_subscription.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_subscription) | resource |\n| [google_pubsub_subscription_iam_member.poller_pubsub](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_subscription_iam_member) | resource |\n| [google_pubsub_topic.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic) | resource |\n| [google_pubsub_topic_iam_member.cloudfunction_pubsub](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic_iam_member) | resource |\n| [google_pubsub_topic_iam_member.sink_pubsub](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic_iam_member) | resource |\n| [google_service_account.cloud_scheduler](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |\n| [google_service_account.cloudfunction](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |\n| [google_service_account.poller](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |\n| [google_service_account_key.poller](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account_key) | resource |\n| [google_storage_bucket.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket) | resource |\n| [google_storage_bucket_iam_member.bucket_iam](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_member) | resource |\n| [google_storage_bucket_iam_member.gcs_function_bucket_iam](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_member) | resource |\n| [random_id.cloudtasks_queue](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/id) | resource |\n| [google_folder.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/folder) | data source |\n| [google_project.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/project) | data source |\n\n## Inputs\n\n| Name | Description | Type | Default | Required |\n|------|-------------|------|---------|:--------:|\n| \u003ca name=\"input_bucket_lifecycle_abort_upload_days\"\u003e\u003c/a\u003e [bucket\\_lifecycle\\_abort\\_upload\\_days](#input\\_bucket\\_lifecycle\\_abort\\_upload\\_days) | The number of days to wait before deleting AbortIncompleteMultipartUpload. | `number` | `7` | no |\n| \u003ca name=\"input_bucket_lifecycle_delete_days\"\u003e\u003c/a\u003e [bucket\\_lifecycle\\_delete\\_days](#input\\_bucket\\_lifecycle\\_delete\\_days) | The number of days to wait before Delete of temporary bucket files. | `number` | `14` | no |\n| \u003ca name=\"input_cloud_function_debug_level\"\u003e\u003c/a\u003e [cloud\\_function\\_debug\\_level](#input\\_cloud\\_function\\_debug\\_level) | The debug level for the GCP cloud functions | `string` | `\"WARNING\"` | no |\n| \u003ca name=\"input_enable_asset_tracking\"\u003e\u003c/a\u003e [enable\\_asset\\_tracking](#input\\_enable\\_asset\\_tracking) | Whether to enable the Cloud function that tracks GCP assets. | `bool` | `true` | no |\n| \u003ca name=\"input_enable_function\"\u003e\u003c/a\u003e [enable\\_function](#input\\_enable\\_function) | DEPRECATED: This variable has been renamed to 'enable\\_asset\\_tracking'. Please update your configuration to use 'enable\\_asset\\_tracking' instead. | `bool` | `null` | no |\n| \u003ca name=\"input_folder_include_children\"\u003e\u003c/a\u003e [folder\\_include\\_children](#input\\_folder\\_include\\_children) | Whether to include all children Projects of a Folder when collecting logs | `bool` | `true` | no |\n| \u003ca name=\"input_function_available_memory_mb\"\u003e\u003c/a\u003e [function\\_available\\_memory\\_mb](#input\\_function\\_available\\_memory\\_mb) | Memory (in MB), available to the function. Default value is 512. Possible values include 128, 256, 512, 1024, etc. | `number` | `4096` | no |\n| \u003ca name=\"input_function_bucket\"\u003e\u003c/a\u003e [function\\_bucket](#input\\_function\\_bucket) | GCS bucket containing the Cloud Function source code | `string` | `\"observeinc\"` | no |\n| \u003ca name=\"input_function_disable_logging\"\u003e\u003c/a\u003e [function\\_disable\\_logging](#input\\_function\\_disable\\_logging) | Whether to disable function logging. | `bool` | `false` | no |\n| \u003ca name=\"input_function_max_instances\"\u003e\u003c/a\u003e [function\\_max\\_instances](#input\\_function\\_max\\_instances) | The limit on the maximum number of function instances that may coexist at a given time. | `number` | `100` | no |\n| \u003ca name=\"input_function_object\"\u003e\u003c/a\u003e [function\\_object](#input\\_function\\_object) | GCS object key of the Cloud Function source code zip file. Will use the latest release unless modified. | `string` | `\"google-cloud-functions-latest.zip\"` | no |\n| \u003ca name=\"input_function_roles\"\u003e\u003c/a\u003e [function\\_roles](#input\\_function\\_roles) | A list of IAM roles to give the Cloud Function. | `set(string)` | \u003cpre\u003e[\u003cbr/\u003e  \"roles/compute.viewer\",\u003cbr/\u003e  \"roles/iam.serviceAccountViewer\",\u003cbr/\u003e  \"roles/cloudscheduler.viewer\",\u003cbr/\u003e  \"roles/cloudasset.viewer\",\u003cbr/\u003e  \"roles/browser\",\u003cbr/\u003e  \"roles/logging.viewer\",\u003cbr/\u003e  \"roles/monitoring.viewer\",\u003cbr/\u003e  \"roles/storage.objectCreator\",\u003cbr/\u003e  \"roles/storage.objectViewer\",\u003cbr/\u003e  \"roles/storage.objectAdmin\",\u003cbr/\u003e  \"roles/storage.admin\",\u003cbr/\u003e  \"roles/cloudfunctions.invoker\",\u003cbr/\u003e  \"roles/cloudtasks.enqueuer\",\u003cbr/\u003e  \"roles/cloudtasks.viewer\",\u003cbr/\u003e  \"roles/cloudtasks.taskDeleter\",\u003cbr/\u003e  \"roles/iam.serviceAccountUser\"\u003cbr/\u003e]\u003c/pre\u003e | no |\n| \u003ca name=\"input_function_schedule_frequency\"\u003e\u003c/a\u003e [function\\_schedule\\_frequency](#input\\_function\\_schedule\\_frequency) | Cron schedule for the job | `string` | `\"0 * * * *\"` | no |\n| \u003ca name=\"input_function_schedule_frequency_rest_of_assets\"\u003e\u003c/a\u003e [function\\_schedule\\_frequency\\_rest\\_of\\_assets](#input\\_function\\_schedule\\_frequency\\_rest\\_of\\_assets) | Cron schedule for the job | `string` | `\"*/5 * * * *\"` | no |\n| \u003ca name=\"input_function_timeout\"\u003e\u003c/a\u003e [function\\_timeout](#input\\_function\\_timeout) | Timeout (in seconds) for the function. Default value is 300 seconds. Cannot be more than 540 seconds. | `number` | `300` | no |\n| \u003ca name=\"input_gcp_region\"\u003e\u003c/a\u003e [gcp\\_region](#input\\_gcp\\_region) | The location where the Task Queue will be created. | `string` | `\"us-central1\"` | no |\n| \u003ca name=\"input_labels\"\u003e\u003c/a\u003e [labels](#input\\_labels) | A map of labels to add to resources (https://cloud.google.com/resource-manager/docs/creating-managing-labels)\"\u003cbr/\u003e\u003cbr/\u003eNote: Many, but not all, Google Cloud SDK resources support labels. | `map(string)` | `{}` | no |\n| \u003ca name=\"input_logging_exclusions\"\u003e\u003c/a\u003e [logging\\_exclusions](#input\\_logging\\_exclusions) | Log entries that match any of these exclusion filters will not be exported.\u003cbr/\u003e\u003cbr/\u003eIf a log entry is matched by both logging\\_filter and one of logging\\_exclusions it will not be exported.\u003cbr/\u003e\u003cbr/\u003eRelevant docs: https://cloud.google.com/logging/docs/reference/v2/rest/v2/billingAccounts.exclusions#LogExclusion | \u003cpre\u003elist(object({\u003cbr/\u003e    name        = string\u003cbr/\u003e    description = string\u003cbr/\u003e    filter      = string\u003cbr/\u003e    disabled    = string\u003cbr/\u003e  }))\u003c/pre\u003e | `[]` | no |\n| \u003ca name=\"input_logging_filter\"\u003e\u003c/a\u003e [logging\\_filter](#input\\_logging\\_filter) | An advanced logs filter. The only exported log entries are those that are\u003cbr/\u003ein the resource owning the sink and that match the filter.\u003cbr/\u003e\u003cbr/\u003eRelevant docs: https://cloud.google.com/logging/docs/view/building-queries | `string` | `\"\"` | no |\n| \u003ca name=\"input_max_attempts\"\u003e\u003c/a\u003e [max\\_attempts](#input\\_max\\_attempts) | The maximum number of retry attempts for a task in case of failure. | `number` | `-1` | no |\n| \u003ca name=\"input_max_concurrent_dispatches\"\u003e\u003c/a\u003e [max\\_concurrent\\_dispatches](#input\\_max\\_concurrent\\_dispatches) | The maximum number of tasks that can be dispatched concurrently. | `number` | `2` | no |\n| \u003ca name=\"input_max_dispatches_per_second\"\u003e\u003c/a\u003e [max\\_dispatches\\_per\\_second](#input\\_max\\_dispatches\\_per\\_second) | The maximum rate at which tasks can be dispatched per second. | `number` | `2` | no |\n| \u003ca name=\"input_max_retry_duration\"\u003e\u003c/a\u003e [max\\_retry\\_duration](#input\\_max\\_retry\\_duration) | The time limit for retrying a task in seconds | `string` | `\"7200s\"` | no |\n| \u003ca name=\"input_min_backoff\"\u003e\u003c/a\u003e [min\\_backoff](#input\\_min\\_backoff) | The minimum amount of time to wait between retries in seconds | `string` | `\"30s\"` | no |\n| \u003ca name=\"input_name\"\u003e\u003c/a\u003e [name](#input\\_name) | Module name. Used as a name prefix. | `string` | `\"obs\"` | no |\n| \u003ca name=\"input_poller_roles\"\u003e\u003c/a\u003e [poller\\_roles](#input\\_poller\\_roles) | A list of IAM roles to give the Observe poller (through the service account key output). | `set(string)` | \u003cpre\u003e[\u003cbr/\u003e  \"roles/monitoring.viewer\"\u003cbr/\u003e]\u003c/pre\u003e | no |\n| \u003ca name=\"input_project_id\"\u003e\u003c/a\u003e [project\\_id](#input\\_project\\_id) | Billing Project ID needed for asset feed. | `string` | `null` | no |\n| \u003ca name=\"input_pubsub_ack_deadline_seconds\"\u003e\u003c/a\u003e [pubsub\\_ack\\_deadline\\_seconds](#input\\_pubsub\\_ack\\_deadline\\_seconds) | Ack deadline for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) | `number` | `60` | no |\n| \u003ca name=\"input_pubsub_maximum_backoff\"\u003e\u003c/a\u003e [pubsub\\_maximum\\_backoff](#input\\_pubsub\\_maximum\\_backoff) | Retry policy maximum backoff for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) | `string` | `\"600s\"` | no |\n| \u003ca name=\"input_pubsub_message_retention_duration\"\u003e\u003c/a\u003e [pubsub\\_message\\_retention\\_duration](#input\\_pubsub\\_message\\_retention\\_duration) | Message retention for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) | `string` | `\"86400s\"` | no |\n| \u003ca name=\"input_pubsub_minimum_backoff\"\u003e\u003c/a\u003e [pubsub\\_minimum\\_backoff](#input\\_pubsub\\_minimum\\_backoff) | Retry policy minimum backoff for the Pub/Sub subscription (https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions) | `string` | `\"10s\"` | no |\n| \u003ca name=\"input_resource\"\u003e\u003c/a\u003e [resource](#input\\_resource) | The identifier of the GCP Resource to monitor.\u003cbr/\u003e\u003cbr/\u003eThe resource can be a project, folder, or organization.\u003cbr/\u003e\u003cbr/\u003eExamples: \"projects/my\\_project-123\", \"folders/1234567899\", \"organizations/34739118321\" | `string` | n/a | yes |\n\n## Outputs\n\n| Name | Description |\n|------|-------------|\n| \u003ca name=\"output_project\"\u003e\u003c/a\u003e [project](#output\\_project) | The ID of the Project in which resources were created |\n| \u003ca name=\"output_service_account_key\"\u003e\u003c/a\u003e [service\\_account\\_key](#output\\_service\\_account\\_key) | A service account key to be passed to the pollers for Pub/Sub and Cloud Monitoring |\n| \u003ca name=\"output_subscription\"\u003e\u003c/a\u003e [subscription](#output\\_subscription) | The Pub/Sub subscription created by this module. |\n| \u003ca name=\"output_topic\"\u003e\u003c/a\u003e [topic](#output\\_topic) | The Pub/Sub topic created by this module. |\n\u003c!-- END_TF_DOCS --\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobserveinc%2Fterraform-google-collection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fobserveinc%2Fterraform-google-collection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobserveinc%2Fterraform-google-collection/lists"}