{"id":20532239,"url":"https://github.com/unitvectory-labs/firepubauditsource","last_synced_at":"2025-09-21T18:25:05.008Z","repository":{"id":259638332,"uuid":"879098628","full_name":"UnitVectorY-Labs/firepubauditsource","owner":"UnitVectorY-Labs","description":"Publishes Firestore data changes to Pub/Sub as JSON audit records for downstream processing.","archived":false,"fork":false,"pushed_at":"2025-09-13T12:14:11.000Z","size":177,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-13T14:41:52.149Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/UnitVectorY-Labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-10-27T01:02:43.000Z","updated_at":"2025-09-13T12:14:14.000Z","dependencies_parsed_at":"2024-11-30T15:19:54.422Z","dependency_job_id":"f3efad0a-2473-4231-8a3a-b06437d74289","html_url":"https://github.com/UnitVectorY-Labs/firepubauditsource","commit_stats":{"total_commits":34,"total_committers":2,"mean_commits":17.0,"dds":"0.23529411764705888","last_synced_commit":"e1545837f1243629c9f97c7d276e4f135f1b8d56"},"previous_names":["unitvectory-labs/firepubauditsource"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/UnitVectorY-Labs/firepubauditsource","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UnitVectorY-Labs%2Ffirepubauditsource","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UnitVectorY-Labs%2Ffirepubauditsource/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UnitVectorY-Labs%2Ffirepubauditsource/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UnitVectorY-Labs%2Ffirepubauditsource/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/UnitVectorY-Labs","download_url":"https://codeload.github.com/UnitVectorY-Labs/firepubauditsource/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UnitVectorY-Labs%2Ffirepubauditsource/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276283525,"owners_count":25616058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-21T02:00:07.055Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T00:13:47.815Z","updated_at":"2025-09-21T18:25:04.956Z","avatar_url":"https://github.com/UnitVectorY-Labs.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![Work In Progress](https://img.shields.io/badge/Status-Work%20In%20Progress-yellow)](https://guide.unitvectorylabs.com/bestpractices/status/#work-in-progress)\n\n# firepubauditsource\n\nPublishes Firestore data changes to Pub/Sub as JSON audit records for downstream processing.\n\n## References\n\n- [firepubauditsource](https://github.com/UnitVectorY-Labs/firepubauditsource) - Publishes Firestore data changes to Pub/Sub as JSON audit records for downstream processing.\n- [firepubauditsource-tofu](https://github.com/UnitVectorY-Labs/firepubauditsource-tofu) - A module for OpenTofu that deploys firepubauditsource to GCP Cloud Run, along with configuring essential services including Eventarc for Firestore and Pub/Sub.\n- [bqpubauditsink](https://github.com/UnitVectorY-Labs/bqpubauditsink) - Ingests Pub/Sub audit JSON events and inserts the records into BigQuery.\n- [bqpubauditsink-tofu](https://github.com/UnitVectorY-Labs/bqpubauditsink-tofu) - A module for OpenTofu that deploys bqpubauditsink to GCP Cloud Run, along with configuring essential services including the Pub/Sub subscription and BigQuery dataset and table.\n\n## Overview\n\nThe purpose of this application is to take all record changes from a Firestore table and publish them to a Pub/Sub topic. This allows for downstream applications to process the complete database record changes including the old and new values.\n\nThis application is designed to run in Cloud Run and is triggered by changes to Firestore records using Eventarc.\n\n### Use Case: BigQuery\n\n- A companion application [bqpubauditsink](https://github.com/UnitVectorY-Labs/bqpubauditsink) takes the Pub/Sub messages and writes them to BigQuery in a way that allows BigQuery to be a direct replica of the data stored in Firestore.\n\n## Configuration\n\nThis application is run as a docker container and requires the following environment variables to be set:\n\n- `PROJECT_ID`: The GCP project ID where the Firestore and Pub/Sub resources are located.\n- `PUBSUB_TOPIC`: The Pub/Sub topic to publish the audit records to.\n\n## Example Pub/Sub Message\n\nThe following show what the JSON message will look like when published to Pub/Sub. The `oldValue` field will be `null` for inserts, and `value` will be `null` for deletes.\n\nInserting a Record:\n\n```json\n{\n  \"timestamp\": \"2024-10-27 12:00:00.000000\",\n  \"database\": \"(default)\",\n  \"documentPath\": \"mycollection/mydoc\",\n  \"value\": {\n    \"foo\": \"new\"\n  },\n  \"oldValue\": null\n}\n```\n\nUpdating a Record:\n\n```json\n{\n  \"timestamp\": \"2024-10-27 12:00:10.000000\",\n  \"database\": \"(default)\",\n  \"documentPath\": \"mycollection/mydoc\",\n  \"value\": {\n    \"foo\": \"updated\"\n  },\n  \"oldValue\": {\n    \"foo\": \"bar\"\n  }\n}\n```\n\nDeleting a Record:\n\n```json\n{\n  \"timestamp\": \"2024-10-27 12:00:20.000000\",\n  \"database\": \"(default)\",\n  \"documentPath\": \"mycollection/mydoc\",\n  \"value\": null,\n  \"oldValue\": {\n    \"foo\": \"bar\"\n  }\n}\n```\n\n## Pub/Sub Attributes\n\nIn order to assist with subscriptions that may want to only process a subset of messages, the following attributes are added to the Pub/Sub message:\n\n- `database`: The Firestore database ID.\n- `action`: The Firestore action that triggered the message. This will be one of `create`, `update`, or `delete`.\n\nThe document `path` is also included as an attribute split into parts on the `/` character. This allows for subscriptions to filter on specific collections or documents. The attribute names are `path0`, `path1`, etc. so that the first part of the path is `path0` represents the collection and `path1` represents the document, but additional parts are also included for nested documents.\n\n## Limitations\n\n- The translation from the Firestore document to JSON is not perfect, it uses [firestoreproto2json](https://github.com/UnitVectorY-Labs/firestoreproto2json) with the default settings to convert the Firestore document to JSON. This means that some data types may not be converted correctly.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funitvectory-labs%2Ffirepubauditsource","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funitvectory-labs%2Ffirepubauditsource","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funitvectory-labs%2Ffirepubauditsource/lists"}