{"id":43067027,"url":"https://github.com/cloudchacho/hedwig-firehose-dataflow","last_synced_at":"2026-01-31T12:51:55.592Z","repository":{"id":57732808,"uuid":"381860637","full_name":"cloudchacho/hedwig-firehose-dataflow","owner":"cloudchacho","description":"A firehose implementation using Dataflow for Hedwig","archived":false,"fork":false,"pushed_at":"2022-01-20T00:14:14.000Z","size":47,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2023-07-28T10:08:46.456Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudchacho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-30T23:53:05.000Z","updated_at":"2021-12-16T17:13:25.000Z","dependencies_parsed_at":"2022-09-26T22:11:07.440Z","dependency_job_id":null,"html_url":"https://github.com/cloudchacho/hedwig-firehose-dataflow","commit_stats":null,"previous_names":[],"tags_count":4,"template":null,"template_full_name":null,"purl":"pkg:github/cloudchacho/hedwig-firehose-dataflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudchacho%2Fhedwig-firehose-dataflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudchacho%2Fhedwig-firehose-dataflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudchacho%2Fhedwig-firehose-dataflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudchacho%2Fhedwig-firehose-dataflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudchacho","download_url":"https://codeload.github.com/cloudchacho/hedwig-firehose-dataflow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudchacho%2Fhedwig-firehose-dataflow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28943639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-31T12:10:04.904Z","status":"ssl_error","status_checked_at":"2026-01-31T12:09:58.894Z","response_time":128,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-31T12:51:54.969Z","updated_at":"2026-01-31T12:51:55.581Z","avatar_url":"https://github.com/cloudchacho.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hedwig Firehose Dataflow\n\nThis is a dataflow template that can be used for Hedwig Firehose. It reads input messages from your subscriptions and\nwrites them into a GCS file. The messages are decoded using protobuf format on a best-effort basis. If a message can't\nbe decoded, its still written to the file as a base64 encoded string.\n\n**Assumptions:**\n- Protobuf validator using message transport attributes.\n\nOutput files are compressed using gzip by default.\n\nOutput structure in GCS:\n- Top level bucket / directory: `gs://$BUCKET/$DIR` (supplied as input parameter `outputDirectory`).\n- Directory for current project, or the other project for cross-project subscriptions: `gs://$BUCKET/$DIR/$PROJECT/ ...`\n- Directory for every message type and major version: `gs://$BUCKET/$DIR/$PROJECT/user-created-v1/ ...`\n- Directory for year: `gs://$BUCKET/$DIR/$PROJECT/user-created-v1/2020/ ...`\n- Directory for month: `gs://$BUCKET/$DIR/$PROJECT/user-created-v1/2020/08/ ...`\n- Directory for day: `gs://$BUCKET/$DIR/$PROJECT/user-created-v1/2020/08/25/ ...`\n- Files bucketed in multiple shards in x-min windows: `gs://$BUCKET/$DIR/$PROJECT/user-created-v1/2020/08/25/user-created-v1-21:04:00-21:06:00-0-of-1.gz ...` (window supplied as input parameter `windowDuration`).\n\nCode is based on [this blog post](https://medium.com/@robinriclet/streaming-multiple-pubsub-subscriptions-to-gcs-with-fixed-windows-and-dynamic-naming-7f70cdd4584).\n\n## Reading files with unknown protobuf messages\n\n```shell script\nOUTPUT_FILE=\u003cpath to firehosed file (must start with gs://)\u003e\nSCHEMA_FILE=\u003cpath to your schema .proto file\u003e\nCONTAINER_SCHEMA_FILE=\u003cpath to container schema .proto file\u003e\n\ngsutil cat gs://${OUTPUT_FILE} | \\\n  gzip -d | \\\n  while IFS= read -r line; do \\\n    echo -n $line | \\\n    base64 -d | \\\n    protoc --decode PayloadV1 --proto_path=/usr/local/lib/protobuf/include --proto_path=$(dirname ${CONTAINER_SCHEMA_FILE}) ${CONTAINER_SCHEMA_FILE} | \\\n    python -c 'import sys, pathlib; sys.stdout.buffer.write(bytes(str(next(line[6:].strip(b\"\\\"\") for line in sys.stdin.buffer.read().split(b\"\\n\") if line.startswith(b\"data:\")), \"unicode_escape\"), \"raw_unicode_escape\"))' | \\\n    protoc --decode hedwig.VisitCreatedV1 --proto_path=/usr/local/lib/protobuf/include --proto_path=$(dirname ${SCHEMA_FILE}) ${SCHEMA_FILE}; \\\n  done;\n```\n\n## Deploying\n\n- Install `protoc`\n- Install Hedwig custom options:\n    ```shell\n    git clone https://github.com/cloudchacho/hedwig.git /usr/local/lib/protobuf/include/hedwig\n    ```\n- Define your Hedwig schema in one or more files.\n- Compile all the files in your schema into a fileDescriptorSet file:\n    ```shell\n    protoc --descriptor_set_out=schema-v1 -I /usr/local/lib/protobuf/include/ hedwig/protobuf/options.proto google/protobuf/descriptor.proto \u003cSCHEMA FILES...\u003e\n    ```\n  If your schema uses any other dependencies, make sure to compile them in as well (e.g. for timestamp, add `google/protobuf/timestamp.proto` to the command).\n- Create a pom file:\n    ```xml\n    \u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n    \u003cproject xmlns=\"http://maven.apache.org/POM/4.0.0\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd\"\u003e\n      \u003cmodelVersion\u003e4.0.0\u003c/modelVersion\u003e\n\n    \u003cgroupId\u003ecom.example\u003c/groupId\u003e\n    \u003cartifactId\u003ehedwig-firehose-dataflow\u003c/artifactId\u003e\n    \u003cversion\u003e0.1\u003c/version\u003e\n\n      \u003cdependencies\u003e\n        \u003cdependency\u003e\n          \u003cgroupId\u003eio.github.cloudchacho\u003c/groupId\u003e\n          \u003cartifactId\u003ehedwig-firehose-dataflow\u003c/artifactId\u003e\n          \u003cversion\u003e0.4\u003c/version\u003e\n        \u003c/dependency\u003e\n      \u003c/dependencies\u003e\n\n      \u003cbuild\u003e\n        \u003cplugins\u003e\n          \u003cplugin\u003e\n            \u003cgroupId\u003eorg.codehaus.mojo\u003c/groupId\u003e\n            \u003cartifactId\u003eexec-maven-plugin\u003c/artifactId\u003e\n            \u003cversion\u003e3.0.0\u003c/version\u003e\n            \u003cconfiguration\u003e\n              \u003cincludeProjectDependencies\u003etrue\u003c/includeProjectDependencies\u003e\n              \u003cmainClass\u003eio.github.cloudchacho.hedwig.Firehose\u003c/mainClass\u003e\n            \u003c/configuration\u003e\n          \u003c/plugin\u003e\n        \u003c/plugins\u003e\n      \u003c/build\u003e\n    \u003c/project\u003e\n    ```\n- Deploy your template using the following command, adjusting variables as necessary:\n    ```shell\n    DATAFLOW_BUCKET=\u003c...\u003e\n    FIREHOSE_BUCKET=\u003c...\u003e\n    REGION=\"us-central1\"\n\n    version=$(mvn -q -Dexec.executable=echo -Dexec.args='${project.version}' --non-recursive exec:exec)    firehose_location=\"gs://${FIREHOSE_BUCKET}/firehose\"\n    dataflow_bucket=\"gs://${DATAFLOW_BUCKET}\"\n    dataflow_template=\"${dataflow_bucket}/templates/hedwig-firehose-v${version}\"\n    dataflow_temp=\"${dataflow_bucket}/temp\"\n    dataflow_staging=\"${dataflow_bucket}/stage\"\n    schema_file=\"${dataflow_bucket}/schemas/schema-v1\"\n    args=\"\\\n    --runner=DataflowRunner \\\n    --project=${GCP_PROJECT} \\\n    --stagingLocation=${dataflow_staging} \\\n    --templateLocation=${dataflow_template} \\\n    --region=${REGION} \\\n    --tempLocation=${dataflow_temp} \\\n    --userTempLocation=${dataflow_bucket}/tmp/ \\\n    --outputDirectory=${firehose_location} \\\n    --inputSubscriptions=\u003c...\u003e \\\n    --inputSubscriptionsCrossProject=\u003c...\u003e \\\n    --schemaFileDescriptorSetFile=${schema_file}\"\n\n    mvn compile exec:java -Dexec.args=\"$args\"\n    ```\n\n    To enable debug logging, add:\n    ```shell\n    --workerLogLevelOverrides='{\\\"io.cloudchacho.hedwig.Firehose\\\":\\\"DEBUG\\\"}'\n    ```\n    to `args`.\n\n## Development\n\nReleasing new version of this library:\n\n1. Make code changes, PR, and update version in pom.xml\n2. `mvn clean deploy -P release`\n3. Log onto Nexus: https://s01.oss.sonatype.org/#stagingRepositories\n4. Go to staging repositories\n5. Find the repository that contains the uploaded version (there would normally only be one)\n6. 'close' the repo\n7. Wait a few minutes\n8. 'release' the repo\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudchacho%2Fhedwig-firehose-dataflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudchacho%2Fhedwig-firehose-dataflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudchacho%2Fhedwig-firehose-dataflow/lists"}