{"id":24994420,"url":"https://github.com/marekaf/gcp-stackdriver-logging-extract-textpayload","last_synced_at":"2025-04-12T03:24:37.231Z","repository":{"id":54205682,"uuid":"167682507","full_name":"marekaf/gcp-stackdriver-logging-extract-textpayload","owner":"marekaf","description":"Extract textPayload from json from Stackdriver Logging exports to Google Storage Bucket","archived":false,"fork":false,"pushed_at":"2021-03-03T07:55:04.000Z","size":5,"stargazers_count":3,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-06T12:37:35.460Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marekaf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-26T11:43:31.000Z","updated_at":"2021-03-03T07:54:58.000Z","dependencies_parsed_at":"2022-08-13T09:10:41.520Z","dependency_job_id":null,"html_url":"https://github.com/marekaf/gcp-stackdriver-logging-extract-textpayload","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marekaf%2Fgcp-stackdriver-logging-extract-textpayload","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marekaf%2Fgcp-stackdriver-logging-extract-textpayload/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marekaf%2Fgcp-stackdriver-logging-extract-textpayload/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marekaf%2Fgcp-stackdriver-logging-extract-textpayload/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marekaf","download_url":"https://codeload.github.com/marekaf/gcp-stackdriver-logging-extract-textpayload/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248511083,"owners_count":21116349,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-04T14:53:21.472Z","updated_at":"2025-04-12T03:24:37.210Z","avatar_url":"https://github.com/marekaf.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Stackdriver log transfomer\nDeploy a GCP Cloud Function (nodejs 6 runtime) that is triggered on `google.storage.object.finalize` event.\n\nThe function does:\n* downloads the file from SOURCE_BUCKET_NAME bucket\n* takes the whole json and extracts the `.textPayload` of each and every line\n* writes it to a local fileStream\n* changes the MIME type from `application/json` to `text/plain`\n* uploads it to DEST_BUCKET_NAME with the same file name\n\nThis is useful as Stackdriver Logging exports to bucket are exporting json with metadata for each log entry BUT you need only the textPayload (the raw log entry).\n\nYou should have the export sink to bucket already created. \n\nMore info here: \n\n## deploy it to google cloud functions\n```\nSOURCE_BUCKET_NAME=my-bucket-exported-logs\nDEST_BUCKET_NAME=my-bucket-exported-logs-nojson\n\ngcloud beta functions deploy logTransformer \\ \n   --set-env-vars DEST_BUCKET_NAME=\"$DEST_BUCKET_NAME\" \\\n   --runtime nodejs6 \\\n   --region europe-west1 \\\n   --memory 128MB  \\\n   --trigger-resource \"$SOURCE_BUCKET_NAME\" \\\n   --trigger-event google.storage.object.finalize\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarekaf%2Fgcp-stackdriver-logging-extract-textpayload","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarekaf%2Fgcp-stackdriver-logging-extract-textpayload","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarekaf%2Fgcp-stackdriver-logging-extract-textpayload/lists"}