https://github.com/marekaf/gcp-stackdriver-logging-extract-textpayload
Extract textPayload from json from Stackdriver Logging exports to Google Storage Bucket
https://github.com/marekaf/gcp-stackdriver-logging-extract-textpayload
Last synced: over 1 year ago
JSON representation
Extract textPayload from json from Stackdriver Logging exports to Google Storage Bucket
- Host: GitHub
- URL: https://github.com/marekaf/gcp-stackdriver-logging-extract-textpayload
- Owner: marekaf
- Created: 2019-01-26T11:43:31.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2021-03-03T07:55:04.000Z (over 5 years ago)
- Last Synced: 2025-04-06T12:37:35.460Z (over 1 year ago)
- Language: JavaScript
- Size: 4.88 KB
- Stars: 3
- Watchers: 2
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Stackdriver log transfomer
Deploy a GCP Cloud Function (nodejs 6 runtime) that is triggered on `google.storage.object.finalize` event.
The function does:
* downloads the file from SOURCE_BUCKET_NAME bucket
* takes the whole json and extracts the `.textPayload` of each and every line
* writes it to a local fileStream
* changes the MIME type from `application/json` to `text/plain`
* uploads it to DEST_BUCKET_NAME with the same file name
This is useful as Stackdriver Logging exports to bucket are exporting json with metadata for each log entry BUT you need only the textPayload (the raw log entry).
You should have the export sink to bucket already created.
More info here:
## deploy it to google cloud functions
```
SOURCE_BUCKET_NAME=my-bucket-exported-logs
DEST_BUCKET_NAME=my-bucket-exported-logs-nojson
gcloud beta functions deploy logTransformer \
--set-env-vars DEST_BUCKET_NAME="$DEST_BUCKET_NAME" \
--runtime nodejs6 \
--region europe-west1 \
--memory 128MB \
--trigger-resource "$SOURCE_BUCKET_NAME" \
--trigger-event google.storage.object.finalize
```