{"id":25528941,"url":"https://github.com/meltanolabs/target-jsonl-blob","last_synced_at":"2025-04-11T07:55:13.540Z","repository":{"id":39617079,"uuid":"465921188","full_name":"MeltanoLabs/target-jsonl-blob","owner":"MeltanoLabs","description":"JSONL Singer target for local storage, S3, GCS and Azure Blob Storage","archived":false,"fork":false,"pushed_at":"2025-03-31T19:50:00.000Z","size":451,"stargazers_count":4,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-01T08:51:38.240Z","etag":null,"topics":["blob-storage","golang","jsonlines","singer-io"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MeltanoLabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-03T23:49:30.000Z","updated_at":"2025-03-31T19:48:43.000Z","dependencies_parsed_at":"2023-10-12T07:05:53.138Z","dependency_job_id":"34380998-3cee-4d9b-8eb8-52e3f9563713","html_url":"https://github.com/MeltanoLabs/target-jsonl-blob","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltanoLabs%2Ftarget-jsonl-blob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltanoLabs%2Ftarget-jsonl-blob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltanoLabs%2Ftarget-jsonl-blob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltanoLabs%2Ftarget-jsonl-blob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MeltanoLabs","download_url":"https://codeload.github.com/MeltanoLabs/target-jsonl-blob/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248360032,"owners_count":21090635,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blob-storage","golang","jsonlines","singer-io"],"created_at":"2025-02-19T23:29:34.135Z","updated_at":"2025-04-11T07:55:13.501Z","avatar_url":"https://github.com/MeltanoLabs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# target-jsonl-blob\n\nJSONL Singer target for local storage, S3, GCS and Azure Blob Storage.\n\n## Installation\n\nTo install this Singer tap, you can download a [prebuilt binary](https://github.com/MeltanoLabs/target-jsonl-blob/releases), or you can [build it from source](#build-from-source).\n\n## Configuration\n\n| Setting | Required | Default | Description |\n|----------------|----------|-------------------------|-------------------------------|\n| `bucket` | Yes | - | Blob storage [bucket URL](#bucket-urls) |\n| `key_template` | No | `{{.StreamName}}.jsonl` | Template string for file keys |\n\n### Bucket URLs\n\n| Storage | Example URL                 | Supported URL parameters                                                            |\n|---------|-----------------------------|-------------------------------------------------------------------------------------|\n| local   | `file:///path/to/directory` | See [supported parameters](https://pkg.go.dev/gocloud.dev/blob/fileblob#URLOpener)  |\n| S3      | `s3://my-bucket`            | See [supported parameters](https://pkg.go.dev/gocloud.dev/blob/s3blob#URLOpener)    |\n| Azure   | `azblob://my-container`     | See [supported parameters](https://pkg.go.dev/gocloud.dev/blob/azureblob#URLOpener) |\n| GCS     | `gs://my-bucket`            | See [supported parameters](https://pkg.go.dev/gocloud.dev/blob/gcsblob#URLOpener)   |\n\n### Available fields for `key_template`\n\n- `StreamName`\n- `Date` (YYYY-MM-DD)\n- `TimestampSeconds`\n- `Minute`\n- `Hour`\n- `Day`\n- `Month`\n- `Year`\n\nExample: `{{.StreamName}}/{{.Year}}/{{.Month}}/{{.Day}}/{{.Hour}}/{{.Minute}}/{{.StreamName}}.jsonl`\n\n## Build from source\n\n```shell\ngo build -o target-jsonl-blob\n```\n\n## Usage with Meltano\n\n1. [Download the appropriate asset](#1-download-the-appropriate-asset)\n1. [Add a custom Meltano plugin to your project](#3-add-a-custom-meltano-plugin-to-your-project)\n1. [Run a pipeline](#4-run-a-pipeline)\n\n### 1. Download the appropriate asset\n\nYou can see the full list of assets in the release page: https://github.com/MeltanoLabs/target-jsonl-blob/releases/latest.\n\nThe [`gh`](https://cli.github.com/) tool makes downloading an asset easy:\n\n```bash\ngh release download v0.0.6 \\\n  -R MeltanoLabs/target-jsonl-blob \\\n  -p '*darwin_amd64*' \\\n  --clobber\n\ntar -xvf target-jsonl-blob_0.0.6_darwin_amd64.tar.gz target-jsonl-blob\n```\n\nYou can also install with Homebrew:\n\n```bash\nbrew tap MeltanoLabs/target-jsonl-blob https://github.com/MeltanoLabs/target-jsonl-blob\nbrew install target-jsonl-blob\n```\n\n### 2. Add a custom Meltano plugin to your project\n\n```yaml\n# meltano.yml\nplugins:\n  loaders:\n  - name: target-jsonl-blob\n    namespace: target_jsonl_blob\n    executable: ./target-jsonl-blob\n    settings:\n    - name: bucket\n      label: Bucket\n      description: Target directory (local, S3, Azure Blob)\n    - name: key_template\n      label: Key Template\n      description: Template string for file keys\n    config:\n      bucket: file://./output/my-bucket\n      key_template: $MELTANO_EXTRACTOR_NAMESPACE/{{.StreamName}}.jsonl\n```\n\nYou also need to ensure the local \"bucket\" exists:\n\n```bash\nmkdir output/my-bucket\n```\n\n### 3. Run a pipeline\n\n```bash\nmeltano run tap-github target-jsonl-blob\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeltanolabs%2Ftarget-jsonl-blob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmeltanolabs%2Ftarget-jsonl-blob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeltanolabs%2Ftarget-jsonl-blob/lists"}