https://github.com/edgarrmondragon/meltano-s3-csv-example
Example Meltano project to extract data from files in S3
https://github.com/edgarrmondragon/meltano-s3-csv-example
etl-pipeline meltano
Last synced: about 1 month ago
JSON representation
Example Meltano project to extract data from files in S3
- Host: GitHub
- URL: https://github.com/edgarrmondragon/meltano-s3-csv-example
- Owner: edgarrmondragon
- Created: 2023-09-27T23:03:50.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-03-15T05:39:07.000Z (about 1 year ago)
- Last Synced: 2025-02-01T05:26:25.399Z (3 months ago)
- Topics: etl-pipeline, meltano
- Homepage: https://docs.meltano.com
- Size: 69.3 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Example Meltano project to extract data from files in S3
## Prerequisites
* [Docker][install-docker]
* [Meltano][install-meltano]
## Setup
1. Clone this repository
2. Run `meltano install` to install the project's dependencies
3. Run `docker compose up -d` to start the project's services (MinIO)
4. Run `meltano run tap-spreadsheets-anywhere target-duckdb` to run the pipeline
You should now be be able to query the data in the `output/warehouse.duckdb` file:
```console
$ duckdb output/warehouse.duckdb -c 'select id, email from tap_spreadsheets_anywhere.customers limit 5;'
┌────────┬───────────────────────────┐
│ id │ email │
│ int128 │ varchar │
├────────┼───────────────────────────┤
│ 1 │ [email protected] │
│ 2 │ [email protected] │
│ 3 │ [email protected] │
│ 4 │ [email protected] │
│ 5 │ [email protected] │
└────────┴───────────────────────────┘
```
[install-docker]: https://docs.docker.com/engine/install/
[install-meltano]: https://docs.meltano.com/getting-started/installation/