{"id":18521597,"url":"https://github.com/transferwise/pipelinewise-tap-s3-csv","last_synced_at":"2025-12-30T06:05:06.609Z","repository":{"id":37382158,"uuid":"189064689","full_name":"transferwise/pipelinewise-tap-s3-csv","owner":"transferwise","description":"Singer.io Tap for CSV files on S3 - PipelineWise compatible","archived":true,"fork":false,"pushed_at":"2024-09-19T23:43:19.000Z","size":155,"stargazers_count":7,"open_issues_count":15,"forks_count":34,"subscribers_count":65,"default_branch":"master","last_synced_at":"2025-03-03T04:44:52.617Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://transferwise.github.io/pipelinewise/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/transferwise.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-28T16:27:36.000Z","updated_at":"2024-09-23T03:16:45.000Z","dependencies_parsed_at":"2024-06-21T14:23:18.193Z","dependency_job_id":"c68a1a6f-4fcf-45e3-b6d6-361437f8bec8","html_url":"https://github.com/transferwise/pipelinewise-tap-s3-csv","commit_stats":{"total_commits":147,"total_committers":17,"mean_commits":8.647058823529411,"dds":0.6122448979591837,"last_synced_commit":"de4468228a0f69455b0edf066957bf7520d0033a"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Fpipelinewise-tap-s3-csv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Fpipelinewise-tap-s3-csv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Fpipelinewise-tap-s3-csv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Fpipelinewise-tap-s3-csv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/transferwise","download_url":"https://codeload.github.com/transferwise/pipelinewise-tap-s3-csv/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248012848,"owners_count":21033252,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T17:26:42.075Z","updated_at":"2025-12-13T18:06:10.405Z","avatar_url":"https://github.com/transferwise.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Notice\nTo better serve Wise business and customer needs, the PipelineWise codebase needs to shrink.\nWe have made the difficult decision that, going forward many components of PipelineWise will be removed or incorporated in the main repo.\nThe last version before this decision is [v0.64.1](https://github.com/transferwise/pipelinewise/tree/v0.64.1)\n\nWe thank all in the open-source community, that over the past 6 years, have helped to make PipelineWise a robust product for heterogeneous replication of many many Terabytes, daily\n\n# pipelinewise-tap-s3-csv\n\n[![PyPI version](https://badge.fury.io/py/pipelinewise-tap-s3-csv.svg)](https://badge.fury.io/py/pipelinewise-tap-s3-csv)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pipelinewise-tap-s3-csv.svg)](https://pypi.org/project/pipelinewise-tap-s3-csv/)\n[![License: MIT](https://img.shields.io/badge/License-GPLv3-yellow.svg)](https://opensource.org/licenses/GPL-3.0)\n\nThis is a [Singer](https://singer.io) tap that reads data from files located inside a given S3 bucket and produces JSON-formatted data following the [Singer spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).\n\nThis is a [PipelineWise](https://transferwise.github.io/pipelinewise) compatible tap connector.\n\n## How to use it\n\nThe recommended method of running this tap is to use it from [PipelineWise](https://transferwise.github.io/pipelinewise). When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation at [Tap S3 CSV](https://transferwise.github.io/pipelinewise/connectors/taps/s3_csv.html)\n\nIf you want to run this [Singer Tap](https://singer.io) independently please read further.\n\n### Install and Run\n\nFirst, make sure Python 3 is installed on your system or follow these\ninstallation instructions for [Mac](http://docs.python-guide.org/en/latest/starting/install3/osx/) or\n[Ubuntu](https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-local-programming-environment-on-ubuntu-16-04).\n\nIt's recommended to use a virtualenv:\n\n```bash\n  python3 -m venv venv\n  pip install pipelinewise-tap-s3-csv\n```\n\nor\n\n```bash\n  python3 -m venv venv\n  . venv/bin/activate\n  pip install --upgrade pip\n  pip install .\n```\n\n### Configuration\n\nHere is an example of basic config, that's using the defualt Profile based authentication:\n\n    ```json\n    {\n        \"start_date\": \"2000-01-01T00:00:00Z\",\n        \"bucket\": \"tradesignals-crawler\",\n        \"tables\": [{\n            \"search_prefix\": \"feeds\",\n            \"search_pattern\": \".csv\",\n            \"table_name\": \"my_table\",\n            \"key_properties\": [\"id\"],\n            \"delimiter\": \",\"\n        }]\n    }\n    ```\n\n### Profile based authentication\n\nProfile based authentication used by default using the `default` profile. To use another profile set `aws_profile` parameter in `config.json` or set the `AWS_PROFILE` environment variable.\n\n### Non-Profile based authentication\n\nFor non-profile based authentication set `aws_access_key_id` , `aws_secret_access_key` and optionally the `aws_session_token` parameter in the `config.json`. Alternatively you can define them out of `config.json` by setting `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` environment variables.\n\n\n A bit of a run down on each of the properties:\n\n- **aws_profile**: AWS Profile name for Profile based authentication. If not provided, `AWS_PROFILE` environment variable will be used.\n- **aws_access_key_id**: AWS access key ID for Non-Profile based authentication. If not provided, `AWS_ACCESS_KEY_ID` environment variable will be used.\n- **aws_secret_access_key**: AWS secret access key for Non-Profile based authentication. If not provided, `AWS_SECRET_ACCESS_KEY` environment variable will be used.\n- **aws_session_token**: AWS session token for Non-Profile based authentication. If not provided, `AWS_SESSION_TOKEN` environment variable will be used.\n- **aws_endpoint_url**: (Optional): The complete URL to use for the constructed client. Normally, botocore will automatically construct the appropriate URL to use when communicating with a service. You can specify a complete URL (including the \"http/https\" scheme) to override this behavior. For example https://nyc3.digitaloceanspaces.com\n- **start_date**: This is the datetime that the tap will use to look for newly updated or created files, based on the modified timestamp of the file.\n- **bucket**: The name of the bucket to search for files under.\n- **tables**: JSON object that the tap will use to search for files, and emit records as \"tables\" from those files.\n\nThe `table` field consists of one or more objects, that describe how to find files and emit records. A more detailed (and unescaped) example below:\n\n```\n[\n    {\n        \"search_prefix\": \"exports\"\n        \"search_pattern\": \"my_table\\\\/.*\\\\.csv\",\n        \"table_name\": \"my_table\",\n        \"key_properties\": [\"id\"],\n        \"date_overrides\": [\"created_at\"],\n        \"delimiter\": \",\"\n    },\n    ...\n]\n```\n\n- **search_prefix**: This is a prefix to apply after the bucket, but before the file search pattern, to allow you to find files in \"directories\" below the bucket.\n- **search_pattern**: This is an escaped regular expression that the tap will use to find files in the bucket + prefix. It's a bit strange, since this is an escaped string inside of an escaped string, any backslashes in the RegEx will need to be double-escaped.\n- **table_name**: This value is a string of your choosing, and will be used to name the stream that records are emitted under for files matching content.\n- **key_properties**: These are the \"primary keys\" of the CSV files, to be used by the target for deduplication and primary key definitions downstream in the destination.\n- **date_overrides**: Specifies field names in the files that are supposed to be parsed as a datetime. The tap doesn't attempt to automatically determine if a field is a datetime, so this will make it explicit in the discovered schema.\n- **delimiter**: This allows you to specify a custom delimiter, such as `\\t` or `|`, if that applies to your files.\n\nA sample configuration is available inside [config.sample.json](config.sample.json)\n\n### To run tests:\n\n1. Install python test dependencies in a virtual env and run nose unit and integration tests\n```\n  make venv\n```\n\n2. To run unit tests:\n```\n  make unit_tests\n```\n\n3. To run integration tests:\n\nIntegration tests require a valid S3 bucket and credentials should be passed as environment variables, this project uses Minio server.\n\nFist, start a Minio server docker container:\n```shell\nmkdir -p ./minio/data/awesome_bucket\nUID=$(id -u) GID=$(id -g) docker-compose up -d\n```\n\nRun integration tests:\n```shell\n  make integration_tests\n```\n\n### To run pylint:\n\n1. Install python dependencies and run python linter\n```\n  make venv pylint\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftransferwise%2Fpipelinewise-tap-s3-csv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftransferwise%2Fpipelinewise-tap-s3-csv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftransferwise%2Fpipelinewise-tap-s3-csv/lists"}