{"id":20849627,"url":"https://github.com/augustoscher/csv-streaming","last_synced_at":"2026-05-08T14:32:55.548Z","repository":{"id":38767192,"uuid":"284777045","full_name":"augustoscher/csv-streaming","owner":"augustoscher","description":"Streaming csv files from S3 bucket to an SQS queue.","archived":false,"fork":false,"pushed_at":"2023-01-27T02:19:20.000Z","size":2027,"stargazers_count":2,"open_issues_count":18,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-31T06:44:15.811Z","etag":null,"topics":["aws","lambda-functions","node","node-streams","s3","serverless-framework","sqs"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/augustoscher.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-03T18:27:07.000Z","updated_at":"2023-03-09T00:26:30.000Z","dependencies_parsed_at":"2023-02-15T04:15:54.714Z","dependency_job_id":null,"html_url":"https://github.com/augustoscher/csv-streaming","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/augustoscher/csv-streaming","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/augustoscher%2Fcsv-streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/augustoscher%2Fcsv-streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/augustoscher%2Fcsv-streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/augustoscher%2Fcsv-streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/augustoscher","download_url":"https://codeload.github.com/augustoscher/csv-streaming/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/augustoscher%2Fcsv-streaming/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32784421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-08T08:22:46.396Z","status":"ssl_error","status_checked_at":"2026-05-08T08:22:45.650Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","lambda-functions","node","node-streams","s3","serverless-framework","sqs"],"created_at":"2024-11-18T03:05:58.437Z","updated_at":"2026-05-08T14:32:55.520Z","avatar_url":"https://github.com/augustoscher.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# csv-streaming\n\nServerless app to stream an CSV file from S3 bucket to SQS queue.\n\n## Requirements\n\n- Docker and Docker compose needs to be installed to run locally.\n- AWS account with aws client working properly.\n- Node Serverless framework client installed.\n\n## Setup\n\nClone repository and install node dependencies.\n\n```bash\ngit clone git@github.com:augustoscher/csv-streaming.git\ncd csv-streaming\nnpm i\n```\n\n## Running\n\nYou can run it locally throw docker container or in AWS cloud. Just follow the steps below.\n\n### Locally\n\n#### Setup docker containers\n\n```bash\nmake run-logs\n```\n\nNow you should see localstack resources on [localhost:8080](http://localhost:8080/#!/infra).\n\n#### Setup resources\n\nIn another terminal, run:\n\n```bash\nmake setup\n```\n\nHere we're creating S3 bucket and SQS queue on localstack.\nNow you should see resources by refreshing localstack page.\n\nAnd the last thing to do is run app locally. To do that, type:\n\n```bash\nmake start\n```\n\n#### Sending CSV file to local s3 bucket\n\nNow It's time to send csv to our local s3 bucket and see if it's streaming, converting and sending to sqs queue.\n\nOpen a new terminal window and run:\n\n```bash\nmake test-s3\n```\n\nPs: Here we're uploading an csv file to local s3 bucket and manually calling s3listener because localstack triggers is not setup.\n\nNow we'll se if SQS queue receive objetcs, by getting All attributes:\n\n```bash\nmake test-sqs\n```\n\nYou should see `get-queue-attributes` output:\n\n```json\n{\n  \"Attributes\": {\n    \"VisibilityTimeout\": \"30\",\n    \"DelaySeconds\": \"0\",\n    \"ReceiveMessageWaitTimeSeconds\": \"0\",\n    \"ApproximateNumberOfMessages\": \"400\",\n    \"ApproximateNumberOfMessagesNotVisible\": \"0\",\n    \"ApproximateNumberOfMessagesDelayed\": \"0\",\n    \"CreatedTimestamp\": \"1596490739\",\n    \"LastModifiedTimestamp\": \"1596490739\",\n    \"QueueArn\": \"arn:aws:sqs:us-east-1:000000000000:csv-streaming\"\n  }\n}\n```\n\n### AWS Cloud\n\nLet's deploy and run it on AWS cloud.\n\n#### Deploying\n\nYou obviously need to deploy it on AWS. We're using serverless-framework:\n\n```bash\nsls deploy\n```\n\nAll resources (lambdas, bucket and sqs) will be created automatically.\n\n```bash\naws sqs list-queues\naws s3 ls | grep csv-streaming\n```\n\n#### Testing\n\nRun each command in diferent terminal:\n\n1. Sending csv file to ou s3 bucket:\n\n```bash\naws s3 cp \\\n  testing/file.csv \\\n  s3://csv-streaming\n```\n\n2. Run logs on s3 bucket in AWS cloud:\n\n```bash\nnpx sls logs -f s3listener -t\n```\n\n3. Run logs on sqs queue in AWS cloud:\n\n```bash\nnpx sls logs -f sqslistener -t\n```\n\nOptionally: Upload huge csv\n\n```bash\naws s3 cp testing/survey_results_public.csv s3://csv-streaming\n```\n\n#### Removing\n\nFinal step: Remove all resources, including s3 bucket and sqs queue:\n\n```bash\nsls remove\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faugustoscher%2Fcsv-streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faugustoscher%2Fcsv-streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faugustoscher%2Fcsv-streaming/lists"}