{"id":31763707,"url":"https://github.com/kidtronnix/barf","last_synced_at":"2026-07-13T18:32:32.597Z","repository":{"id":57591587,"uuid":"83918666","full_name":"kidtronnix/barf","owner":"kidtronnix","description":"when you need to barf up data!","archived":false,"fork":false,"pushed_at":"2017-03-09T16:37:10.000Z","size":5459,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-10-30T14:52:00.372Z","etag":null,"topics":["aws-s3","data-science","data-wrangling","golang-library","unix-command"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kidtronnix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-03-04T19:22:29.000Z","updated_at":"2017-03-27T18:43:01.000Z","dependencies_parsed_at":"2022-09-26T19:43:16.205Z","dependency_job_id":null,"html_url":"https://github.com/kidtronnix/barf","commit_stats":null,"previous_names":["smaxwellstewart/barf"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/kidtronnix/barf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kidtronnix%2Fbarf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kidtronnix%2Fbarf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kidtronnix%2Fbarf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kidtronnix%2Fbarf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kidtronnix","download_url":"https://codeload.github.com/kidtronnix/barf/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kidtronnix%2Fbarf/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35432816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-13T02:00:06.543Z","response_time":119,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-s3","data-science","data-wrangling","golang-library","unix-command"],"created_at":"2025-10-09T23:23:37.846Z","updated_at":"2026-07-13T18:32:32.592Z","avatar_url":"https://github.com/kidtronnix.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg alt=\"barf-logo\" src=\"https://raw.githubusercontent.com/smaxwellstewart/barf/master/logo.png\" width=\"200\" align=\"right\" /\u003e\n\n# barf\n\nBarf is a data wrangling tool for producing data streams from an AWS S3 bucket.\n\nEvery line of every file is sent through the stream until it closes. This is very useful for processing lines of csv / event log files.\n\nAs a bonus, any files in your S3 bucket that end with `.gz` or `.bz2` will be\nautomatically decompressed using gzip or bzip2 algorithms respectively.\n\nBarf uses the AWS recommended way of authenticating by automatically reading the needed credentials from the enviroment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. If these enviroment variables are not set the binary will panic.\n\nLogo by [Enemist Arts](https://www.instagram.com/enemistarts/).\n\n## cli\n\n### Install\n\nThe latest binaries for mac, windows and linux are hosted on the [releases](https://github.com/smaxwellstewart/barf/releases)\npage of the github project. You will need to download them and put them somewhere on your machine. It is recommend you put them on you path.\n\n#### Linux / OSX\n\nFirst, download the correct binary for your machine.\n\n```\n# move binary onto your path\n$ mv ./barf_linux /usr/local/bin/barf # use _osx for osx binary\n\n# make binary executable\nchmod +x /usr/local/bin/barf\n\n# test!\nbarf\n```\n\n### Usage\n\nThink of this as a recursive `cat` command for all files\nin your S3 bucket under a certain prefix.\n\n```sh\n$ barf s3://bucket/prefix/\n# prepare for the data vom!\n```\n\nIn the more advanced example below, we set a value for `flow` which controls the\nrate of http calls / sec. We also set a `duration` to get a small amount of data\nby opening up our stream for a limited time.\nFinally we pipe everything to some output file for safe keeping.\n\n```sh\n$ barf -flow=\"10.0\" -duration=\"3s\" s3://bucket/prefix/ \u003e output\n```\n\n\n## go library\n\nThe other way of using barf is as golang library in your own projects.\n\n### Example\n\n```go\npackage main\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"log\"\n\t\"time\"\n\n\t\"github.com/seedboxtech/barf/gobarf\"\n)\n\n\nfunc main() {\n\t// setup new stream of data\n\tb := barf.New(barf.Stream{\n\t\tSrc:      \"s3://myawsbucket/prefix/to/my_files\",\n\t})\n\tdefer b.Close()\n\n\t// read stream and print stdout\n  \tfor line := range b.Barf() {\n\t\t// do something with the line of content\n\t}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkidtronnix%2Fbarf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkidtronnix%2Fbarf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkidtronnix%2Fbarf/lists"}