{"id":17485667,"url":"https://github.com/nickrobison/pars3","last_synced_at":"2025-06-18T02:37:25.836Z","repository":{"id":31400525,"uuid":"34963741","full_name":"nickrobison/ParS3","owner":"nickrobison","description":"Parallel file tool for Amazon S3 storage","archived":false,"fork":false,"pushed_at":"2015-05-04T08:25:39.000Z","size":140,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-19T05:22:36.257Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nickrobison.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-05-02T21:22:03.000Z","updated_at":"2015-05-02T21:30:19.000Z","dependencies_parsed_at":"2022-09-25T05:01:13.608Z","dependency_job_id":null,"html_url":"https://github.com/nickrobison/ParS3","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/nickrobison/ParS3","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickrobison%2FParS3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickrobison%2FParS3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickrobison%2FParS3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickrobison%2FParS3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nickrobison","download_url":"https://codeload.github.com/nickrobison/ParS3/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickrobison%2FParS3/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260475935,"owners_count":23014942,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-19T02:08:26.710Z","updated_at":"2025-06-18T02:37:20.824Z","avatar_url":"https://github.com/nickrobison.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ParS3\n\nParS3 is a programming for downloading and uploading files from Amazon's\nS3 service, in parallel.\n\nThe project was created due to the fact that (at the time) s3cmd would\noverflow when getting millions of files. ParS3 uses channels to\nasynchronously download files before the entirety of the directory is\nlisted, which means it's much quicker than some other implementations.\n\nWhere it really shines is pulling down log files from Elastic Map\nReduce. It's really good at that.\n\n## Getting Started\n\n### Installation\n- Install Golang: https://golang.org/doc/install\n- Get ParS3: go get github.com/nickrobison/ParS3\n\n### Usage\n\nYou need to create a json file in your home user directory named ParS3\nwith the following contents:\n\n```javascript\n{\n  \"AWSCredentials\": {\n    \"AccessKey\": (your AWS Access Key),\n    \"SecretKey\": (your AWS Secret Key)\n  }\n}\n```\n\nParS3 currently supports the ability to get/put files using the\nfollowing syntax:\n\n```bash\nParS3 get s3://(source-bucket)/(source-directory) (destination-directory)/\n\nParS3 put (source-directory)/*\ns3://(destination-bucket)/(destination-diretory)\n```\n\nMore options will be coming in the near future.\n\n## Other Notes\n\nI'm just now getting around to posting this project but it stands as the first go program that I ever wrote (First commit was July,\n2013) so yes, it is terrible. I knew nothing of interfaces or really\nany idiomatic go. So the source code is terrible, but the program works\nand I'm working on a pretty substantial refactoring that should make\nthings much better.\n\nI'm also planning on extending it to support more storage platforms, so\nhopefully it'll become even more useful. \n\n### TODO List:\n- [ ] Refactor into interfaces\n- [ ] Add OpenStack Swift support\n- [ ] Add Azure Storage\n- [ ] Add HDFS support\n- [ ] Remove all the cruft from a more ignorant self\n- [ ] Fix directory listing\n\nGood luck to us all.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnickrobison%2Fpars3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnickrobison%2Fpars3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnickrobison%2Fpars3/lists"}