Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/aboisvert/s3cp

Amazon S3 command-line tools (e.g., cp, ls, cat, ...)
https://github.com/aboisvert/s3cp

Last synced: 10 days ago
JSON representation

Amazon S3 command-line tools (e.g., cp, ls, cat, ...)

Awesome Lists containing this project

README 

        
S3CP: Commands-line tools for Amazon S3 file manipulation

=============================================================



Just a few simple command-line utilities to list, copy, view S3 files, e.g.,  `s3cp`, `s3ls`, `s3cat`, `s3rm`, etc.



### Installing ###



Make sure you have Rubygems installed on your system then run:



    # gem install s3cp



### Examples ###



    export AWS_ACCESS_KEY_ID=...

    export AWS_SECRET_ACCESS_KEY=...



    s3ls s3://mybucket/path/to/some/files

    s3dir s3://mybucket/path/to/some/files

    s3cat s3://mybucket/path/to/some/file.txt

    s3cp local_file.bin s3://mybucket/some/path

    s3mod s3://mybucket/path/to/some/file.txt public-read

    s3stat s3://mybucket/path/to/some/file.txt

    s3du --depth 2 --unit mb s3://mybucket/some/path/



Use the `-h` option to learn about command-line options.



All commands support both `s3://bucket/path/to/file` and the legacy `bucket:path/to/file` syntax.



Commands are also TTY-aware;  when run in an interactive shell, their behavior will change.  For example, `s3cat` will launch your favorite `PAGER` or `less` (the default pager) whereas `s3ls` will display N items at a time, where N is the number of display lines on your terminal and pause between pages.



### Security / Credentials ###



Starting with v1.1.0, S3CP uses the default credential provider from the aws-sdk which makes a best effort to locate your AWS credentials.  It checks a variety of locations in the following order:



* Static credentials from AWS.config (e.g. AWS.config.access_key_id, AWS.config.secret_access_key)



* The environment (e.g. ENV['AWS_ACCESS_KEY_ID'] or ENV['AMAZON_ACCESS_KEY_ID'])



* EC2 metadata service (checks for credentials provided by roles for instances).



### Usage ###



    $ s3cp

    s3cp supports 4 copying use cases:

      1. Copy from local machine to S3

      2. Copy from S3 to local machine

      3. Copy from S3 to S3

      4. Copy from local machine to another path on local machine (for completeness)



    Local to S3:

      s3cp LOCAL_PATH S3_PATH



    S3 to Local:

      s3cp S3_PATH LOCAL_PATH



    S3 to S3:

      s3cp S3_PATH S3_PATH2



    Local to Local:

      s3cp LOCAL_PATH LOCAL_PATH2



    -r, --recursive                  Recursive mode



        --headers 'Header1: Header1Value','Header2: Header2Value'

                                     Headers to set on the item in S3.

        e.g.,

              HTTP headers: 'Content-Type: image/jpg'

               AMZ headers: 'x-amz-acl: public-read'



        --verbose                    Verbose mode

        --debug                      Debug mode

    -h, --help                       Show this message



---



    $ s3ls

    s3ls [path]



    -l                               Long listing format

        --date-format FORMAT         Date format (see http://strfti.me/)

        --verbose                    Verbose mode

        --rows ROWS                  Rows per page

    -h, --help                       Show this message



---



    $ s3cat

    s3cat [path]



        --debug                      Debug mode

        --tty                        TTY mode

    -h, --help                       Show this message



---



    $ s3du [path] # Display disk usage



        --unit UNIT                  Force unit to use for file size display: B, KB, MB, GB, TB, EB, ZB, YB, BB.

        --precision PRECISION        Precision used to display sizes, e.g. 3 => 0.123GB. (default 0)

        --depth DEPTH                Depth to report space usage (default 0).

        --regex REGEX                Regular expression to match keys.

    -h, --help                       Show this message



---



    $ s3mod

    s3mod [path] [permission]



    where [permission] is one of:



     * private

     * authenticated-read

     * public-read

     * public-read-write



    -h, --help                       Show this message



---



    $ s3rm

    s3rm [path]



    -r, --recursive                  Delete S3 keys matching provided prefix.

    -i, --include REGEX              Delete only S3 objects matching the following regular expression.

    -x, --exclude REGEX              Do not delete any S3 objects matching provided regular expression.

    -F, --fail-if-not-exist          Fail if no S3 object match provided key, prefix and/or regex

    -t, --test                       Only display matching keys; do not actually delete anything.

        --silent                     Do not display keys as they are deleted.

        --verbose                    Verbose mode

    -h, --help                       Show this message



### Bash completion for S3 URLs ###



To install Bash completion for S3 URLs, add the following to ~/.bashrc:



    for cmd in [ s3cat s3cp s3dir s3ls s3mod s3rm s3stat ]; do

      complete -C s3cp_complete $cmd

    done



### Dependencies ###



* extensions '~> 0.6'     (portability between Ruby 1.8.7/1.9.2)

* highline `>=1.5.1`      (console/terminal size guessing)

* aws-sdk '~> 1.6.3'      (underlying Amazon S3 API)

* progressbar '~> 0.10.0' (nice console progress output)



### Target platform ###



* Ruby 1.8.7 / 1.9.2



### Development ###



If you want to hack on s3cp and build the gem yourself, you will need Bundler (http://gembundler.com/) installed.  Here is a typical development setup:



    # git clone [email protected]:aboisvert/s3cp.git

    # cd s3cp

    # bundle install

    (... hack on s3cp ...)

    # bundle exec rake gem

    # gem install s3cp-*.gem



### License ###



S3CP is is licensed under the terms of the Apache Software License v2.0.