{"id":13857863,"url":"https://github.com/atlanhq/flyio","last_synced_at":"2025-07-13T22:31:18.632Z","repository":{"id":98730264,"uuid":"155813087","full_name":"atlanhq/flyio","owner":"atlanhq","description":"Input Output Files in R from Cloud or Local","archived":false,"fork":false,"pushed_at":"2020-02-11T10:56:58.000Z","size":507,"stargazers_count":46,"open_issues_count":7,"forks_count":11,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-11-22T15:41:27.063Z","etag":null,"topics":["amazon","aws-s3","cloud","gcs","google","r-package","rstats"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/atlanhq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-02T04:13:04.000Z","updated_at":"2024-05-31T00:05:29.000Z","dependencies_parsed_at":"2023-05-25T01:15:06.339Z","dependency_job_id":null,"html_url":"https://github.com/atlanhq/flyio","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/atlanhq/flyio","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atlanhq%2Fflyio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atlanhq%2Fflyio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atlanhq%2Fflyio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atlanhq%2Fflyio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/atlanhq","download_url":"https://codeload.github.com/atlanhq/flyio/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atlanhq%2Fflyio/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265218209,"owners_count":23729496,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon","aws-s3","cloud","gcs","google","r-package","rstats"],"created_at":"2024-08-05T03:01:49.076Z","updated_at":"2025-07-13T22:31:18.611Z","avatar_url":"https://github.com/atlanhq.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"# flyio - Make data fly to R \u003cimg src=\"https://i.imgur.com/XtsxAmX.png\" align=\"right\" /\u003e\nInput and output data from R — download, upload, read and write objects from AWS S3, GoogleCloudStorage or local file system from a single interface.\n\n![](http://www.r-pkg.org/badges/version/flyio) ![](https://cranlogs.r-pkg.org/badges/flyio)\n\n## Overview\n\n**flyio** provides a common interface to interact with data from cloud storage providers or local storage directly from R. It currently supports AWS S3 and Google Cloud Storage, thanks to the API wrappers provided by cloudyr. **flyio** also supports reading or writing tables, rasters, shapefiles and R objects to the data source from memory.\n\n\u003cimg src=\"https://i.imgur.com/qDnhvuB.png\" align=\"centre\" /\u003e\n\n  - `flyio_set_datasource()`: Set the data source (GCS, S3 or local) for all the other functions in flyio.\n  - `flyio_auth()`: Authenticate data source (GCS or S3) so that you have access to the data. In a single session, different data sources can be authenticated.\n  - `flyio_set_bucket()`: Set the bucket name once for any or both data sources so that you don't need to write it in each function.\n  - `list_files()`: List the files in the bucket/folder.\n  - `file_exists()`: Check if a file exists in the bucket/folder.\n  - `export_[file/folder]()`: Upload a file/folder to S3 or GCS from R.\n  - `import_file()`: Download a file from S3 or GCS. \n  - `import_[table/raster/stack/shp/rds/rda/st]()`: Read a file from the set data source and bucket from a user-defined function.\n  - `export_[table/raster/shp/rds/rda/st]()`: Write a file to the set data source and bucket from a user-defined function.\n  \nFor global usage, the datsource, authentication keys and bucket can be set in the environment variables of the machine so that one does not have to input it every time. \n  - For datasource:`CLOUD_STORAGE_NAME`\n  - For bucket name: `flyioBucketS3` or `flyioBucketGcs`\n  - For authentication: `GCS_AUTH_FILE` or `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION` (For AWS S3, if the awscli is athenticated, then this step is not needed)\n \n## Installation\n\n``` r\n# Install the stable version from CRAN:\ninstall.packages(\"flyio\")\n\n# Install the latest dev version from GitHub:\ninstall.packages(\"devtools\")\ndevtools::install_github(\"atlanhq/flyio\")\n\n# Load the library\nlibrary(flyio)\n```\nIf you encounter a bug, please file an issue with steps to reproduce it on Github. Please use the same for any feature requests, enhancements or suggestions.\n\n### Example\n\n``` r\n# Setting the data source\nflyio_set_datasource(\"gcs\")\n\n# Verify if the data source is set\nflyio_get_datasource()\n\n# Authenticate the default data source and set bucket\nflyio_auth(\"key.json\")\nflyio_set_bucket(\"atlanhq-flyio\")\n\n# Authenticate S3 also\nflyio_auth(c(\"AWS_ACCESS_KEY_ID\", \"AWS_SECRET_ACCESS_KEY\", \"AWS_DEFAULT_REGION\", \"AWS_SESSION_TOKEN\"), data_source = \"s3\")\nflyio_set_bucket(\"atlanhq-flyio\", data_source = \"s3\")\n\n# Listing the files in GCS\nlist_files(path = \"test\", pattern = \"*csv\")\n\n# Saving mtcars to all the data sources using default function write.csv\nexport_table(mtcars, \"~/Downloads/mtcars.csv\", data_source = \"local\")\nexport_table(mtcars, \"test/mtcars.csv\") # saving to GCS, need not mention as set globally\nexport_table(mtcars, \"test/mtcars.csv\", data_source = \"s3\")\n\n# Check if the file written exists in GCS\nfile_exists(\"test/mtcars.csv\")\n\n# Read the file from GCS using readr library\nmtcars \u003c- import_table(\"test/mtcars.csv\", FUN = readr::read_csv)\n\n```\n\n## References\n* Cloudyr GCS wrapper: https://github.com/cloudyr/googleCloudStorageR\n* Cloudyr S3 wrapper: https://github.com/cloudyr/aws.s3\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n\n\u003cimg src=\"https://user-images.githubusercontent.com/408863/66741678-a78ab780-ee93-11e9-8d90-b274af222339.png\" align=\"centre\" /\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatlanhq%2Fflyio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fatlanhq%2Fflyio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatlanhq%2Fflyio/lists"}