{"id":21533432,"url":"https://github.com/akawashiro/ros3fs","last_synced_at":"2025-04-28T16:13:22.813Z","repository":{"id":150706691,"uuid":"609419478","full_name":"akawashiro/ros3fs","owner":"akawashiro","description":"ros3fs is a Linux FUSE adapter for AWS S3 and S3 compatible object storages.","archived":false,"fork":false,"pushed_at":"2023-10-19T12:35:08.000Z","size":155,"stargazers_count":14,"open_issues_count":5,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-28T16:13:11.133Z","etag":null,"topics":["aws","filesystem","fuse-filesystem","linux","s3"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akawashiro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-03-04T05:34:11.000Z","updated_at":"2024-02-29T01:56:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"78e4dc6a-b969-4e8c-851a-a5add2cbe802","html_url":"https://github.com/akawashiro/ros3fs","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akawashiro%2Fros3fs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akawashiro%2Fros3fs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akawashiro%2Fros3fs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akawashiro%2Fros3fs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akawashiro","download_url":"https://codeload.github.com/akawashiro/ros3fs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251342725,"owners_count":21574245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","filesystem","fuse-filesystem","linux","s3"],"created_at":"2024-11-24T02:30:33.107Z","updated_at":"2025-04-28T16:13:22.784Z","avatar_url":"https://github.com/akawashiro.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ros3fs\nros3fs, Read Only S3 File System, is a Linux FUSE adapter for AWS S3 and S3\ncompatible object storage such as Apache Ozone. ros3fs focuses on speed, only speed.\n\n## Motivation\nAlthough Object storage systems such as AWS S3 or Apache Ozone cluster are\nhandy, we want to handle them as regular files sometimes. Linux FUSE enables us\nto do so, and there are some works to mount S3-like systems such as\n[s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) or [mountpoint-s3](https://github.com/awslabs/mountpoint-s3).\n\nHowever, they have a critical problem. Listing files is very slow, such as\n`ls` or `find`. This is because we must access S3-like systems every time\nreading directory. I developed `ros3fs` to resolve this problem.\n\nThe most important insight for S3-like systems is we update data on S3-like\nsystems infrequently. For example, I am using an S3-like system to back up.\nAlthough I check the contents of the backup frequently, I back up only once or\ntwice a month.\n\nUnder such a situation, we can cache everything locally, especially the\ndirectory structure. Caching this information lets us speed up searching and\nlisting files in our storage. Of course, we must invalidate cache data\nmanually, but it is fine if you update your data infrequently.\n\n## Important notice\nros3fs caches everything locally. This means the changes are not reflected if\nyou update your data in your bucket after ros3fs mounts the bucket. In other\nwords, ros3fs is suitable for accessing immutable data such as backup data,\nmachine learning training data, or log data.\n\n## How to use\n### Required environment\n- Linux\n    - I confirmed on Ubuntu 22.04 but ros3fs must work on other distributions.\n- Windows\n    - I confirmed on WSL2.\n- Mac OS\n    - I haven't confirmed but ros3fs should work on Docker for Mac.\n\n### Build and install\nNote: Use [CMAKE_INSTALL_PREFIX](https://cmake.org/cmake/help/v3.0/variable/CMAKE_INSTALL_PREFIX.html) to change the install destination.\n```\n# Install equivalent packages on Linux distributions other than Ubuntu.\n$ sudo apt-get install -y cmake g++ git libfuse3-dev ninja-build zlib1g-dev libcurl4-openssl-dev libssl-dev ccache pkg-config\n$ git clone https://github.com/akawashiro/ros3fs.git\n$ cd ros3fs\n$ mkdir build\n$ ./build-aws-sdk-cpp.sh ./build\n$ cmake -S . -B build\n$ cmake --build build -- -j\n$ cmake --build build -- install\n```\n\n### Mount your bucket\n```\n# Note: You need '=' in the command\n# For example, ros3fs ./build/ros3fs_mountpoint -f -d -s --endpoint=http://localhost:9878 --bucket_name=bucket1/ --cache_dir=./build/ros3fs_cache_dir\n$ ros3fs \u003cMOUNTPOINT\u003e -f -d --endpoint=\u003cENDPOINT URL\u003e --bucket_name=\u003cBUCKET NAME ENDS WITH '/'\u003e --cache_dir=\u003cCACHE DIRECTORY\u003e\n```\n\n### Other options\n```\n$ ros3fs --help\nusage: ./build/ros3fs [options] \u003cmountpoint\u003e\nExample: ./build/ros3fs example_mountpoint_dir -f -d --endpoint=http://localhost:9878 \\\n         --bucket_name=example_bucket/ --cache_dir=example_cache_dir\n\nros3fs specific options. '=' is mandatory.:\n--endpoint=URL         S3 endpoint (required)\n--bucket_name=NAME     S3 bucket name (required)\n--cache_dir=PATH       Cache directory (required)\n--clear_cache          Clear cache files (optional)\n--update_seconds=SECS  Update period seconds (optional)\n                       Default value is 3600\n--list_max_keys=KEYS   The number of keys fetched in one request (optional)\n                       Default value is 1000\n\nFUSE specific options:\n-d, -odebug\n    Causes debug information for subsequent FUSE library calls\n    to be output to stderr. Implies -f.\n-f\n    If this is specified then fg will be set to 1 on success. \n    This flag indicates that the file system should not detach from \n    the controlling terminal and run in the foreground.\n-h, --help, -ho\n    Print usage information for the options supported by \n    fuse_parse_cmdline().\n-s\n    If this is specified then mt will be set to 0 on success. \n    This flag indicates that the file system should be run \n    in multi-threaded mode. -s is currently ignored and mt will always be 0.\n```\n\n### Develop using local Ozone cluster using Docker\nFirst, install [AWS CLI](https://docs.aws.amazon.com/ja_jp/cli/latest/userguide/getting-started-install.html).\n\nIn a terminal,\n```\n$ ./launch-ozone.sh\n```\n\nIn another terminal,\n```\n$ ./create-files.sh\n$ ./mount-ros3fs.sh\n```\n\nIn the third terminal,\n```\n$ ls build/ros3fs_mountpoint\ndir_a/  testfile_a  testfile_b  testfile_c\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakawashiro%2Fros3fs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakawashiro%2Fros3fs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakawashiro%2Fros3fs/lists"}