{"id":16366779,"url":"https://github.com/centminmod/centminmod-juicefs","last_synced_at":"2025-08-28T17:27:04.796Z","repository":{"id":37473339,"uuid":"496088324","full_name":"centminmod/centminmod-juicefs","owner":"centminmod","description":"JuiceFS file system for S3 mount storage","archived":false,"fork":false,"pushed_at":"2023-05-28T07:48:50.000Z","size":104,"stargazers_count":25,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-12T02:47:43.500Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/centminmod.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-25T04:59:58.000Z","updated_at":"2024-07-09T21:09:18.000Z","dependencies_parsed_at":"2023-01-17T14:45:29.579Z","dependency_job_id":null,"html_url":"https://github.com/centminmod/centminmod-juicefs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centminmod%2Fcentminmod-juicefs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centminmod%2Fcentminmod-juicefs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centminmod%2Fcentminmod-juicefs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centminmod%2Fcentminmod-juicefs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/centminmod","download_url":"https://codeload.github.com/centminmod/centminmod-juicefs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221842712,"owners_count":16890186,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T02:47:32.489Z","updated_at":"2024-10-28T14:46:12.959Z","avatar_url":"https://github.com/centminmod.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JuiceFS Setup\n\nInstalling [JuiceFS](https://juicefs.com/docs/community/introduction/) high performanced POSIX compatible shared file system on Centmin Mod LEMP stack using [JuiceFS caching](https://juicefs.com/docs/community/cache_management) with [Cloudflare R2](https://blog.cloudflare.com/r2-open-beta/) - S3 compatible object storage and local sqlite3 Metadata Engine. Check out JuiceFS Github discussion forum https://github.com/juicedata/juicefs/discussions if you have questions.\n\nJuiceFS implements an architecture that seperates \"data\" and \"metadata\" storage. When using JuiceFS to store data, the data itself is persisted in [object storage](https://juicefs.com/docs/community/how_to_setup_object_storage/) (e.g., Amazon S3, OpenStack Swift, Ceph, Azure Blob or MinIO), and the corresponding metadata can be persisted in various databases ([Metadata Engines](https://juicefs.com/docs/community/databases_for_metadata/)) such as Redis, Amazon MemoryDB, MariaDB, MySQL, TiKV, etcd, SQLite, KeyDB, PostgreSQL, BadgerDB, or FoundationDB.\n\nFrom https://juicefs.com/en/blog/usage-tips/juicefs-24-qas-for-beginners\n\n**How is the performance of JuiceFS?**\n\nJuiceFS is a distributed file system. The latency of metadata is determined by 1 to 2 network round trip(s) between the mount point and metadata service (generally 1-3 ms), and the latency of data depends on the object storage latency (generally 20-100ms). The throughput of sequential read and write can reachup to 2800 MiB/s (see Benchmark with fio), depending on the network bandwidth and whether the data can be easily compressed.\n\nJuiceFS has a built-in multi-level cache (invalidated automatically). Once the cache is warmed up, latency and throughput can be very close to a local file system (although the use of FUSE may bring a small amount of overhead).\n\n# Table Of Contents\n\n* [Install JuiceFS binary](#install-juicefs-binary)\n* [Upgrade JuiceFS binary](#upgrade-juicefs-binary)\n* [Setup JuiceFS logrotation](#setup-juicefs-logrotation)\n* [Format Cloudflare R2 S3 Storage](#format-cloudflare-r2-s3-storage)\n* [Mount the JuiceFS Formatted R2 S3 Storage](#mount-the-juicefs-formatted-r2-s3-storage)\n * [Manual Mount](#manual-mount)\n * [systemd service Mount](#systemd-service-mount)\n* [Setup JuiceFS S3 Gateway](#setup-juicefs-s3-gateway)\n * [Manually Starting JuiceFS S3 Gateway](#manually-starting-juicefs-s3-gateway)\n * [systemd service Starting JuiceFS S3 Gateway](#systemd-service-starting-juicefs-s3-gateway)\n* [Working With Cloudflare R2 S3 Mounted Directory and JuiceFS S3 Gateway](#working-with-cloudflare-r2-s3-mounted-directory-and-juicefs-s3-gateway)\n * [Mount Info](#mount-info)\n * [Inspecting JuiceFS metadata engine status](#inspecting-juicefs-metadata-engine-status)\n * [Warmup Local Cache](#warmup-local-cache)\n * [Check Disk Size](#check-disk-size)\n* [JuiceFS Benchmarks](#juicefs-benchmarks)\n * [Redis Metadata Cache + Sharded R2 Mount On Intel Xeon E-2276G 6C/12T, 32GB memory and 2x 960GB NVMe raid 1](#redis-metadata-cache--sharded-r2-mount-on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1)\n * [JuiceFS Benchmarks 100x R2 Sharded Mount + Redis Metadata Caching](#juicefs-benchmarks-100x-r2-sharded-mount--redis-metadata-caching)\n * [JuiceFS Benchmarks 61x R2 Sharded Mount + Redis Metadata Caching](#juicefs-benchmarks-61x-r2-sharded-mount--redis-metadata-caching)\n * [JuiceFS Benchmarks 21x R2 Sharded Mount + Redis Metadata Caching](#juicefs-benchmarks-21x-r2-sharded-mount--redis-metadata-caching)\n * [JuiceFS Benchmarks 10x R2 Sharded Mount + Redis Metadata Caching](#juicefs-benchmarks-10x-r2-sharded-mount--redis-metadata-caching)\n * [Sharded R2 Mount On Intel Xeon E-2276G 6C/12T, 32GB memory and 2x 960GB NVMe raid 1](#sharded-r2-mount-on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1)\n * [10x Cloudflare R2 sharded JuiceFS mount](#10x-r2-sharded-juicefs-mount)\n * [5x Cloudflare R2 sharded JuiceFS mount](#5x-r2-sharded-juicefs-mount)\n * [On Intel Xeon E-2276G 6C/12T, 32GB memory and 2x 960GB NVMe raid 1](#on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1)\n * [with R2 bucket created with location hint North American East](#with-r2-bucket-created-with-location-hint-north-american-east)\n * [with R2 bucket created with location hint North American West](#with-r2-bucket-created-with-location-hint-north-american-west)\n * [with R2 bucket created on server](#with-r2-bucket-created-on-server)\n * [File copy tests](#file-copy-tests)\n * [fio test for E-2276G server](#fio-test-for-e-2276g-server)\n * [On Intel Core i7 4790K 4C/8T, 32GB memory and 2x 240GB SSD raid 1](#on-intel-core-i7-4790k-4c8t-32gb-memory-and-2x-240gb-ssd-raid-1)\n * [fio tests](#fio-test)\n* [Destroying JuiceFS Filesystem](#destroying-juicefs-filesystem)\n* [Backup JuiceFS Metadata Script](#backup-juicefs-metadata-script)\n * [JuiceFS Backup Metadata Cronjob](#juicefs-backup-metadata-cronjob)\n\n# Install JuiceFS binary\n\n```\ncd /svr-setup\n\nJFS_LATEST_TAG=$(curl -s https://api.github.com/repos/juicedata/juicefs/releases/latest | grep 'tag_name' | cut -d '\"' -f 4 | tr -d 'v')\n\nwget \"https://github.com/juicedata/juicefs/releases/download/v${JFS_LATEST_TAG}/juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz\" -O juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz\n\ntar -zxf \"juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz\"\n\ninstall juicefs /usr/local/bin\n\\cp -af /usr/local/bin/juicefs /sbin/mount.juicefs\n```\n```\njuicefs -V\njuicefs version 1.0.0-beta3+2022-05-05.0fb9155\n```\n```\njuicefs --help\nNAME:\n juicefs - A POSIX file system built on Redis and object storage.\n\nUSAGE:\n juicefs [global options] command [command options] [arguments...]\n\nVERSION:\n 1.0.0-beta3+2022-05-05.0fb9155\n\nCOMMANDS:\n ADMIN:\n format Format a volume\n config Change configuration of a volume\n destroy Destroy an existing volume\n gc Garbage collector of objects in data storage\n fsck Check consistency of a volume\n dump Dump metadata into a JSON file\n load Load metadata from a previously dumped JSON file\n INSPECTOR:\n status Show status of a volume\n stats Show real time performance statistics of JuiceFS\n profile Show profiling of operations completed in JuiceFS\n info Show internal information of a path or inode\n SERVICE:\n mount Mount a volume\n umount Unmount a volume\n gateway Start an S3-compatible gateway\n webdav Start a WebDAV server\n TOOL:\n bench Run benchmark on a path\n warmup Build cache for target directories/files\n rmr Remove directories recursively\n sync Sync between two storages\n\nGLOBAL OPTIONS:\n --verbose, --debug, -v enable debug log (default: false)\n --quiet, -q only warning and errors (default: false)\n --trace enable trace log (default: false)\n --no-agent disable pprof (:6060) and gops (:6070) agent (default: false)\n --no-color disable colors (default: false)\n --help, -h show help (default: false)\n --version, -V print only the version (default: false)\n\nCOPYRIGHT:\n Apache License 2.0\n```\n\n# Upgrade JuiceFS Binary\n\nFollowing instructions for upgrading JuiceFS client [here](https://github.com/juicedata/juicefs/blob/main/docs/en/faq.md#how-to-upgrade-juicefs-client) involves:\n\n1. Unmounting the JuiceFS mount. If you setup using [systemd JuiceFS service file](#mount-the-juicefs-formatted-r2-s3-storage), then it's just a service stop for it and the [JuiceFS S3 Gateway service](#systemd-service-starting-juicefs-s3-gateway).\n\nUpgrading to [JuiceFS v1.0.0](https://github.com/juicedata/juicefs/releases/tag/v1.0.0):\n\n```\nsystemctl stop juicefs.service juicefs-gateway.service\n```\n\n2. Updating JuiceFS binary\n\n```\ncd /svr-setup\n\nJFS_LATEST_TAG=$(curl -s https://api.github.com/repos/juicedata/juicefs/releases/latest | grep 'tag_name' | cut -d '\"' -f 4 | tr -d 'v')\n\nwget \"https://github.com/juicedata/juicefs/releases/download/v${JFS_LATEST_TAG}/juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz\" -O juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz\n\ntar -zxf \"juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz\"\n\ninstall juicefs /usr/local/bin\n\\cp -af /usr/local/bin/juicefs /sbin/mount.juicefs\n```\n\n3. Starting JuiceFS and JuiceFS S3 Gateway services\n\n```\nsystemctl start juicefs.service juicefs-gateway.service\nsystemctl status juicefs.service juicefs-gateway.service --no-pager\n```\n\n4. Checking updated JuiceFS binary and mount.\n\n```\njuicefs -V\njuicefs version 1.0.4+2023-04-06.f1c475d\n\ndf -hT /home/juicefs_mount\nFilesystem Type Size Used Avail Use% Mounted on\nJuiceFS:myjuicefs fuse.juicefs 1.0P 0 1.0P 0% /home/juicefs_mount\n```\n\n```\nmkdir -p /home/juicefs\ncd /home/juicefs\n\njuicefs status sqlite3:///home/juicefs/myjuicefs.db\n2022/06/21 13:54:45.570232 juicefs[28472] \u003cINFO\u003e: Meta address: sqlite3:///home/juicefs/myjuicefs.db [interface.go:397]\n{\n \"Setting\": {\n \"Name\": \"myjuicefs\",\n \"UUID\": \"2109366a-5f4f-4449-8723-dfec21f48e8f\",\n \"Storage\": \"s3\",\n \"Bucket\": \"https://juicefs.cfaccountid.r2.cloudflarestorage.com\",\n \"AccessKey\": \"cfaccesskey\",\n \"SecretKey\": \"removed\",\n \"BlockSize\": 4096,\n \"Compression\": \"none\",\n \"Shards\": 0,\n \"HashPrefix\": false,\n \"Capacity\": 0,\n \"Inodes\": 0,\n \"KeyEncrypted\": true,\n \"TrashDays\": 0,\n \"MetaVersion\": 1,\n \"MinClientVersion\": \"\",\n \"MaxClientVersion\": \"\"\n },\n \"Sessions\": [\n {\n \"Sid\": 19,\n \"Expire\": \"2022-08-12T12:58:32Z\",\n \"Version\": \"1.0.4+2023-04-06.f1c475d\",\n \"HostName\": \"host.domain.com\",\n \"MountPoint\": \"/home/juicefs_mount\",\n \"ProcessID\": 28376\n },\n {\n \"Sid\": 20,\n \"Expire\": \"2022-08-12T12:58:32Z\",\n \"Version\": \"1.0.4+2023-04-06.f1c475d\",\n \"HostName\": \"host.domain.com\",\n \"MountPoint\": \"s3gateway\",\n \"ProcessID\": 28387\n }\n ]\n}\n```\n\n## Setup JuiceFS logrotation\n\n```\ncat \u003e \"/etc/logrotate.d/juicefs\" \u003c\u003cEND\n/var/log/juicefs.log {\n daily\n dateext\n missingok\n rotate 10\n maxsize 500M\n compress\n delaycompress\n notifempty\n}\nEND\n```\n```\nlogrotate -d /etc/logrotate.d/juicefs\nreading config file /etc/logrotate.d/juicefs\nAllocating hash table for state file, size 15360 B\n\nHandling 1 logs\n\nrotating pattern: /var/log/juicefs.log after 1 days (10 rotations)\nempty log files are not rotated, log files \u003e= 524288000 are rotated earlier, old logs are removed\nconsidering log /var/log/juicefs.log\n log does not need rotating (log has been rotated at 2022-5-25 3:0, that is not day ago yet)\n```\n\n# Format Cloudflare R2 S3 Storage\n\nFill in variables for your Cloudflare account id, R2 bucket access key and secret key and the R2 bucket name - create the R2 bucket before hand. The sqlite3 database will be saved at `/home/juicefs/myjuicefs.db`. \n\n* JuiceFS supports compression algorithms which can be enabled via `--compress` option which have 3 available options - lz4, zstd or none (default).\n* `--trash-days` - number of days after which removed files will be permanently deleted. Default = 1.\n* `--block-size` - size of block in KiB\n* Other various format options listed at https://juicefs.com/docs/community/command_reference#options.\n\n```\ncfaccountid='CF_ACCOUNT_ID'\ncfaccesskey=''\ncfsecretkey=''\ncfbucketname='juicefs'\n\nmkdir -p /home/juicefs\ncd /home/juicefs\n\njuicefs format --storage s3 \\\n --bucket https://${cfbucketname}.${cfaccountid}.r2.cloudflarestorage.com \\\n --access-key $cfaccesskey \\\n --secret-key $cfsecretkey \\\n --compress none \\\n --trash-days 0 \\\n --block-size 4096 \\\n sqlite3:///home/juicefs/myjuicefs.db myjuicefs\n```\n\n# Mount the JuiceFS Formatted R2 S3 Storage\n\nCreate the mount directory and cache directories.\n\n```\nmkdir -p /home/juicefs_mount /home/juicefs_cache\n```\n\n## Manual Mount\n\nThere are additional JuiceFS mounting options outlined at https://juicefs.com/docs/community/command_reference#options-1\n\nManually mount the R2 S3 storage at `/home/juicefs_mount`. Note for Cloudflare R2, you need to [disable automatica metadata backups](https://juicefs.com/docs/community/how_to_setup_object_storage/#r2) with `--backup-meta 0`. Otherwise, for non-R2 S3 providers, you can set `--backup-meta 1h`.\n\n```\njuicefs mount sqlite3:///home/juicefs/myjuicefs.db /home/juicefs_mount \\\n--cache-dir /home/juicefs_cache \\\n--cache-size 102400 \\\n--buffer-size 2048 \\\n--open-cache 0 \\\n--attr-cache 1 \\\n--entry-cache 1 \\\n--dir-entry-cache 1 \\\n--cache-partial-only false \\\n--free-space-ratio 0.1 \\\n--writeback \\\n--no-usage-report \\\n--max-uploads 20 \\\n--max-deletes 10 \\\n--backup-meta 0 \\\n--log /var/log/juicefs.log \\\n--get-timeout 300 \\\n--put-timeout 900 \\\n--io-retries 90 \\\n--prefetch 1 -d\n```\n\nNote: As `--backup-meta 0` is set for Cloudflare R2 to disable automatic metadata backups, you can manually run the backup command to backup to a file i.e. `meta-dump.json`:\n\n```\njuicefs dump sqlite3:///home/juicefs/myjuicefs.db meta-dump.json\n```\n\n## systemd service Mount\n\nOr instead of manually mounting, setup systemd service file to manage mounting and unmounting the directory\n\n`/usr/lib/systemd/system/juicefs.service`\n\n```\n[Unit]\nDescription=JuiceFS\nAssertPathIsDirectory=/home/juicefs_mount\nAfter=network-online.target\n\n[Service]\nType=simple\nWorkingDirectory=/home/juicefs\nExecStart=/usr/local/bin/juicefs mount \\\n \"sqlite3:///home/juicefs/myjuicefs.db\" \\\n /home/juicefs_mount \\\n --no-usage-report \\\n --writeback \\\n --cache-size 102400 \\\n --cache-dir /home/juicefs_cache \\\n --buffer-size 2048 \\\n --open-cache 0 \\\n --attr-cache 1 \\\n --entry-cache 1 \\\n --dir-entry-cache 1 \\\n --cache-partial-only false \\\n --free-space-ratio 0.1 \\\n --max-uploads 20 \\\n --max-deletes 10 \\\n --backup-meta 0 \\\n --log /var/log/juicefs.log \\\n --get-timeout 300 \\\n --put-timeout 900 \\\n --io-retries 90 \\\n --prefetch 1\n\nExecStop=/usr/local/bin/juicefs umount /home/juicefs_mount\nRestart=always\nRestartSec=5\n\n[Install]\nWantedBy=default.target\n```\n```\nmkdir -p /etc/systemd/system/juicefs.service.d\n\ncat \u003e \"/etc/systemd/system/juicefs.service.d/openfileslimit.conf\" \u003c\u003cTDG\n[Service]\nLimitNOFILE=524288\nTDG\n```\n\n```\nsystemctl daemon-reload\nsystemctl start juicefs\nsystemctl enable juicefs\nsystemctl status juicefs\njournalctl -u juicefs --no-pager | tail -50\n```\n\n```\nsystemctl status juicefs | sed -e \"s|$(hostname)|hostname|g\"\n● juicefs.service - JuiceFS\n Loaded: loaded (/usr/lib/systemd/system/juicefs.service; enabled; vendor preset: disabled)\n Drop-In: /etc/systemd/system/juicefs.service.d\n └─openfileslimit.conf\n Active: active (running) since Wed 2022-05-25 04:26:33 UTC; 14min ago\n Process: 25661 ExecStop=/usr/local/bin/juicefs umount /home/juicefs_mount (code=exited, status=0/SUCCESS)\n Main PID: 26947 (juicefs)\n Tasks: 17\n Memory: 18.7M\n CGroup: /system.slice/juicefs.service\n └─26947 /usr/local/bin/juicefs mount sqlite3:///home/juicefs/myjuicefs.db /home/juicefs_mount --no-usage-report --writeback --cache-size 102400 --cache-dir /home/juicefs_cache --free-space-ratio 0.1 --max-uploads 20 --max-deletes 10 --backup-meta 0 --log /var/log/juicefs.log - \n\nMay 25 04:26:33 hostname systemd[1]: Started JuiceFS.\nMay 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.125185 juicefs[26947] \u003cINFO\u003e: Meta address: sqlite3:///home/juicefs/myjuicefs.db [interface.go:385]\nMay 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.126772 juicefs[26947] \u003cINFO\u003e: Data use s3://juicefs/myjuicefs/ [mount.go:289]\nMay 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.127088 juicefs[26947] \u003cINFO\u003e: Disk cache (/home/juicefs_cache/3c874e07-a62c-42a9-ae67-5865491dd4a8/): capacity (102400 MB), free ratio (10%), max pending pages (51) [disk_cache.go:90]\nMay 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.138212 juicefs[26947] \u003cINFO\u003e: create session 1 OK [base.go:185]\nMay 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.138802 juicefs[26947] \u003cINFO\u003e: Prometheus metrics listening on 127.0.0.1:9567 [mount.go:157]\nMay 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.138890 juicefs[26947] \u003cINFO\u003e: Mounting volume myjuicefs at /home/juicefs_mount ... [mount_unix.go:177]\nMay 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.628570 juicefs[26947] \u003cINFO\u003e: OK, myjuicefs is ready at /home/juicefs_mount [mount_unix.go:45]\nMay 25 04:32:19 hostname juicefs[26947]: 2022/05/25 04:32:19.284310 juicefs[26947] \u003cWARNING\u003e: Secret key is removed for the sake of safety [sql.go:2770]\nMay 25 04:32:20 hostname juicefs[26947]: 2022/05/25 04:32:20.804652 juicefs[26947] \u003cINFO\u003e: backup metadata succeed, used 1.527736137s [backup.go:69]\n```\n\nUsing AWS CLI profile for r2 user to check underlying JuiceFS metadata:\n\n```\ncfbucketname='juicefs'\n\naws s3 ls --recursive --profile r2 --endpoint-url=$url s3://$cfbucketname/\n2022-05-25 04:26:25 36 myjuicefs/juicefs_uuid\n2022-05-25 04:32:20 598 myjuicefs/meta/dump-2022-05-25-043219.json.gz\n```\n\n# Setup JuiceFS S3 Gateway\n\nSetup [JuiceFS S3 Gateway](https://juicefs.com/docs/community/s3_gateway#use-the-aws-cli) and setup AWS CLI profile `juicefs` using my [awscli-get.sh](https://awscli-get.centminmod.com/) script to configure the profile.\n\nInstall `awscli-get.sh`:\n\n```\ncurl -4s https://awscli-get.centminmod.com/awscli-get.sh -o awscli-get.sh\nchmod +x awscli-get.sh\n```\n\nChange `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD` variable values to your descired S3 Gateway access and secret keys. Make sure they're different from your Cloudflare R2 access and secret key credentials.\n\nSetup AWS CLI profile using `awscli-get.sh`:\n\n```\nexport MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE\nexport MINIO_ROOT_PASSWORD=12345678\n\n# https://awscli-get.centminmod.com/\nexport AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER\nexport AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD\nexport AWS_DEFAULT_REGION=auto\nexport AWS_DEFAULT_OUTPUT=text\n./awscli-get.sh install juicefs\n```\n\nExample output from [awscli-get.sh](https://awscli-get.centminmod.com/) script installing AWS CLI profile named `juicefs`:\n\n```\n./awscli-get.sh install juicefs\n\nexisting config file detected: /root/.aws/config\nexisting credential file detected: /root/.aws/credentials\n\nconfigure aws-cli profile: juicefs\nconfigure aws cli for Cloudflare R2\naws configure set s3.max_concurrent_requests 2 --profile juicefs\naws configure set s3.multipart_threshold 50MB --profile juicefs\naws configure set s3.multipart_chunksize 50MB --profile juicefs\naws configure set s3.addressing_style path --profile juicefs\n\naws-cli profile: juicefs set:\n\naws_access_key_id: AKIAIOSFODNN7EXAMPLE\naws_secret_access_key: 12345678\ndefault.region: auto\ndefault output format: text\n\nlist aws-cli profiles:\n\ndefault\nr2\njuicefs\n```\n\n## Manually Starting JuiceFS S3 Gateway\n\nManually starting created JuiceFS S3 Gateway. Note for Cloudflare R2, you need to [disable automatica metadata backups](https://juicefs.com/docs/community/how_to_setup_object_storage/#r2) with `--backup-meta 0`. Otherwise, for non-R2 S3 providers, you can set `--backup-meta 1h`.\n\nPrivate local access only:\n\n```\n# local private access\njuicefs gateway \\\n--cache-dir /home/juicefs_cache \\\n--cache-size 102400 \\\n--attr-cache 1 \\\n--entry-cache 0 \\\n--dir-entry-cache 1 \\\n--prefetch 1 \\\n--free-space-ratio 0.1 \\\n--writeback \\\n--backup-meta 0 \\\n--no-usage-report \\\n--buffer-size 2048 sqlite3:///home/juicefs/myjuicefs.db localhost:3777\n```\n\nPublic net accessible mode:\n\n```\n# public access\njuicefs gateway \\\n--cache-dir /home/juicefs_cache \\\n--cache-size 102400 \\\n--attr-cache 1 \\\n--entry-cache 0 \\\n--dir-entry-cache 1 \\\n--prefetch 1 \\\n--free-space-ratio 0.1 \\\n--writeback \\\n--backup-meta 0 \\\n--no-usage-report \\\n--buffer-size 2048 sqlite3:///home/juicefs/myjuicefs.db 0.0.0.0:3777\n```\n\n## systemd service Starting JuiceFS S3 Gateway\n\nOr instead of manually creating JuiceFS S3 Gateway, use systemd service file.\n\nBelow is using private local access only.\n\n`/usr/lib/systemd/system/juicefs-gateway.service`\n\n```\n[Unit]\nDescription=JuiceFS Gateway\nAfter=network-online.target\n\n[Service]\nEnvironment='MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE'\nEnvironment='MINIO_ROOT_PASSWORD=12345678'\nType=simple\nWorkingDirectory=/home/juicefs\nExecStart=/usr/local/bin/juicefs gateway \\\n --no-usage-report \\\n --writeback \\\n --cache-size 102400 \\\n --cache-dir /home/juicefs_cache \\\n --attr-cache 1 \\\n --entry-cache 0 \\\n --dir-entry-cache 1 \\\n --prefetch 1 \\\n --free-space-ratio 0.1 \\\n --max-uploads 20 \\\n --max-deletes 10 \\\n --backup-meta 0 \\\n --get-timeout 300 \\\n --put-timeout 900 \\\n --io-retries 90 \\\n --buffer-size 2048 \\\n \"sqlite3:///home/juicefs/myjuicefs.db\" \\\n localhost:3777\n\nRestart=always\nRestartSec=5\n\n[Install]\nWantedBy=default.target\n```\n```\nmkdir -p /etc/systemd/system/juicefs-gateway.service.d\n\ncat \u003e \"/etc/systemd/system/juicefs-gateway.service.d/openfileslimit.conf\" \u003c\u003cTDG\n[Service]\nLimitNOFILE=524288\nTDG\n```\n```\nsystemctl start juicefs-gateway\nsystemctl enable juicefs-gateway\nsystemctl status juicefs-gateway\njournalctl -u juicefs-gateway --no-pager | tail -50\n```\n\n```\nsystemctl status juicefs-gateway | sed -e \"s|$(hostname)|hostname|g\"\n● juicefs-gateway.service - JuiceFS Gateway\n Loaded: loaded (/usr/lib/systemd/system/juicefs-gateway.service; enabled; vendor preset: disabled)\n Drop-In: /etc/systemd/system/juicefs-gateway.service.d\n └─openfileslimit.conf\n Active: active (running) since Wed 2022-05-25 04:26:33 UTC; 18min ago\n Main PID: 26957 (juicefs)\n Tasks: 13\n Memory: 18.3M\n CGroup: /system.slice/juicefs-gateway.service\n └─26957 /usr/local/bin/juicefs gateway --no-usage-report --writeback --cache-size 102400 --cache-dir /home/juicefs_cache --free-space-ratio 0.1 --max-uploads 20 --max-deletes 10 --backup-meta 0 --get-timeout 300 --put-timeout 900 --io-retries 90 --prefetch 1 --bu \n\nMay 25 04:26:33 hostname juicefs[26957]: 2022/05/25 04:26:33.159004 juicefs[26957] \u003cINFO\u003e: Prometheus metrics listening on 127.0.0.1:10037 [mount.go:157]\nMay 25 04:26:33 hostname juicefs[26957]: Endpoint: http://localhost:3777\nMay 25 04:26:33 hostname juicefs[26957]: Browser Access:\nMay 25 04:26:33 hostname juicefs[26957]: http://localhost:3777\nMay 25 04:26:33 hostname juicefs[26957]: Object API (Amazon S3 compatible):\nMay 25 04:26:33 hostname juicefs[26957]: Go: https://docs.min.io/docs/golang-client-quickstart-guide\nMay 25 04:26:33 hostname juicefs[26957]: Java: https://docs.min.io/docs/java-client-quickstart-guide\nMay 25 04:26:33 hostname juicefs[26957]: Python: https://docs.min.io/docs/python-client-quickstart-guide\nMay 25 04:26:33 hostname juicefs[26957]: JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide\nMay 25 04:26:33 hostname juicefs[26957]: .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide\n```\n\n# Working With Cloudflare R2 S3 Mounted Directory and JuiceFS S3 Gateway\n\nUsing AWS CLI `r2` profile to inspect underlying JuiceFS metadata engine data.\n\n```\nurl=${cfaccountid}.r2.cloudflarestorage.com\n\necho 1 \u003e /home/juicefs_mount/file1.txt\n\naws s3 ls --recursive --profile r2 --endpoint-url=$url s3://$cfbucketname/\n\n2022-05-25 04:48:46 2 myjuicefs/chunks/0/0/1_0_2\n2022-05-25 04:26:25 36 myjuicefs/juicefs_uuid\n2022-05-25 04:32:20 598 myjuicefs/meta/dump-2022-05-25-043219.json.gz\n```\n\nUsing AWS CLI `juicefs` profile to inspect the JuiceFS S3 Gateway.\n\n```\naws --endpoint-url http://localhost:3777 s3 ls --recursive myjuicefs\n2022-05-25 04:48:45 2 file1.txt\n```\n\n## Mount Info\n\n```\njuicefs info /home/juicefs_mount/\n/home/juicefs_mount/ :\n inode: 1\n files: 1\n dirs: 1\n length: 2\n size: 8192\n```\n\n## Inspecting JuiceFS metadata engine status\n\n```\njuicefs status sqlite3:///home/juicefs/myjuicefs.db\n2022/05/25 04:50:06.356669 juicefs[33155] \u003cINFO\u003e: Meta address: sqlite3:///home/juicefs/myjuicefs.db [interface.go:385]\n{\n \"Setting\": {\n \"Name\": \"myjuicefs\",\n \"UUID\": \"3c874e07-a62c-42a9-ae67-5865491dd4a8\",\n \"Storage\": \"s3\",\n \"Bucket\": \"https://juicefs.cfaccountid.r2.cloudflarestorage.com\",\n \"AccessKey\": \"cfaccesskey\",\n \"SecretKey\": \"removed\",\n \"BlockSize\": 4096,\n \"Compression\": \"none\",\n \"Shards\": 0,\n \"HashPrefix\": false,\n \"Capacity\": 0,\n \"Inodes\": 0,\n \"KeyEncrypted\": true,\n \"TrashDays\": 1,\n \"MetaVersion\": 1,\n \"MinClientVersion\": \"\",\n \"MaxClientVersion\": \"\"\n },\n \"Sessions\": [\n {\n \"Sid\": 1,\n \"Expire\": \"2022-05-25T04:50:59Z\",\n \"Version\": \"1.0.0-beta3+2022-05-05.0fb9155\",\n \"HostName\": \"host.domain.com\",\n \"MountPoint\": \"/home/juicefs_mount\",\n \"ProcessID\": 26947\n },\n {\n \"Sid\": 2,\n \"Expire\": \"2022-05-25T04:51:03Z\",\n \"Version\": \"1.0.0-beta3+2022-05-05.0fb9155\",\n \"HostName\": \"host.domain.com\",\n \"MountPoint\": \"s3gateway\",\n \"ProcessID\": 26957\n }\n ]\n}\n```\n\n## Warmup Local Cache\n\n```\njuicefs warmup -p 2 /home/juicefs_mount\nWarmed up paths count: 1 / 1 [==============================================================] done \n2022/05/25 05:29:18.497915 juicefs[43684] \u003cINFO\u003e: Successfully warmed up 1 paths [warmup.go:209]\n```\n\n## Check Disk Size\n\n```\ndf -hT /home/juicefs_mount\nFilesystem Type Size Used Avail Use% Mounted on\nJuiceFS:myjuicefs fuse.juicefs 1.0P 4.0K 1.0P 1% /home/juicefs_mount\n```\n\n## Metrics\n\n```\ncurl -s http://localhost:9567/metrics\n```\n\nchecking blockcache metrics\n\n```\ncurl -s http://localhost:9567/metrics | grep blockcache | egrep -v '\\#|hist'\n\njuicefs_blockcache_blocks{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_blockcache_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_blockcache_drops{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_blockcache_evicts{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_blockcache_hit_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.62144e+07\njuicefs_blockcache_hits{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 200\njuicefs_blockcache_miss{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 647\njuicefs_blockcache_miss_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.680160256e+09\njuicefs_blockcache_write_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.173698048e+09\njuicefs_blockcache_writes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 712\n```\n\nfiltered metrics\n\n```\ncurl -s http://localhost:9567/metrics | egrep -v '\\#|hist|bucket'\n\njuicefs_blockcache_blocks{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_blockcache_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_blockcache_drops{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_blockcache_evicts{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_blockcache_hit_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.62144e+07\njuicefs_blockcache_hits{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 200\njuicefs_blockcache_miss{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 647\njuicefs_blockcache_miss_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.680160256e+09\njuicefs_blockcache_write_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.173698048e+09\njuicefs_blockcache_writes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 712\njuicefs_cpu_usage{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 21.072261\njuicefs_fuse_open_handlers{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_fuse_read_size_bytes_sum{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.173698048e+09\njuicefs_fuse_read_size_bytes_count{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 16584\njuicefs_fuse_written_size_bytes_sum{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.173698048e+09\njuicefs_fuse_written_size_bytes_count{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 16584\njuicefs_go_build_info{checksum=\"\",mp=\"/home/juicefs_mount\",path=\"github.com/juicedata/juicefs\",version=\"(devel)\",vol_name=\"myjuicefs\"} 1\njuicefs_go_gc_duration_seconds{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\",quantile=\"0\"} 2.4418e-05\njuicefs_go_gc_duration_seconds{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\",quantile=\"0.25\"} 4.3148e-05\njuicefs_go_gc_duration_seconds{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\",quantile=\"0.5\"} 5.6996e-05\njuicefs_go_gc_duration_seconds{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\",quantile=\"0.75\"} 0.000106379\njuicefs_go_gc_duration_seconds{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\",quantile=\"1\"} 0.000342952\njuicefs_go_gc_duration_seconds_sum{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0.001999786\njuicefs_go_gc_duration_seconds_count{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 22\njuicefs_go_goroutines{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 62\njuicefs_go_info{mp=\"/home/juicefs_mount\",version=\"go1.17.8\",vol_name=\"myjuicefs\"} 1\njuicefs_go_memstats_alloc_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.8662952e+07\njuicefs_go_memstats_alloc_bytes_total{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.377878736e+09\njuicefs_go_memstats_buck_hash_sys_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.537716e+06\njuicefs_go_memstats_frees_total{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 4.703242e+06\njuicefs_go_memstats_gc_cpu_fraction{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.1818653907586683e-05\njuicefs_go_memstats_gc_sys_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 4.8828976e+07\njuicefs_go_memstats_heap_alloc_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.8662952e+07\njuicefs_go_memstats_heap_idle_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.28196608e+09\njuicefs_go_memstats_heap_inuse_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 3.3079296e+07\njuicefs_go_memstats_heap_objects{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 53970\njuicefs_go_memstats_heap_released_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.278754816e+09\njuicefs_go_memstats_heap_sys_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.315045376e+09\njuicefs_go_memstats_last_gc_time_seconds{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.6535426808430629e+09\njuicefs_go_memstats_lookups_total{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_go_memstats_mallocs_total{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 4.757212e+06\njuicefs_go_memstats_mcache_inuse_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 9600\njuicefs_go_memstats_mcache_sys_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 16384\njuicefs_go_memstats_mspan_inuse_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 312256\njuicefs_go_memstats_mspan_sys_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.736128e+06\njuicefs_go_memstats_next_gc_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 5.738088e+07\njuicefs_go_memstats_other_sys_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.769556e+06\njuicefs_go_memstats_stack_inuse_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.0354688e+07\njuicefs_go_memstats_stack_sys_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.0354688e+07\njuicefs_go_memstats_sys_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.380288824e+09\njuicefs_go_threads{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 271\njuicefs_memory{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 9.64608e+07\njuicefs_object_request_data_bytes{method=\"GET\",mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.147483648e+09\njuicefs_object_request_data_bytes{method=\"PUT\",mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.205155328e+09\njuicefs_object_request_errors{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 337\njuicefs_process_cpu_seconds_total{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 21.06\njuicefs_process_max_fds{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 524288\njuicefs_process_open_fds{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 23\njuicefs_process_resident_memory_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 9.64608e+07\njuicefs_process_start_time_seconds{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.65354147984e+09\njuicefs_process_virtual_memory_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 2.159013888e+09\njuicefs_process_virtual_memory_max_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1.8446744073709552e+19\njuicefs_staging_block_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_staging_blocks{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_store_cache_size_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_transaction_restart{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 368\njuicefs_uptime{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 1246.457965465\njuicefs_used_buffer_size_bytes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 7.471104e+06\njuicefs_used_inodes{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\njuicefs_used_space{mp=\"/home/juicefs_mount\",vol_name=\"myjuicefs\"} 0\n```\n\n# JuiceFS Benchmarks\n\n## Sharded R2 Mount On Intel Xeon E-2276G 6C/12T, 32GB memory and 2x 960GB NVMe raid 1\n\nThe server runs on 2x mismatched 960GB NVMe drives in raid 1 so take note of that for the potential peak read and write performance of the resulting benchmarks:\n\n* Samsung SSD PM983 960GB 2.5 U.2 Gen 3.0 x4 PCIe NVMe\n * Up to 3,000MB/s Read, 1,050MB/s Write\n * 4K random read/write 400,000/40,000 IOPS\n * 1366 TBW / 1.3 DWPD\n * Power: 4 Watt (idle) 8.6 Watt (read) 8.1 Watt (write)\n* Kingston DC1500M U.2 Enterprise SSD Gen 3.0 x4 PCIe NVME\n * Up to 3,100MB/s Read, 1,700MB/s Write\n * Steady-state 4k read/write 440,000/150,000 IOPS\n * 1681 TBW (1 DWPD/5yrs) (1.6 DWPD/3yrs)\n * Power: Idle: 6.30W Average read: 6.21W Average write: 11.40W Max read: 6.60W Max write: 12.24W\n\nThe table below shows comparison between [10x Cloudflare R2 sharded JuiceFS mount](#10x-r2-sharded-juicefs-mount) vs [5x Cloudflare R2 sharded JuiceFS mount](#5x-r2-sharded-juicefs-mount) vs [1x Cloudflare JuiceFS mount (default)](#on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1). All R2 storage locations are with location hint North American East.\n\nFor 1024MB big file size\n\n| ITEM | VALUE (10x R2 Sharded) | COST (10x R2 Sharded) | VALUE (5x R2 Sharded) | COST (5x R2 Sharded) | VALUE (1x R2 Default) | COST (1x R2 Default) |\n| --- | --- | --- | --- | --- | --- | --- |\n| Write big file | 906.04 MiB/s | 4.52 s/file | 960.47 MiB/s | 4.26 s/file | 1374.08 MiB/s | 2.98 s/file |\n| Read big file | 223.19 MiB/s | 18.35 s/file | 174.17 MiB/s | 23.52 s/file | 152.23 MiB/s | 26.91 s/file |\n| Write small file | 701.2 files/s | 5.70 ms/file | 777.4 files/s | 5.15 ms/file | 780.3 files/s | 5.13 ms/file |\n| Read small file | 6378.3 files/s | 0.63 ms/file | 7940.0 files/s | 0.50 ms/file | 8000.9 files/s | 0.50 ms/file |\n| Stat file | 21123.7 files/s | 0.19 ms/file | 29344.7 files/s | 0.14 ms/file | 27902.2 files/s | 0.14 ms/file |\n| FUSE operation | 71555 operations | 2.16 ms/op | 71597 operations | 2.67 ms/op | 71649 operations | 3.06 ms/op |\n| Update meta | 6271 operations | 9.01 ms/op | 6041 operations | 4.09 ms/op | 6057 operations | 2.50 ms/op |\n| Put object | 1152 operations | 403.23 ms/op | 1136 operations | 428.27 ms/op | 1106 operations | 547.32 ms/op |\n| Get object | 1034 operations | 278.61 ms/op | 1049 operations | 299.50 ms/op | 1030 operations | 301.80 ms/op |\n| Delete object | 316 operations | 124.32 ms/op | 60 operations | 120.73 ms/op | 29 operations | 234.02 ms/op |\n| Write into cache | 1424 operations | 24.92 ms/op | 1424 operations | 83.12 ms/op | 1424 operations | 12.91 ms/op |\n| Read from cache | 400 operations | 0.05 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.04 ms/op |\n\nFor 1MB big file size\n\n| ITEM | VALUE (10x R2 Sharded) | COST (10x R2 Sharded) | VALUE (5x R2 Sharded) | COST (5x R2 Sharded) | VALUE (1x R2 Default) | COST (1x R2 Default) |\n| --- | --- | --- | --- | --- | --- | --- |\n| Write big file | 452.66 MiB/s | 0.01 s/file | 448.20 MiB/s | 0.01 s/file | 230.82 MiB/s | 0.02 s/file |\n| Read big file | 1545.95 MiB/s | 0.00 s/file | 1376.38 MiB/s | 0.00 s/file | 1276.38 MiB/s | 0.00 s/file |\n| Write small file | 682.8 files/s | 5.86 ms/file | 792.5 files/s | 5.05 ms/file | 675.7 files/s | 5.92 ms/file |\n| Read small file | 6299.4 files/s | 0.63 ms/file | 7827.1 files/s | 0.51 ms/file | 7833.1 files/s | 0.51 ms/file |\n| Stat file | 21365.2 files/s | 0.19 ms/file | 24308.1 files/s | 0.16 ms/file | 28226.1 files/s | 0.14 ms/file |\n| FUSE operation | 5757 operations | 0.42 ms/op | 5750 operations | 0.38 ms/op | 5756 operations | 0.41 ms/op |\n| Update meta | 5814 operations | 0.72 ms/op | 5740 operations | 0.74 ms/op | 5770 operations | 0.70 ms/op |\n| Put object | 107 operations | 282.68 ms/op | 94 operations | 286.35 ms/op | 118 operations | 242.35 ms/op |\n| Get object | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op |\n| Delete object | 133 operations | 116.84 ms/op | 59 operations | 117.93 ms/op | 95 operations | 83.94 ms/op |\n| Write into cache | 404 operations | 0.12 ms/op | 404 operations | 0.12 ms/op | 404 operations | 0.14 ms/op |\n| Read from cache | 408 operations | 0.06 ms/op | 408 operations | 0.05 ms/op | 408 operations | 0.06 ms/op |\n\n\n### 10x R2 Sharded JuiceFS Mount\n\nBenchmark with [`--shard`](https://juicefs.com/docs/community/how_to_setup_object_storage#enable-data-sharding) mount option for [sharded Cloudflare R2 mounted JuiceFS](https://juicefs.com/docs/community/how_to_setup_object_storage#enable-data-sharding) over 10x sharded R2 object storage locations - `juicefs-shard-0`,`juicefs-shard-`,`juicefs-shard-1`,`juicefs-shard-3`, `juicefs-shard-4`, `juicefs-shard-5`, `juicefs-shard-6`, `juicefs-shard-7`, `juicefs-shard-8`, `juicefs-shard-9` with location hint North American East.\n\n```\ncfaccountid='CF_ACCOUNT_ID'\ncfaccesskey=''\ncfsecretkey=''\ncfbucketname='juicefs-shard'\n\nmkdir -p /home/juicefs\ncd /home/juicefs\n\njuicefs format --storage s3 \\\n --shards 10 \\\n --bucket https://${cfbucketname}-%d.${cfaccountid}.r2.cloudflarestorage.com \\\n --access-key $cfaccesskey \\\n --secret-key $cfsecretkey \\\n --compress none \\\n --trash-days 0 \\\n --block-size 4096 \\\n sqlite3:///home/juicefs/myjuicefs.db myjuicefs\n```\n\nJuiceFS 10x sharded Cloudflare R2 benchmark with location hint North American East and 1024MB big file size.\n\n```\njuicefs bench -p 4 /home/juicefs_mount/\n Write big blocks count: 4096 / 4096 [===========================================================] done \n Read big blocks count: 4096 / 4096 [===========================================================] done \nWrite small blocks count: 400 / 400 [=============================================================] done \n Read small blocks count: 400 / 400 [=============================================================] done \n Stat small files count: 400 / 400 [=============================================================] done \nBenchmark finished!\nBlockSize: 1 MiB, BigFileSize: 1024 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 4\nTime used: 25.4 s, CPU: 127.8%, Memory: 1742.7 MiB\n+------------------+------------------+--------------+\n| ITEM | VALUE | COST |\n+------------------+------------------+--------------+\n| Write big file | 906.04 MiB/s | 4.52 s/file |\n| Read big file | 223.19 MiB/s | 18.35 s/file |\n| Write small file | 701.2 files/s | 5.70 ms/file |\n| Read small file | 6378.3 files/s | 0.63 ms/file |\n| Stat file | 21123.7 files/s | 0.19 ms/file |\n| FUSE operation | 71555 operations | 2.16 ms/op |\n| Update meta | 6271 operations | 9.01 ms/op |\n| Put object | 1152 operations | 403.23 ms/op |\n| Get object | 1034 operations | 278.61 ms/op |\n| Delete object | 316 operations | 124.32 ms/op |\n| Write into cache | 1424 operations | 24.92 ms/op |\n| Read from cache | 400 operations | 0.05 ms/op |\n+------------------+------------------+--------------+\n```\n\nJuiceFS 10x sharded Cloudflare R2 benchmark with location hint North American East and 1MB big file size.\n\n```\njuicefs bench -p 4 /home/juicefs_mount/ --big-file-size 1\n Write big blocks count: 4 / 4 [==============================================================] done \n Read big blocks count: 4 / 4 [==============================================================] done \nWrite small blocks count: 400 / 400 [=============================================================] done \n Read small blocks count: 400 / 400 [=============================================================] done \n Stat small files count: 400 / 400 [=============================================================] done \nBenchmark finished!\nBlockSize: 1 MiB, BigFileSize: 1 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 4\nTime used: 1.7 s, CPU: 97.8%, Memory: 1946.3 MiB\n+------------------+-----------------+--------------+\n| ITEM | VALUE | COST |\n+------------------+-----------------+--------------+\n| Write big file | 452.66 MiB/s | 0.01 s/file |\n| Read big file | 1545.95 MiB/s | 0.00 s/file |\n| Write small file | 682.8 files/s | 5.86 ms/file |\n| Read small file | 6299.4 files/s | 0.63 ms/file |\n| Stat file | 21365.2 files/s | 0.19 ms/file |\n| FUSE operation | 5757 operations | 0.42 ms/op |\n| Update meta | 5814 operations | 0.72 ms/op |\n| Put object | 107 operations | 282.68 ms/op |\n| Get object | 0 operations | 0.00 ms/op |\n| Delete object | 133 operations | 116.84 ms/op |\n| Write into cache | 404 operations | 0.12 ms/op |\n| Read from cache | 408 operations | 0.06 ms/op |\n+------------------+-----------------+--------------+\n```\n\nInspecting actual JuiceFS mount\n\n```\nls -lah /home/juicefs_mount\ntotal 9.5K\ndrwxrwxrwx 2 root root 4.0K May 24 21:09 .\ndrwxr-xr-x. 13 root root 4.0K May 21 18:58 ..\n-r-------- 1 root root 0 May 24 21:04 .accesslog\n-r-------- 1 root root 1.3K May 24 21:04 .config\n-r--r--r-- 1 root root 0 May 24 21:04 .stats\ndr-xr-xr-x 2 root root 0 May 24 21:04 .trash\n\n```\n\nJuiceFS mount stats\n\n```\ncat /home/juicefs_mount/.stats \njuicefs_blockcache_blocks 0\njuicefs_blockcache_bytes 0\njuicefs_blockcache_drops 0\njuicefs_blockcache_evicts 0\njuicefs_blockcache_hit_bytes 436207616\njuicefs_blockcache_hits 3232\njuicefs_blockcache_miss 4241\njuicefs_blockcache_miss_bytes 17720934400\njuicefs_blockcache_read_hist_seconds_total 3232\njuicefs_blockcache_read_hist_seconds_sum 0.16809886599999965\njuicefs_blockcache_write_bytes 17616076800\njuicefs_blockcache_write_hist_seconds_total 7312\njuicefs_blockcache_write_hist_seconds_sum 391.8615498160001\njuicefs_blockcache_writes 7312\njuicefs_compact_size_histogram_bytes_total 0\njuicefs_compact_size_histogram_bytes_sum 0\njuicefs_cpu_usage 141.95404200000002\njuicefs_fuse_open_handlers 1\njuicefs_fuse_ops_durations_histogram_seconds_total 310446\njuicefs_fuse_ops_durations_histogram_seconds_sum 897.3332114739511\njuicefs_fuse_read_size_bytes_total 134400\njuicefs_fuse_read_size_bytes_sum 17616076800\njuicefs_fuse_written_size_bytes_total 134400\njuicefs_fuse_written_size_bytes_sum 17616076800\njuicefs_go_build_info__github.com/juicedata/juicefs_(devel) 1\njuicefs_go_goroutines 79\njuicefs_go_info_go1.19.2 1\njuicefs_go_memstats_alloc_bytes 32336136\njuicefs_go_memstats_alloc_bytes_total 2939952080\njuicefs_go_memstats_buck_hash_sys_bytes 1805266\njuicefs_go_memstats_frees_total 35324415\njuicefs_go_memstats_gc_cpu_fraction 0.000014170426577562373\njuicefs_go_memstats_gc_sys_bytes 84006704\njuicefs_go_memstats_heap_alloc_bytes 32336136\njuicefs_go_memstats_heap_idle_bytes 1917329408\njuicefs_go_memstats_heap_inuse_bytes 37642240\njuicefs_go_memstats_heap_objects 61603\njuicefs_go_memstats_heap_released_bytes 1916534784\njuicefs_go_memstats_heap_sys_bytes 1954971648\njuicefs_go_memstats_last_gc_time_seconds 1684981564.030393\njuicefs_go_memstats_lookups_total 0\njuicefs_go_memstats_mallocs_total 35386018\njuicefs_go_memstats_mcache_inuse_bytes 14400\njuicefs_go_memstats_mcache_sys_bytes 15600\njuicefs_go_memstats_mspan_inuse_bytes 357136\njuicefs_go_memstats_mspan_sys_bytes 10918080\njuicefs_go_memstats_next_gc_bytes 64394520\njuicefs_go_memstats_other_sys_bytes 2965030\njuicefs_go_memstats_stack_inuse_bytes 12156928\njuicefs_go_memstats_stack_sys_bytes 12156928\njuicefs_go_memstats_sys_bytes 2066839256\njuicefs_go_threads 278\njuicefs_memory 154210304\njuicefs_meta_ops_durations_histogram_seconds_total 33621\njuicefs_meta_ops_durations_histogram_seconds_sum 304.82784600500264\njuicefs_object_request_data_bytes_GET 17234395136\njuicefs_object_request_data_bytes_PUT 17573871616\njuicefs_object_request_durations_histogram_seconds_DELETE_total 7312\njuicefs_object_request_durations_histogram_seconds_DELETE_sum 894.7742219350012\njuicefs_object_request_durations_histogram_seconds_GET_total 4225\njuicefs_object_request_durations_histogram_seconds_GET_sum 1226.6492657670003\njuicefs_object_request_durations_histogram_seconds_PUT_total 6866\njuicefs_object_request_durations_histogram_seconds_PUT_sum 2722.8829050600116\njuicefs_object_request_errors 499\njuicefs_process_cpu_seconds_total 141.95\njuicefs_process_max_fds 524288\njuicefs_process_open_fds 15\njuicefs_process_resident_memory_bytes 154210304\njuicefs_process_start_time_seconds 1684980243.38\njuicefs_process_virtual_memory_bytes 2880843776\njuicefs_process_virtual_memory_max_bytes 18446744073709552000\njuicefs_staging_block_bytes 0\njuicefs_staging_blocks 0\njuicefs_store_cache_size_bytes 0\njuicefs_transaction_durations_histogram_seconds_total 51442\njuicefs_transaction_durations_histogram_seconds_sum 140.4373622320012\njuicefs_transaction_restart 0\njuicefs_uptime 1416.787316108\njuicefs_used_buffer_size_bytes 8388608\njuicefs_used_inodes 0\njuicefs_used_space 0\n```\n\n### 5x R2 Sharded JuiceFS Mount\n\nBenchmark with [`--shard`](https://juicefs.com/docs/community/how_to_setup_object_storage#enable-data-sharding) mount option for [sharded Cloudflare R2 mounted JuiceFS](https://juicefs.com/docs/community/how_to_setup_object_storage#enable-data-sharding) over 5x sharded R2 object storage locations - `juicefs-shard-0`,`juicefs-shard-`,`juicefs-shard-1`,`juicefs-shard-3`, and `juicefs-shard-4` with location hint North American East.\n\n```\ncfaccountid='CF_ACCOUNT_ID'\ncfaccesskey=''\ncfsecretkey=''\ncfbucketname='juicefs-shard'\n\nmkdir -p /home/juicefs\ncd /home/juicefs\n\njuicefs format --storage s3 \\\n --shards 5 \\\n --bucket https://${cfbucketname}-%d.${cfaccountid}.r2.cloudflarestorage.com \\\n --access-key $cfaccesskey \\\n --secret-key $cfsecretkey \\\n --compress none \\\n --trash-days 0 \\\n --block-size 4096 \\\n sqlite3:///home/juicefs/myjuicefs.db myjuicefs\n```\n\noutput\n\n```\n2023/05/24 17:45:14.116161 juicefs[3701895] \u003cINFO\u003e: Meta address: sqlite3:///home/juicefs/myjuicefs.db [interface.go:401]\n2023/05/24 17:45:14.117248 juicefs[3701895] \u003cINFO\u003e: Data use shard5://s3://juicefs-shard-0/myjuicefs/ [format.go:434]\n2023/05/24 17:45:18.423901 juicefs[3701895] \u003cERROR\u003e: Can't list s3://juicefs-shard-0/: InvalidMaxKeys: MaxKeys params must be positive integer \u003c= 1000.\n status code: 400, request id: , host id: [sharding.go:85]\n2023/05/24 17:45:18.423955 juicefs[3701895] \u003cWARNING\u003e: List storage shard5://s3://juicefs-shard-0/myjuicefs/ failed: list s3://juicefs-shard-0/: InvalidMaxKeys: MaxKeys params must be positive integer \u003c= 1000.\n status code: 400, request id: , host id: [format.go:452]\n2023/05/24 17:45:18.709793 juicefs[3701895] \u003cINFO\u003e: Volume is formatted as {\n \"Name\": \"myjuicefs\",\n \"UUID\": \"UUID-UUID-UUID-UUID\",\n \"Storage\": \"s3\",\n \"Bucket\": \"https://juicefs-shard-%d.CF_ACCOUNT_ID.r2.cloudflarestorage.com\",\n \"AccessKey\": \"CF_ACCESS_KEY\",\n \"SecretKey\": \"removed\",\n \"BlockSize\": 4096,\n \"Compression\": \"none\",\n \"Shards\": 5,\n \"KeyEncrypted\": true,\n \"MetaVersion\": 1\n} [format.go:471]\n```\n\nJuiceFS mount info\n\n```\njuicefs info /home/juicefs_mount/\n/home/juicefs_mount/ :\n inode: 1\n files: 0\n dirs: 1\n length: 0 Bytes\n size: 4.00 KiB (4096 Bytes)\n path: /\n```\n\nJuiceFS 5x sharded Cloudflare R2 benchmark with location hint North American East and 1024MB big file size.\n\n```\njuicefs bench -p 4 /home/juicefs_mount/\n Write big blocks count: 4096 / 4096 [===========================================================] done \n Read big blocks count: 4096 / 4096 [===========================================================] done \nWrite small blocks count: 400 / 400 [=============================================================] done \n Read small blocks count: 400 / 400 [=============================================================] done \n Stat small files count: 400 / 400 [=============================================================] done \nBenchmark finished!\nBlockSize: 1 MiB, BigFileSize: 1024 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 4\nTime used: 30.2 s, CPU: 103.5%, Memory: 1364.5 MiB\n+------------------+------------------+--------------+\n| ITEM | VALUE | COST |\n+------------------+------------------+--------------+\n| Write big file | 960.47 MiB/s | 4.26 s/file |\n| Read big file | 174.17 MiB/s | 23.52 s/file |\n| Write small file | 777.4 files/s | 5.15 ms/file |\n| Read small file | 7940.0 files/s | 0.50 ms/file |\n| Stat file | 29344.7 files/s | 0.14 ms/file |\n| FUSE operation | 71597 operations | 2.67 ms/op |\n| Update meta | 6041 operations | 4.09 ms/op |\n| Put object | 1136 operations | 428.27 ms/op |\n| Get object | 1049 operations | 299.50 ms/op |\n| Delete object | 60 operations | 120.73 ms/op |\n| Write into cache | 1424 operations | 83.12 ms/op |\n| Read from cache | 400 operations | 0.05 ms/op |\n+------------------+------------------+--------------+\n```\n\nJuiceFS sharded Cloudflare R2 benchmark with location hint North American East and 1MB big file size.\n\n```\njuicefs bench -p 4 /home/juicefs_mount/ --big-file-size 1\n Write big blocks count: 4 / 4 [==============================================================] done \n Read big blocks count: 4 / 4 [==============================================================] done \nWrite small blocks count: 400 / 400 [=============================================================] done \n Read small blocks count: 400 / 400 [=============================================================] done \n Stat small files count: 400 / 400 [=============================================================] done \nBenchmark finished!\nBlockSize: 1 MiB, BigFileSize: 1 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 4\nTime used: 1.6 s, CPU: 102.4%, Memory: 164.9 MiB\n+------------------+-----------------+--------------+\n| ITEM | VALUE | COST |\n+------------------+-----------------+--------------+\n| Write big file | 448.20 MiB/s | 0.01 s/file |\n| Read big file | 1376.38 MiB/s | 0.00 s/file |\n| Write small file | 792.5 files/s | 5.05 ms/file |\n| Read small file | 7827.1 files/s | 0.51 ms/file |\n| Stat file | 24308.1 files/s | 0.16 ms/file |\n| FUSE operation | 5750 operations | 0.38 ms/op |\n| Update meta | 5740 operations | 0.74 ms/op |\n| Put object | 94 operations | 286.35 ms/op |\n| Get object | 0 operations | 0.00 ms/op |\n| Delete object | 59 operations | 117.93 ms/op |\n| Write into cache | 404 operations | 0.12 ms/op |\n| Read from cache | 408 operations | 0.05 ms/op |\n+------------------+-----------------+--------------+\n```\n\nInspecting Cloudflare R2 sharded storage buckets after JuiceFS benchmark run with location hint North American East\n\n```\naws s3 ls --recursive --profile r2 --endpoint-url=$url s3://juicefs-shard-0\n2023-05-24 18:46:01 131072 myjuicefs/chunks/0/0/980_0_131072\n2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1146_0_131072\n2023-05-24 18:46:30 131072 myjuicefs/chunks/0/1/1540_0_131072\n\naws s3 ls --recursive --profile r2 --endpoint-url=$url s3://juicefs-shard-1\n2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1154_0_131072\n2023-05-24 18:46:29 131072 myjuicefs/chunks/0/1/1386_0_131072\n2023-05-24 18:46:31 131072 myjuicefs/chunks/0/1/1688_0_131072\n2023-05-24 17:45:18 36 myjuicefs/juicefs_uuid\n\naws s3 ls --recursive --profile r2 --endpoint-url=$url s3://juicefs-shard-2\n2023-05-24 17:52:09 131072 myjuicefs/chunks/0/0/574_0_131072\n2023-05-24 18:46:01 131072 myjuicefs/chunks/0/1/1000_0_131072\n2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1142_0_131072\n\naws s3 ls --recursive --profile r2 --endpoint-url=$url s3://juicefs-shard-3\n2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1130_0_131072\n2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1150_0_131072\n2023-05-24 18:46:05 131072 myjuicefs/chunks/0/1/1226_0_131072\n2023-05-24 18:46:28 131072 myjuicefs/chunks/0/1/1382_0_131072\n2023-05-24 18:46:30 131072 myjuicefs/chunks/0/1/1532_0_131072\n2023-05-24 18:46:30 131072 myjuicefs/chunks/0/1/1552_0_131072\n2023-05-24 18:46:31 131072 myjuicefs/chunks/0/1/1560_0_131072\n2023-05-24 18:46:30 131072 myjuicefs/chunks/0/1/1564_0_131072\n2023-05-24 18:46:31 131072 myjuicefs/chunks/0/1/1568_0_131072\n2023-05-24 18:46:32 131072 myjuicefs/chunks/0/1/1728_0_131072\n2023-05-24 17:53:44 581 myjuicefs/meta/dump-2023-05-24-225343.json.gz\n\naws s3 ls --recursive --profile r2 --endpoint-url=$url s3://juicefs-shard-4\n2023-05-24 18:46:01 131072 myjuicefs/chunks/0/0/988_0_131072\n2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1134_0_131072\n2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1138_0_131072\n2023-05-24 18:46:28 131072 myjuicefs/chunks/0/1/1390_0_131072\n2023-05-24 18:46:28 131072 myjuicefs/chunks/0/1/1394_0_131072\n2023-05-24 18:46:30 131072 myjuicefs/chunks/0/1/1556_0_131072\n```\n\n### fio sharded Cloudflare R2 test for E-2276G server with location hint North American East\n\nfio Sequential Write\n```\nmkdir -p /home/juicefs_mount/fio\n\nfio --name=sequential-write --directory=/home/juicefs_mount/fio --rw=write --refill_buffers --bs=4M --size=1G --end_fsync=1\nsequential-write: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1\nfio-3.19\nStarting 1 process\nsequential-write: Laying out IO file (1 file / 1024MiB)\nJobs: 1 (f=1)\nsequential-write: (groupid=0, jobs=1): err= 0: pid=3704701: Wed May 24 19:01:25 2023\n write: IOPS=279, BW=1119MiB/s (1173MB/s)(1024MiB/915msec); 0 zone resets\n clat (usec): min=2221, max=7356, avg=2961.60, stdev=807.86\n lat (usec): min=2222, max=7357, avg=2962.43, stdev=808.05\n clat percentiles (usec):\n | 1.00th=[ 2245], 5.00th=[ 2311], 10.00th=[ 2376], 20.00th=[ 2442],\n | 30.00th=[ 2540], 40.00th=[ 2638], 50.00th=[ 2704], 60.00th=[ 2802],\n | 70.00th=[ 2966], 80.00th=[ 3163], 90.00th=[ 4424], 95.00th=[ 4948],\n | 99.00th=[ 5735], 99.50th=[ 6718], 99.90th=[ 7373], 99.95th=[ 7373],\n | 99.99th=[ 7373]\n bw ( MiB/s): min= 1067, max= 1067, per=95.35%, avg=1067.08, stdev= 0.00, samples=1\n iops : min= 266, max= 266, avg=266.00, stdev= 0.00, samples=1\n lat (msec) : 4=89.84%, 10=10.16%\n cpu : usr=16.19%, sys=38.95%, ctx=8195, majf=0, minf=9\n IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, \u003e=64=0.0%\n submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n issued rwts: total=0,256,0,0 short=0,0,0,0 dropped=0,0,0,0\n latency : target=0, window=0, percentile=100.00%, depth=1\n\nRun status group 0 (all jobs):\n WRITE: bw=1119MiB/s (1173MB/s), 1119MiB/s-1119MiB/s (1173MB/s-1173MB/s), io=1024MiB (1074MB), run=915-915msec\n```\n```\nls -lah /home/juicefs_mount/fio\ntotal 1.1G\ndrwxr-xr-x 2 root root 4.0K May 24 19:01 .\ndrwxrwxrwx 3 root root 4.0K May 24 19:01 ..\n-rw-r--r-- 1 root root 1.0G May 24 19:01 sequential-write.0.0\n```\n```\njuicefs warmup -p 4 /home/juicefs_mount/fio \nWarming up count: 5 0.06/s \nWarming up bytes: 5.00 GiB (5368709120 Bytes) 57.32 MiB/s \n2023/05/24 19:37:02.236625 juicefs[3705549] \u003cINFO\u003e: Successfully warmed up 5 files (5368709120 bytes) [warmup.go:233]\n```\n\nfio Sequential Read\n\n```\nfio --name=sequential-read --directory=/home/juicefs_mount/fio --rw=read --refill_buffers --bs=4M --size=1G --numjobs=4\nsequential-read: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1\n...\nfio-3.19\nStarting 4 processes\nJobs: 4 (f=4): [R(4)][-.-%][r=2270MiB/s][r=567 IOPS][eta 00m:00s]\nsequential-read: (groupid=0, jobs=1): err= 0: pid=3705616: Wed May 24 19:37:25 2023\n read: IOPS=132, BW=532MiB/s (557MB/s)(1024MiB/1926msec)\n clat (usec): min=2368, max=15013, avg=7167.80, stdev=1697.61\n lat (usec): min=2368, max=15013, avg=7169.52, stdev=1697.67\n clat percentiles (usec):\n | 1.00th=[ 2540], 5.00th=[ 5473], 10.00th=[ 5735], 20.00th=[ 6063],\n | 30.00th=[ 6390], 40.00th=[ 6652], 50.00th=[ 6915], 60.00th=[ 7242],\n | 70.00th=[ 7504], 80.00th=[ 7898], 90.00th=[ 9110], 95.00th=[10421],\n | 99.00th=[13304], 99.50th=[13829], 99.90th=[15008], 99.95th=[15008],\n | 99.99th=[15008]\n bw ( KiB/s): min=457227, max=573440, per=24.57%, avg=534320.67, stdev=66767.53, samples=3\n iops : min= 111, max= 140, avg=130.00, stdev=16.46, samples=3\n lat (msec) : 4=2.34%, 10=92.19%, 20=5.47%\n cpu : usr=0.52%, sys=62.55%, ctx=3056, majf=0, minf=1036\n IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, \u003e=64=0.0%\n submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n issued rwts: total=256,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n latency : target=0, window=0, percentile=100.00%, depth=1\nsequential-read: (groupid=0, jobs=1): err= 0: pid=3705617: Wed May 24 19:37:25 2023\n read: IOPS=132, BW=531MiB/s (557MB/s)(1024MiB/1929msec)\n clat (usec): min=1536, max=18497, avg=7181.80, stdev=1753.73\n lat (usec): min=1536, max=18500, avg=7183.40, stdev=1753.80\n clat percentiles (usec):\n | 1.00th=[ 2343], 5.00th=[ 5211], 10.00th=[ 5669], 20.00th=[ 6063],\n | 30.00th=[ 6456], 40.00th=[ 6718], 50.00th=[ 7046], 60.00th=[ 7373],\n | 70.00th=[ 7701], 80.00th=[ 8225], 90.00th=[ 8979], 95.00th=[10552],\n | 99.00th=[12518], 99.50th=[12649], 99.90th=[18482], 99.95th=[18482],\n | 99.99th=[18482]\n bw ( KiB/s): min=450877, max=572295, per=24.23%, avg=526742.67, stdev=66141.94, samples=3\n iops : min= 110, max= 139, avg=128.33, stdev=15.95, samples=3\n lat (msec) : 2=0.78%, 4=2.34%, 10=91.41%, 20=5.47%\n cpu : usr=0.47%, sys=62.14%, ctx=3051, majf=0, minf=1037\n IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, \u003e=64=0.0%\n submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n issued rwts: total=256,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n latency : target=0, window=0, percentile=100.00%, depth=1\nsequential-read: (groupid=0, jobs=1): err= 0: pid=3705618: Wed May 24 19:37:25 2023\n read: IOPS=133, BW=536MiB/s (562MB/s)(1024MiB/1911msec)\n clat (usec): min=4751, max=13813, avg=7109.46, stdev=1330.79\n lat (usec): min=4754, max=13815, avg=7111.26, stdev=1330.78\n clat percentiles (usec):\n | 1.00th=[ 5014], 5.00th=[ 5342], 10.00th=[ 5800], 20.00th=[ 6128],\n | 30.00th=[ 6390], 40.00th=[ 6652], 50.00th=[ 6849], 60.00th=[ 7111],\n | 70.00th=[ 7439], 80.00th=[ 7832], 90.00th=[ 8586], 95.00th=[ 9503],\n | 99.00th=[12125], 99.50th=[12518], 99.90th=[13829], 99.95th=[13829],\n | 99.99th=[13829]\n bw ( KiB/s): min=476279, max=589824, per=25.24%, avg=548858.00, stdev=63028.99, samples=3\n iops : min= 116, max= 144, avg=133.67, stdev=15.37, samples=3\n lat (msec) : 10=96.48%, 20=3.52%\n cpu : usr=0.63%, sys=64.08%, ctx=3023, majf=0, minf=1036\n IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, \u003e=64=0.0%\n submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n issued rwts: total=256,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n latency : target=0, window=0, percentile=100.00%, depth=1\nsequential-read: (groupid=0, jobs=1): err= 0: pid=3705619: Wed May 24 19:37:25 2023\n read: IOPS=134, BW=536MiB/s (562MB/s)(1024MiB/1910msec)\n clat (usec): min=4812, max=13160, avg=7107.62, stdev=1252.07\n lat (usec): min=4814, max=13163, avg=7109.17, stdev=1252.09\n clat percentiles (usec):\n | 1.00th=[ 4883], 5.00th=[ 5473], 10.00th=[ 5669], 20.00th=[ 6063],\n | 30.00th=[ 6456], 40.00th=[ 6652], 50.00th=[ 6980], 60.00th=[ 7242],\n | 70.00th=[ 7635], 80.00th=[ 7963], 90.00th=[ 8586], 95.00th=[ 9503],\n | 99.00th=[11469], 99.50th=[11731], 99.90th=[13173], 99.95th=[13173],\n | 99.99th=[13173]\n bw ( KiB/s): min=476279, max=598016, per=25.24%, avg=548863.33, stdev=64161.96, samples=3\n iops : min= 116, max= 146, avg=133.67, stdev=15.70, samples=3\n lat (msec) : 10=96.88%, 20=3.12%\n cpu : usr=0.31%, sys=63.75%, ctx=3115, majf=0, minf=1036\n IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, \u003e=64=0.0%\n submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n issued rwts: total=256,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n latency : target=0, window=0, percentile=100.00%, depth=1\n\nRun status group 0 (all jobs):\n READ: bw=2123MiB/s (2227MB/s), 531MiB/s-536MiB/s (557MB/s-562MB/s), io=4096MiB (4295MB), run=1910-1929msec\n```\n\ndirectory fio test\n\n```\nls -lah /home/juicefs_mount/fio\ntotal 5.1G\ndrwxr-xr-x 2 root root 4.0K May 24 19:08 .\ndrwxrwxrwx 3 root root 4.0K May 24 19:01 ..\n-rw-r--r-- 1 root root 1.0G May 24 19:08 sequential-read.0.0\n-rw-r--r-- 1 root root 1.0G May 24 19:08 sequential-read.1.0\n-rw-r--r-- 1 root root 1.0G May 24 19:08 sequential-read.2.0\n-rw-r--r-- 1 root root 1.0G May 24 19:08 sequential-read.3.0\n-rw-r--r-- 1 root root 1.0G May 24 19:01 sequential-write.0.0\n```\n\nChecking the S3 Gateway\n\n```\naws --endpoint-url http://localhost:3777 s3 ls --recursive myjuicefs\n\n2023-05-24 19:08:04 1073741824 fio/sequential-read.0.0\n2023-05-24 19:08:09 1073741824 fio/sequential-read.1.0\n2023-05-24 19:08:07 1073741824 fio/sequential-read.2.0\n2023-05-24 19:08:05 1073741824 fio/sequential-read.3.0\n2023-05-24 19:01:24 1073741824 fio/sequential-write.0.0\n```\n\n## Redis Metadata Cache + Sharded R2 Mount On Intel Xeon E-2276G 6C/12T, 32GB memory and 2x 960GB NVMe raid 1\n\nSwitched from local [sqlite3 metadata caching](https://juicefs.com/docs/community/databases_for_metadata#sqlite) to [Redis server metadata caching](https://juicefs.com/docs/community/databases_for_metadata#redis) with 10x sharded R2 object storage buckets and location hint North American East.\n\nOn Centmin Mod LEMP stack server rather than use default Redis server on port 6379 which maybe used for other applications in a caching capacity, setup a separate Redis server on port 6479 using [`redis-generator.sh`](https://github.com/centminmod/centminmod-redis).\n\n```\npushd /root/tools\ngit clone https://github.com/centminmod/centminmod-redis\ncd centminmod-redis\n```\n\n`redis-generator.sh` options\n\n```\n./redis-generator.sh \n\n* Usage: where X equal postive integer for number of redis\n servers to create with incrementing TCP redis ports\n starting at STARTPORT=6479.\n* prep - standalone prep command installs redis-cluster-tool\n* prepupdate - standalone prep update command updates redis-cluster-tool\n* multi X - no. of standalone redis instances to create\n* multi-cache X - no. of standalone redis instances + disable ondisk persistence\n* clusterprep X - no. of cluster enabled config instances\n* clustermake 6 - to enable cluster mode + create cluster\n* clustermake 9 - flag to enable cluster mode + create cluster\n* replication X - create redis replication\n* replication X 6579 - create replication with custom start port 6579\n* replication-cache X - create redis replication + disable ondisk persistence\n* replication-cache X 6579 - create replication with custom start port 6579\n* delete X - no. of redis instances to delete\n* delete X 6579 - no. of redis instances to delete + custom start port 6579\n\n./redis-generator.sh prep\n./redis-generator.sh prepupdate\n./redis-generator.sh multi X\n./redis-generator.sh multi-cache X\n./redis-generator.sh clusterprep X\n./redis-generator.sh clustermake 6\n./redis-generator.sh clustermake 9\n./redis-generator.sh replication X\n./redis-generator.sh replication X 6579\n./redis-generator.sh replication-cache X\n./redis-generator.sh replication-cache X 6579\n./redis-generator.sh delete X\n./redis-generator.sh delete X 6579\n```\n\nCreate 2x Redis servers on ports 6479 and 6480 by editing script first with `DEBUG_REDISGEN='n'` and then running the multi command:\n\n```\n./redis-generator.sh multi 2\n```\n\nOutputs the following where:\n\n* /etc/redis6479/redis6479.conf config file\n* /etc/redis6480/redis6480.conf config file\n\n```\n./redis-generator.sh multi 2\n\nCreating redis servers starting at TCP = 6479...\n-------------------------------------------------------\ncreating redis server: redis6479.service [increment value: 0]\nredis TCP port: 6479\ncreate systemd redis6479.service\ncp -a /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis6479.service\ncreate /etc/redis6479/redis6479.conf config file\nmkdir -p /etc/redis6479\ncp -a /etc/redis/redis.conf /etc/redis6479/redis6479.conf\n-rw-r----- 1 redis root 92K Apr 18 00:42 /etc/redis6479/redis6479.conf\n-rw-r--r-- 1 root root 474 May 25 02:32 /usr/lib/systemd/system/redis6479.service\nCreated symlink /etc/systemd/system/multi-user.target.wants/redis6479.service → /usr/lib/systemd/system/redis6479.service.\nNote: Forwarding request to 'systemctl enable redis6479.service'.\n## Redis TCP 6479 Info ##\nredis_version:6.2.12\nredis_mode:standalone\nprocess_id:3723969\ntcp_port:6479\nuptime_in_seconds:0\nuptime_in_days:0\nexecutable:/etc/redis6479/redis-server\nconfig_file:/etc/redis6479/redis6479.conf\n-------------------------------------------------------\ncreating redis server: redis6480.service [increment value: 1]\nredis TCP port: 6480\ncreate systemd redis6480.service\ncp -a /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis6480.service\ncreate /etc/redis6480/redis6480.conf config file\nmkdir -p /etc/redis6480\ncp -a /etc/redis/redis.conf /etc/redis6480/redis6480.conf\n-rw-r----- 1 redis root 92K Apr 18 00:42 /etc/redis6480/redis6480.conf\n-rw-r--r-- 1 root root 474 May 25 02:32 /usr/lib/systemd/system/redis6480.service\nCreated symlink /etc/systemd/system/multi-user.target.wants/redis6480.service → /usr/lib/systemd/system/redis6480.service.\nNote: Forwarding request to 'systemctl enable redis6480.service'.\n## Redis TCP 6480 Info ##\nredis_version:6.2.12\nredis_mode:standalone\nprocess_id:3724058\ntcp_port:6480\nuptime_in_seconds:0\nuptime_in_days:0\nexecutable:/etc/redis6480/redis-server\nconfig_file:/etc/redis6480/redis6480.conf\n```\n\n```\nsystemctl status redis6479 redis6480\n● redis6479.service - Redis persistent key-value database\n Loaded: loaded (/usr/lib/systemd/system/redis6479.service; enabled; vendor preset: disabled)\n Drop-In: /etc/systemd/system/redis6479.service.d\n └─limit.conf, user.conf\n Active: active (running) since Thu 2023-05-25 02:32:52 CDT; 2min 4s ago\n Main PID: 3723969 (redis-server)\n Status: \"Ready to accept connections\"\n Tasks: 5 (limit: 203337)\n Memory: 2.9M\n CGroup: /system.slice/redis6479.service\n └─3723969 /etc/redis6479/redis-server 127.0.0.1:6479\n\nMay 25 02:32:52 hostname systemd[1]: Starting Redis persistent key-value database...\nMay 25 02:32:52 hostname systemd[1]: Started Redis persistent key-value database.\n\n● redis6480.service - Redis persistent key-value database\n Loaded: loaded (/usr/lib/systemd/system/redis6480.service; enabled; vendor preset: disabled)\n Drop-In: /etc/systemd/system/redis6480.service.d\n └─limit.conf, user.conf\n Active: active (running) since Thu 2023-05-25 02:32:52 CDT; 2min 3s ago\n Main PID: 3724058 (redis-server)\n Status: \"Ready to accept connections\"\n Tasks: 5 (limit: 203337)\n Memory: 2.6M\n CGroup: /system.slice/redis6480.service\n └─3724058 /etc/redis6480/redis-server 127.0.0.1:6480\n\nMay 25 02:32:52 hostname systemd[1]: Starting Redis persistent key-value database...\nMay 25 02:32:52 hostname systemd[1]: Started Redis persistent key-value database.\n```\n\nFor JuiceFS Redis metadata caching the references in JuiceFS mount formatting command will be for either:\n\n* `redis://:password@localhost:6479/1` or without password `redis://:localhost:6479/1`\n* `redis://:password@localhost:6480/1` or without password `redis://:localhost:6480/1`\n\nSet `/etc/redis6479/redis6479.conf` and `/etc/redis6480/redis6480.conf` config files password for Redis and set `maxmemory-policy noeviction` and `appendonly yes`\n\n```\nrequirepass password\nmaxmemory-policy noeviction\nappendonly yes\n```\n\nAnd set JuiceFS best practices for Redis metadata caching outlined at https://juicefs.com/docs/community/redis_best_practices.\n\n```\ngrep -C1 -i '^appendfsync' /etc/redis6479/redis6479.conf\n# appendfsync always\nappendfsync everysec\n# appendfsync no\n```\n\nRestart Redis servers\n\n```\nsystemctl restart redis6479 redis6480\n```\n\nFormat JuiceFS sharded mount to use Redis metadata caching via `redis://:password@localhost:6479/1`\n\n```\ncfaccountid='CF_ACCOUNT_ID'\ncfaccesskey=''\ncfsecretkey=''\ncfbucketname='juicefs-shard'\n\nmkdir -p /home/juicefs\ncd /home/juicefs\n\njuicefs format --storage s3 \\\n --shards 10 \\\n --bucket https://${cfbucketname}-%d.${cfaccountid}.r2.cloudflarestorage.com \\\n --access-key $cfaccesskey \\\n --secret-key $cfsecretkey \\\n --compress none \\\n --trash-days 0 \\\n --block-size 4096 \\\n redis://:password@localhost:6479/1 myjuicefs\n```\n\nOutputs\n\n```\n2023/05/25 03:12:14.146627 juicefs[3725353] \u003cINFO\u003e: Meta address: redis://:****@localhost:6479/1 [interface.go:401]\n2023/05/25 03:12:14.147407 juicefs[3725353] \u003cINFO\u003e: Ping redis: 29.563µs [redis.go:2904]\n2023/05/25 03:12:14.148376 juicefs[3725353] \u003cINFO\u003e: Data use shard10://s3://juicefs-shard-0/myjuicefs/ [format.go:434]\n2023/05/25 03:12:14.872195 juicefs[3725353] \u003cERROR\u003e: Can't list s3://juicefs-shard-0/: InvalidMaxKeys: MaxKeys params must be positive integer \u003c= 1000.\n status code: 400, request id: , host id: [sharding.go:85]\n2023/05/25 03:12:14.872252 juicefs[3725353] \u003cWARNING\u003e: List storage shard10://s3://juicefs-shard-0/myjuicefs/ failed: list s3://juicefs-shard-0/: InvalidMaxKeys: MaxKeys params must be positive integer \u003c= 1000.\n status code: 400, request id: , host id: [format.go:452]\n2023/05/25 03:12:15.367557 juicefs[3725353] \u003cINFO\u003e: Volume is formatted as {\n \"Name\": \"myjuicefs\",\n \"UUID\": \"UUID-UUID-UUID-UUID-UUID\",\n \"Storage\": \"s3\",\n \"Bucket\": \"https://juicefs-shard-%d.CF_ACCOUNT_ID.r2.cloudflarestorage.com\",\n \"AccessKey\": \"cfaccesskey\",\n \"SecretKey\": \"removed\",\n \"BlockSize\": 4096,\n \"Compression\": \"none\",\n \"Shards\": 10,\n \"KeyEncrypted\": true,\n \"MetaVersion\": 1\n} [format.go:471]\n```\n\nEdit `/usr/lib/systemd/system/juicefs.service`. Note for Cloudflare R2, you need to [disable automatica metadata backups](https://juicefs.com/docs/community/how_to_setup_object_storage/#r2) with `--backup-meta 0`. Otherwise, for non-R2 S3 providers, you can set `--backup-meta 1h`.\n\n``` \n[Unit]\nDescription=JuiceFS\nAssertPathIsDirectory=/home/juicefs_mount\nAfter=network-online.target\n\n[Service]\nType=simple\nWorkingDirectory=/home/juicefs\nExecStart=/usr/local/bin/juicefs mount \\\n \"redis://:password@localhost:6479/1\" \\\n /home/juicefs_mount \\\n --no-usage-report \\\n --writeback \\\n --cache-size 102400 \\\n --cache-dir /home/juicefs_cache \\\n --buffer-size 2048 \\\n --open-cache 0 \\\n --attr-cache 1 \\\n --entry-cache 1 \\\n --dir-entry-cache 1 \\\n --cache-partial-only false \\\n --free-space-ratio 0.1 \\\n --max-uploads 20 \\\n --max-deletes 10 \\\n --backup-meta 0 \\\n --log /var/log/juicefs.log \\\n --get-timeout 300 \\\n --put-timeout 900 \\\n --io-retries 90 \\\n --prefetch 1\n\nExecStop=/usr/local/bin/juicefs umount /home/juicefs_mount\nRestart=always\nRestartSec=5\n\n[Install]\nWantedBy=default.target\n```\n\nEdit `/usr/lib/systemd/system/juicefs-gateway.service`\n\n```\n[Unit]\nDescription=JuiceFS Gateway\nAfter=network-online.target\n\n[Service]\nEnvironment='MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE'\nEnvironment='MINIO_ROOT_PASSWORD=12345678'\nType=simple\nWorkingDirectory=/home/juicefs\nExecStart=/usr/local/bin/juicefs gateway \\\n --no-usage-report \\\n --writeback \\\n --cache-size 102400 \\\n --cache-dir /home/juicefs_cache \\\n --attr-cache 1 \\\n --entry-cache 0 \\\n --dir-entry-cache 1 \\\n --prefetch 1 \\\n --free-space-ratio 0.1 \\\n --max-uploads 20 \\\n --max-deletes 10 \\\n --backup-meta 0 \\\n --get-timeout 300 \\\n --put-timeout 900 \\\n --io-retries 90 \\\n --buffer-size 2048 \\\n \"redis://:password@localhost:6479/1\" \\\n localhost:3777\n\nRestart=always\nRestartSec=5\n\n[Install]\nWantedBy=default.target\n```\n\nRestart Redis servers\n\n```\nsystemctl restart juicefs.service juicefs-gateway.service\n```\n\nNote: As `--backup-meta 0` is set for Cloudflare R2 to disable automatic metadata backups, you can manually run the backup command to backup to a file i.e. `meta-dump.json`:\n\n```\njuicefs dump redis://:password@localhost:6479/1 meta-dump.json\n```\n\n```\njuicefs dump redis://:password@localhost:6479/1 meta-dump.json\n2023/05/26 07:19:58.883775 juicefs[3791823] \u003cINFO\u003e: Meta address: redis://:****@localhost:6479/1 [interface.go:401]\n2023/05/26 07:19:58.884482 juicefs[3791823] \u003cINFO\u003e: Ping redis: 19.157µs [redis.go:2904]\n2023/05/26 07:19:58.884709 juicefs[3791823] \u003cWARNING\u003e: Secret key is removed for the sake of safety [redis.go:3236]\nDumped entries count: 5 / 5 [==============================================================] done \n2023/05/26 07:19:58.885830 juicefs[3791823] \u003cINFO\u003e: Dump metadata into meta-dump.json succeed [dump.go:76]\n```\n\n### JuiceFS Benchmarks 100x R2 Sharded Mount + Redis Metadata Caching\n\nFormat JuiceFS sharded mount to use Redis metadata caching via `redis://:password@localhost:6479/1`\n\n```\ncfaccountid='CF_ACCOUNT_ID'\ncfaccesskey=''\ncfsecretkey=''\ncfbucketname='juicefs-shard'\n\nmkdir -p /home/juicefs\ncd /home/juicefs\n\njuicefs format --storage s3 \\\n --shards 100 \\\n --bucket https://${cfbucketname}-%d.${cfaccountid}.r2.cloudflarestorage.com \\\n --access-key $cfaccesskey \\\n --secret-key $cfsecretkey \\\n --compress none \\\n --trash-days 0 \\\n --block-size 4096 \\\n redis://:password@localhost:6479/1 myjuicefs\n```\n\nEdit `/usr/lib/systemd/system/juicefs.service` raise to `--max-uploads 800` and `--max-deletes 100`. Note for Cloudflare R2, you need to [disable automatica metadata backups](https://juicefs.com/docs/community/how_to_setup_object_storage/#r2) with `--backup-meta 0`. Otherwise, for non-R2 S3 providers, you can set `--backup-meta 1h`.\n\n```\n[Unit]\nDescription=JuiceFS\nAssertPathIsDirectory=/home/juicefs_mount\nAfter=network-online.target\n\n[Service]\nType=simple\nWorkingDirectory=/home/juicefs\nExecStart=/usr/local/bin/juicefs mount \\\n \"redis://:password@localhost:6479/1\" \\\n /home/juicefs_mount \\\n --no-usage-report \\\n --writeback \\\n --cache-size 102400 \\\n --cache-dir /home/juicefs_cache \\\n --buffer-size 3072 \\\n --open-cache 0 \\\n --attr-cache 1 \\\n --entry-cache 1 \\\n --dir-entry-cache 1 \\\n --cache-partial-only false \\\n --free-space-ratio 0.1 \\\n --max-uploads 800 \\\n --max-deletes 100 \\\n --backup-meta 0 \\\n --log /var/log/juicefs.log \\\n --get-timeout 300 \\\n --put-timeout 900 \\\n --io-retries 90 \\\n --prefetch 2\n\nExecStop=/usr/local/bin/juicefs umount /home/juicefs_mount\nRestart=always\nRestartSec=5\n\n[Install]\nWantedBy=default.target\n```\n\nThe table below shows comparison between [100x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-100x-r2-sharded-mount--redis-metadata-caching) vs [61x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-61x-r2-sharded-mount--redis-metadata-caching) vs [21x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-21x-r2-sharded-mount--redis-metadata-caching) vs [10x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-10x-r2-sharded-mount--redis-metadata-caching) vs [10x Cloudflare R2 sharded JuiceFS mount](#10x-r2-sharded-juicefs-mount) vs [1x Cloudflare JuiceFS mount (default)](#on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1). All R2 storage locations are with location hint North American East.\n\nDefault 1024MB big file.\n\n| ITEM | VALUE (100x R2 Sharded + Redis) | COST (100x R2 Sharded + Redis) | VALUE (61x R2 Sharded + Redis) | COST (61x R2 Sharded + Redis) | VALUE (21x R2 Sharded + Redis) | COST (21x R2 Sharded + Redis) | VALUE (10x R2 Sharded + Redis) | COST (10x R2 Sharded + Redis) | VALUE (10x R2 Sharded) | COST (10x R2 Sharded) | VALUE (1x R2 Default) | COST (1x R2 Default) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Write big file | 1103.70 MiB/s | 7.42 s/file | 1778.16 MiB/s | 2.30 s/file | 1774.18 MiB/s | 2.31 s/file | 1904.61 MiB/s | 2.15 s/file | 906.04 MiB/s | 4.52 s/file | 1374.08 MiB/s | 2.98 s/file |\n| Read big file | 342.66 MiB/s | 23.91 s/file | 231.92 MiB/s | 17.66 s/file | 162.36 MiB/s | 25.23 s/file | 201.00 MiB/s | 20.38 s/file | 223.19 MiB/s | 18.35 s/file | 152.23 MiB/s | 26.91 s/file |\n| Write small file | 2505.3 files/s | 3.19 ms/file | 2449.2 files/s | 1.63 ms/file | 2333.5 files/s | 1.71 ms/file | 1319.8 files/s | 3.03 ms/file | 701.2 files/s | 5.70 ms/file | 780.3 files/s | 5.13 ms/file |\n| Read small file | 13321.4 files/s | 0.60 ms/file | 5997.6 files/s | 0.67 ms/file | 10382.7 files/s | 0.39 ms/file | 10279.8 files/s | 0.39 ms/file | 6378.3 files/s | 0.63 ms/file | 8000.9 files/s | 0.50 ms/file |\n| Stat file | 16229.5 files/s | 0.49 ms/file | 38302.2 files/s | 0.10 ms/file | 15955.7 files/s | 0.25 ms/file | 15890.1 files/s | 0.25 ms/file | 21123.7 files/s | 0.19 ms/file | 27902.2 files/s | 0.14 ms/file |\n| FUSE operation | 142585 operations | 2.51 ms/op | 71292 operations | 1.80 ms/op | 71319 operations | 2.79 ms/op | 71338 operations | 2.23 ms/op | 71555 operations | 2.16 ms/op | 71649 operations | 3.06 ms/op |\n| Update meta | 3473 operations | 1.98 ms/op | 1740 operations | 0.25 ms/op | 1739 operations | 0.25 ms/op | 1740 operations | 0.27 ms/op | 6271 operations | 9.01 ms/op | 6057 operations | 2.50 ms/op |\n| Put object | 2847 operations | 470.88 ms/op | 1087 operations | 466.15 ms/op | 1055 operations | 514.85 ms/op | 1083 operations | 390.88 ms/op | 1152 operations | 403.23 ms/op | 1106 operations | 547.32 ms/op |\n| Get object | 2048 operations | 402.89 ms/op | 1024 operations | 319.02 ms/op | 1027 operations | 346.44 ms/op | 1024 operations | 294.63 ms/op | 1034 operations | 278.61 ms/op | 1030 operations | 301.80 ms/op |\n| Delete object | 440 operations | 174.48 ms/op | 215 operations | 201.12 ms/op | 736 operations | 195.40 ms/op | 754 operations | 125.28 ms/op | 316 operations | 124.32 ms/op | 29 operations | 234.02 ms/op |\n| Write into cache | 2848 operations | 13.31 ms/op | 1424 operations | 5.36 ms/op | 1424 operations | 7.19 ms/op | 1424 operations | 4.85 ms/op | 1424 operations | 24 ms/op | 1424 operations | 7.19 ms/op |\n| Read from cache | 800 operations | 0.09 ms/op | 400 operations | 0.07 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.04 ms/op |\n\nDefault 1MB big file.\n\n| ITEM | VALUE (100x R2 Sharded + Redis) | COST (100x R2 Sharded + Redis) | VALUE (61x R2 Sharded + Redis) | COST (61x R2 Sharded + Redis) | VALUE (21x R2 Sharded + Redis) | COST (21x R2 Sharded + Redis) | VALUE (10x R2 Sharded + Redis) | COST (10x R2 Sharded + Redis) | VALUE (1x R2 Default) | COST (1x R2 Default) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Write big file | 637.16 MiB/s | 0.01 s/file | 617.15 MiB/s | 0.01 s/file | 600.01 MiB/s | 0.01 s/file | 530.10 MiB/s | 0.01 s/file | 230.82 MiB/s | 0.02 s/file |\n| Read big file | 1764.73 MiB/s | 0.00 s/file | 1600.85 MiB/s | 0.00 s/file | 1300.69 MiB/s | 0.00 s/file | 1914.40 MiB/s | 0.00 s/file | 1276.38 MiB/s | 0.00 s/file |\n| Write small file | 2666.9 files/s | 3.00 ms/file | 2808.3 files/s | 1.42 ms/file | 2648.3 files/s | 1.51 ms/file | 2715.4 files/s | 1.47 ms/file | 675.7 files/s | 5.92 ms/file |\n| Read small file | 10905.8 files/s | 0.73 ms/file | 10154.0 files/s | 0.39 ms/file | 10442.4 files/s | 0.38 ms/file | 10069.0 files/s | 0.40 ms/file | 7833.1 files/s | 0.51 ms/file |\n| Stat file | 22475.0 files/s | 0.36 ms/file | 15935.2 files/s | 0.25 ms/file | 16277.5 files/s | 0.25 ms/file | 16545.3 files/s | 0.24 ms/file | 28226.1 files/s | 0.14 ms/file |\n| FUSE operation | 11485 operations | 0.12 ms/op | 5761 operations | 0.09 ms/op | 5765 operations | 0.09 ms/op | 5767 operations | 0.09 ms/op | 5756 operations | 0.41 ms/op |\n| Update meta | 3233 operations | 0.19 ms/op | 1617 operations | 0.19 ms/op | 1617 operations | 0.18 ms/op | 1617 operations | 0.19 ms/op | 5770 operations | 0.70 ms/op |\n| Put object | 735 operations | 344.93 ms/op | 32 operations | 377.01 ms/op | 30 operations | 369.65 ms/op | 37 operations | 290.94 ms/op | 118 operations | 242.35 ms/op |\n| Get object | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op |\n| Delete object | 117 operations | 123.92 ms/op | 76 operations | 189.67 ms/op | 22 operations | 268.03 ms/op | 48 operations | 103.83 ms/op | 95 operations | 83.94 ms/op |\n| Write into cache | 808 operations | 0.13 ms/op | 404 operations | 0.11 ms/op | 404 operations | 0.11 ms/op | 404 operations | 0.11 ms/op | 404 operations | 0.14 ms/op |\n| Read from cache | 816 operations | 0.13 ms/op | 408 operations | 0.07 ms/op | 408 operations | 0.08 ms/op | 408 operations | 0.06 ms/op | 408 operations | 0.06 ms/op |\n\n100x R2 sharded JuiceFS mount with Redis metadata caching with location hint North American East\n\nDefault 1024MB big file.\n\n```\njuicefs bench -p 8 /home/juicefs_mount/\n Write big blocks count: 8192 / 8192 [===========================================================] done \n Read big blocks count: 8192 / 8192 [===========================================================] done \nWrite small blocks count: 800 / 800 [=============================================================] done \n Read small blocks count: 800 / 800 [=============================================================] done \n Stat small files count: 800 / 800 [=============================================================] done \nBenchmark finished!\nBlockSize: 1 MiB, BigFileSize: 1024 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 8\nTime used: 33.5 s, CPU: 238.9%, Memory: 1687.5 MiB\n+------------------+-------------------+--------------+\n| ITEM | VALUE | COST |\n+------------------+-------------------+--------------+\n| Write big file | 1103.70 MiB/s | 7.42 s/file |\n| Read big file | 342.66 MiB/s | 23.91 s/file |\n| Write small file | 2505.3 files/s | 3.19 ms/file |\n| Read small file | 13321.4 files/s | 0.60 ms/file |\n| Stat file | 16229.5 files/s | 0.49 ms/file |\n| FUSE operation | 142585 operations | 2.51 ms/op |\n| Update meta | 3473 operations | 1.98 ms/op |\n| Put object | 2847 operations | 470.88 ms/op |\n| Get object | 2048 operations | 402.89 ms/op |\n| Delete object | 440 operations | 174.48 ms/op |\n| Write into cache | 2848 operations | 13.31 ms/op |\n| Read from cache | 800 operations | 0.09 ms/op |\n+------------------+-------------------+--------------+\n```\n\nDefault 1MB big file.\n\n```\njuicefs bench -p 8 /home/juicefs_mount/ --big-file-size 1\n Write big blocks count: 8 / 8 [==============================================================] done \n Read big blocks count: 8 / 8 [==============================================================] done \nWrite small blocks count: 800 / 800 [=============================================================] done \n Read small blocks count: 800 / 800 [=============================================================] done \n Stat small files count: 800 / 800 [=============================================================] done \nBenchmark finished!\nBlockSize: 1 MiB, BigFileSize: 1 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 8\nTime used: 0.8 s, CPU: 210.4%, Memory: 605.2 MiB\n+------------------+------------------+--------------+\n| ITEM | VALUE | COST |\n+------------------+------------------+--------------+\n| Write big file | 637.16 MiB/s | 0.01 s/file |\n| Read big file | 1764.73 MiB/s | 0.00 s/file |\n| Write small file | 2666.9 files/s | 3.00 ms/file |\n| Read small file | 10905.8 files/s | 0.73 ms/file |\n| Stat file | 22475.0 files/s | 0.36 ms/file |\n| FUSE operation | 11485 operations | 0.12 ms/op |\n| Update meta | 3233 operations | 0.19 ms/op |\n| Put object | 735 operations | 344.93 ms/op |\n| Get object | 0 operations | 0.00 ms/op |\n| Delete object | 117 operations | 123.92 ms/op |\n| Write into cache | 808 operations | 0.13 ms/op |\n| Read from cache | 816 operations | 0.13 ms/op |\n+------------------+------------------+--------------+\n```\n\n100x Cloudflare R2 buckets info using `sharded-buckets-info-r2.sh` passing AWS CLI profile `r2` with R2 bucket prefix `juicefs-shard-` for `100` sharded buckets and `https://cfaccountid.r2.cloudflarestorage.com` R2 endpoint.\n\n```\n./sharded-buckets-info-r2.sh r2 juicefs-shard- 100 https://cfaccountid.r2.cloudflarestorage.com\nBucket: juicefs-shard-0, Total Files: 34, Total Size: 4456448\nBucket: juicefs-shard-1, Total Files: 22, Total Size: 2627642\nBucket: juicefs-shard-2, Total Files: 28, Total Size: 3670016\nBucket: juicefs-shard-3, Total Files: 31, Total Size: 3935716\nBucket: juicefs-shard-4, Total Files: 28, Total Size: 3670016\nBucket: juicefs-shard-5, Total Files: 39, Total Size: 5111808\nBucket: juicefs-shard-6, Total Files: 35, Total Size: 4459354\nBucket: juicefs-shard-7, Total Files: 36, Total Size: 5506235\nBucket: juicefs-shard-8, Total Files: 140, Total Size: 20185088\nBucket: juicefs-shard-9, Total Files: 59, Total Size: 7733248\nBucket: juicefs-shard-10, Total Files: 50, Total Size: 6553600\nBucket: juicefs-shard-11, Total Files: 40, Total Size: 5242880\nBucket: juicefs-shard-12, Total Files: 63, Total Size: 10092544\nBucket: juicefs-shard-13, Total Files: 28, Total Size: 3670016\nBucket: juicefs-shard-14, Total Files: 31, Total Size: 4063232\nBucket: juicefs-shard-15, Total Files: 72, Total Size: 9437184\nBucket: juicefs-shard-16, Total Files: 37, Total Size: 4849664\nBucket: juicefs-shard-17, Total Files: 41, Total Size: 5373952\nBucket: juicefs-shard-18, Total Files: 123, Total Size: 18874368\nBucket: juicefs-shard-19, Total Files: 39, Total Size: 6029312\nBucket: juicefs-shard-20, Total Files: 30, Total Size: 3932160\nBucket: juicefs-shard-21, Total Files: 30, Total Size: 3932160\nBucket: juicefs-shard-22, Total Files: 42, Total Size: 5505024\nBucket: juicefs-shard-23, Total Files: 173, Total Size: 23592960\nBucket: juicefs-shard-24, Total Files: 42, Total Size: 5505024\nBucket: juicefs-shard-25, Total Files: 41, Total Size: 5373952\nBucket: juicefs-shard-26, Total Files: 33, Total Size: 4198517\nBucket: juicefs-shard-27, Total Files: 35, Total Size: 6422528\nBucket: juicefs-shard-28, Total Files: 62, Total Size: 8126464\nBucket: juicefs-shard-29, Total Files: 34, Total Size: 4327957\nBucket: juicefs-shard-30, Total Files: 28, Total Size: 3670016\nBucket: juicefs-shard-31, Total Files: 40, Total Size: 5242880\nBucket: juicefs-shard-32, Total Files: 32, Total Size: 4194304\nBucket: juicefs-shard-33, Total Files: 76, Total Size: 9961472\nBucket: juicefs-shard-34, Total Files: 31, Total Size: 4063232\nBucket: juicefs-shard-35, Total Files: 45, Total Size: 6685553\nBucket: juicefs-shard-36, Total Files: 41, Total Size: 5242916\nBucket: juicefs-shard-37, Total Files: 29, Total Size: 3801088\nBucket: juicefs-shard-38, Total Files: 123, Total Size: 16911519\nBucket: juicefs-shard-39, Total Files: 47, Total Size: 6160384\nBucket: juicefs-shard-40, Total Files: 27, Total Size: 3538944\nBucket: juicefs-shard-41, Total Files: 46, Total Size: 5899407\nBucket: juicefs-shard-42, Total Files: 34, Total Size: 4326157\nBucket: juicefs-shard-43, Total Files: 34, Total Size: 4456448\nBucket: juicefs-shard-44, Total Files: 42, Total Size: 6422528\nBucket: juicefs-shard-45, Total Files: 159, Total Size: 21757952\nBucket: juicefs-shard-46, Total Files: 45, Total Size: 5898240\nBucket: juicefs-shard-47, Total Files: 56, Total Size: 8257536\nBucket: juicefs-shard-48, Total Files: 40, Total Size: 5242880\nBucket: juicefs-shard-49, Total Files: 51, Total Size: 6684672\nBucket: juicefs-shard-50, Total Files: 29, Total Size: 3801088\nBucket: juicefs-shard-51, Total Files: 179, Total Size: 23461888\nBucket: juicefs-shard-52, Total Files: 40, Total Size: 5112935\nBucket: juicefs-shard-53, Total Files: 138, Total Size: 19795756\nBucket: juicefs-shard-54, Total Files: 62, Total Size: 8914826\nBucket: juicefs-shard-55, Total Files: 41, Total Size: 6291456\nBucket: juicefs-shard-56, Total Files: 72, Total Size: 9437184\nBucket: juicefs-shard-57, Total Files: 45, Total Size: 6815744\nBucket: juicefs-shard-58, Total Files: 32, Total Size: 4064159\nBucket: juicefs-shard-59, Total Files: 38, Total Size: 6685639\nBucket: juicefs-shard-60, Total Files: 34, Total Size: 4456448\nBucket: juicefs-shard-61, Total Files: 41, Total Size: 5373952\nBucket: juicefs-shard-62, Total Files: 34, Total Size: 4329917\nBucket: juicefs-shard-63, Total Files: 61, Total Size: 8912896\nBucket: juicefs-shard-64, Total Files: 50, Total Size: 7471104\nBucket: juicefs-shard-65, Total Files: 37, Total Size: 4849664\nBucket: juicefs-shard-66, Total Files: 38, Total Size: 4980736\nBucket: juicefs-shard-67, Total Files: 119, Total Size: 16515072\nBucket: juicefs-shard-68, Total Files: 115, Total Size: 15990784\nBucket: juicefs-shard-69, Total Files: 39, Total Size: 5111808\nBucket: juicefs-shard-70, Total Files: 63, Total Size: 9966336\nBucket: juicefs-shard-71, Total Files: 44, Total Size: 5767168\nBucket: juicefs-shard-72, Total Files: 31, Total Size: 4063232\nBucket: juicefs-shard-73, Total Files: 42, Total Size: 5244609\nBucket: juicefs-shard-74, Total Files: 25, Total Size: 3276800\nBucket: juicefs-shard-75, Total Files: 47, Total Size: 6030141\nBucket: juicefs-shard-76, Total Files: 69, Total Size: 9043968\nBucket: juicefs-shard-77, Total Files: 54, Total Size: 7995392\nBucket: juicefs-shard-78, Total Files: 128, Total Size: 16518412\nBucket: juicefs-shard-79, Total Files: 104, Total Size: 13631488\nBucket: juicefs-shard-80, Total Files: 77, Total Size: 10092544\nBucket: juicefs-shard-81, Total Files: 49, Total Size: 6422528\nBucket: juicefs-shard-82, Total Files: 54, Total Size: 7077888\nBucket: juicefs-shard-83, Total Files: 84, Total Size: 11010048\nBucket: juicefs-shard-84, Total Files: 96, Total Size: 12582912\nBucket: juicefs-shard-85, Total Files: 50, Total Size: 7471104\nBucket: juicefs-shard-86, Total Files: 32, Total Size: 4194304\nBucket: juicefs-shard-87, Total Files: 41, Total Size: 6291456\nBucket: juicefs-shard-88, Total Files: 81, Total Size: 10616832\nBucket: juicefs-shard-89, Total Files: 114, Total Size: 14942208\nBucket: juicefs-shard-90, Total Files: 37, Total Size: 5767168\nBucket: juicefs-shard-91, Total Files: 54, Total Size: 7077888\nBucket: juicefs-shard-92, Total Files: 62, Total Size: 8126464\nBucket: juicefs-shard-93, Total Files: 56, Total Size: 7340032\nBucket: juicefs-shard-94, Total Files: 35, Total Size: 4587520\nBucket: juicefs-shard-95, Total Files: 46, Total Size: 6946816\nBucket: juicefs-shard-96, Total Files: 41, Total Size: 6291456\nBucket: juicefs-shard-97, Total Files: 151, Total Size: 19791872\nBucket: juicefs-shard-98, Total Files: 92, Total Size: 12976128\nBucket: juicefs-shard-99, Total Files: 177, Total Size: 24117248\nTotal for all buckets, Total Files: 5804, Total Size: 792512215\n```\n\n### JuiceFS Benchmarks 61x R2 Sharded Mount + Redis Metadata Caching\n\nFormat JuiceFS sharded mount to use Redis metadata caching via `redis://:password@localhost:6479/1`\n\n```\ncfaccountid='CF_ACCOUNT_ID'\ncfaccesskey=''\ncfsecretkey=''\ncfbucketname='juicefs-shard'\n\nmkdir -p /home/juicefs\ncd /home/juicefs\n\njuicefs format --storage s3 \\\n --shards 61 \\\n --bucket https://${cfbucketname}-%d.${cfaccountid}.r2.cloudflarestorage.com \\\n --access-key $cfaccesskey \\\n --secret-key $cfsecretkey \\\n --compress none \\\n --trash-days 0 \\\n --block-size 4096 \\\n redis://:password@localhost:6479/1 myjuicefs\n```\n\nEdit `/usr/lib/systemd/system/juicefs.service` raise `--max-uploads 20` and `--max-deletes 10` values to `--max-uploads 30` and `--max-deletes 30`. Note for Cloudflare R2, you need to [disable automatica metadata backups](https://juicefs.com/docs/community/how_to_setup_object_storage/#r2) with `--backup-meta 0`. Otherwise, for non-R2 S3 providers, you can set `--backup-meta 1h`.\n\n```\n[Unit]\nDescription=JuiceFS\nAssertPathIsDirectory=/home/juicefs_mount\nAfter=network-online.target\n\n[Service]\nType=simple\nWorkingDirectory=/home/juicefs\nExecStart=/usr/local/bin/juicefs mount \\\n \"redis://:password@localhost:6479/1\" \\\n /home/juicefs_mount \\\n --no-usage-report \\\n --writeback \\\n --cache-size 102400 \\\n --cache-dir /home/juicefs_cache \\\n --buffer-size 2048 \\\n --open-cache 0 \\\n --attr-cache 1 \\\n --entry-cache 1 \\\n --dir-entry-cache 1 \\\n --cache-partial-only false \\\n --free-space-ratio 0.1 \\\n --max-uploads 30 \\\n --max-deletes 30 \\\n --backup-meta 0 \\\n --log /var/log/juicefs.log \\\n --get-timeout 300 \\\n --put-timeout 900 \\\n --io-retries 90 \\\n --prefetch 1\n\nExecStop=/usr/local/bin/juicefs umount /home/juicefs_mount\nRestart=always\nRestartSec=5\n\n[Install]\nWantedBy=default.target\n```\n\nThe table below shows comparison between [61x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-61x-r2-sharded-mount--redis-metadata-caching) vs [21x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-21x-r2-sharded-mount--redis-metadata-caching) vs [10x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-10x-r2-sharded-mount--redis-metadata-caching) vs [10x Cloudflare R2 sharded JuiceFS mount](#10x-r2-sharded-juicefs-mount) vs [1x Cloudflare JuiceFS mount (default)](#on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1). All R2 storage locations are with location hint North American East.\n\nDefault 1024MB big file.\n\n| ITEM | VALUE (61x R2 Sharded + Redis) | COST (61x R2 Sharded + Redis) | VALUE (21x R2 Sharded + Redis) | COST (21x R2 Sharded + Redis) | VALUE (10x R2 Sharded + Redis) | COST (10x R2 Sharded + Redis) | VALUE (10x R2 Sharded) | COST (10x R2 Sharded) | VALUE (1x R2 Default) | COST (1x R2 Default) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Write big file | 1778.16 MiB/s | 2.30 s/file | 1774.18 MiB/s | 2.31 s/file | 1904.61 MiB/s | 2.15 s/file | 906.04 MiB/s | 4.52 s/file | 1374.08 MiB/s | 2.98 s/file |\n| Read big file | 231.92 MiB/s | 17.66 s/file | 162.36 MiB/s | 25.23 s/file | 201.00 MiB/s | 20.38 s/file | 223.19 MiB/s | 18.35 s/file | 152.23 MiB/s | 26.91 s/file |\n| Write small file | 2449.2 files/s | 1.63 ms/file | 2333.5 files/s | 1.71 ms/file | 1319.8 files/s | 3.03 ms/file | 701.2 files/s | 5.70 ms/file | 780.3 files/s | 5.13 ms/file |\n| Read small file | 5997.6 files/s | 0.67 ms/file | 10382.7 files/s | 0.39 ms/file | 10279.8 files/s | 0.39 ms/file | 6378.3 files/s | 0.63 ms/file | 8000.9 files/s | 0.50 ms/file |\n| Stat file | 38302.2 files/s | 0.10 ms/file | 15955.7 files/s | 0.25 ms/file | 15890.1 files/s | 0.25 ms/file | 21123.7 files/s | 0.19 ms/file | 27902.2 files/s | 0.14 ms/file |\n| FUSE operation | 71292 operations | 1.80 ms/op | 71319 operations | 2.79 ms/op | 71338 operations | 2.23 ms/op | 71555 operations | 2.16 ms/op | 71649 operations | 3.06 ms/op |\n| Update meta | 1740 operations | 0.25 ms/op | 1739 operations | 0.25 ms/op | 1740 operations | 0.27 ms/op | 6271 operations | 9.01 ms/op | 6057 operations | 2.50 ms/op |\n| Put object | 1087 operations | 466.15 ms/op | 1055 operations | 514.85 ms/op | 1083 operations | 390.88 ms/op | 1152 operations | 403.23 ms/op | 1106 operations | 547.32 ms/op |\n| Get object | 1024 operations | 319.02 ms/op | 1027 operations | 346.44 ms/op | 1024 operations | 294.63 ms/op | 1034 operations | 278.61 ms/op | 1030 operations | 301.80 ms/op |\n| Delete object | 215 operations | 201.12 ms/op | 736 operations | 195.40 ms/op | 754 operations | 125.28 ms/op | 316 operations | 124.32 ms/op | 29 operations | 234.02 ms/op |\n| Write into cache | 1424 operations | 5.36 ms/op | 1424 operations | 7.19 ms/op | 1424 operations | 4.85 ms/op | 1424 operations | 24 ms/op | 1424 operations | 7.19 ms/op |\n| Read from cache | 400 operations | 0.07 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.04 ms/op |\n\nDefault 1MB big file.\n\n| ITEM | VALUE (61x R2 Sharded + Redis) | COST (61x R2 Sharded + Redis) | VALUE (21x R2 Sharded + Redis) | COST (21x R2 Sharded + Redis) | VALUE (10x R2 Sharded + Redis) | COST (10x R2 Sharded + Redis) | VALUE (1x R2 Default) | COST (1x R2 Default) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Write big file | 617.15 MiB/s | 0.01 s/file | 600.01 MiB/s | 0.01 s/file | 530.10 MiB/s | 0.01 s/file | 230.82 MiB/s | 0.02 s/file |\n| Read big file | 1600.85 MiB/s | 0.00 s/file | 1300.69 MiB/s | 0.00 s/file | 1914.40 MiB/s | 0.00 s/file | 1276.38 MiB/s | 0.00 s/file |\n| Write small file | 2808.3 files/s | 1.42 ms/file | 2648.3 files/s | 1.51 ms/file | 2715.4 files/s | 1.47 ms/file | 675.7 files/s | 5.92 ms/file |\n| Read small file | 10154.0 files/s | 0.39 ms/file | 10442.4 files/s | 0.38 ms/file | 10069.0 files/s | 0.40 ms/file | 7833.1 files/s | 0.51 ms/file |\n| Stat file | 15935.2 files/s | 0.25 ms/file | 16277.5 files/s | 0.25 ms/file | 16545.3 files/s | 0.24 ms/file | 28226.1 files/s | 0.14 ms/file |\n| FUSE operation | 5761 operations | 0.09 ms/op | 5765 operations | 0.09 ms/op | 5767 operations | 0.09 ms/op | 5756 operations | 0.41 ms/op |\n| Update meta | 1617 operations | 0.19 ms/op | 1617 operations | 0.18 ms/op | 1617 operations | 0.19 ms/op | 5770 operations | 0.70 ms/op |\n| Put object | 32 operations | 377.01 ms/op | 30 operations | 369.65 ms/op | 37 operations | 290.94 ms/op | 118 operations | 242.35 ms/op |\n| Get object | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op |\n| Delete object | 76 operations | 189.67 ms/op | 22 operations | 268.03 ms/op | 48 operations | 103.83 ms/op | 95 operations | 83.94 ms/op |\n| Write into cache | 404 operations | 0.11 ms/op | 404 operations | 0.11 ms/op | 404 operations | 0.11 ms/op | 404 operations | 0.14 ms/op |\n| Read from cache | 408 operations | 0.07 ms/op | 408 operations | 0.08 ms/op | 408 operations | 0.06 ms/op | 408 operations | 0.06 ms/op |\n\n61x R2 sharded JuiceFS mount with Redis metadata caching with location hint North American East\n\nDefault 1024MB big file.\n\n```\njuicefs bench -p 4 /home/juicefs_mount/\n Write big blocks count: 4096 / 4096 [===========================================================] done \n Read big blocks count: 4096 / 4096 [===========================================================] done \nWrite small blocks count: 400 / 400 [========================","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcentminmod%2Fcentminmod-juicefs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcentminmod%2Fcentminmod-juicefs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcentminmod%2Fcentminmod-juicefs/lists"}