https://github.com/centminmod/centminmod-juicefs
JuiceFS file system for S3 mount storage
https://github.com/centminmod/centminmod-juicefs
Last synced: 10 months ago
JSON representation
JuiceFS file system for S3 mount storage
- Host: GitHub
- URL: https://github.com/centminmod/centminmod-juicefs
- Owner: centminmod
- Created: 2022-05-25T04:59:58.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2023-05-28T07:48:50.000Z (about 3 years ago)
- Last Synced: 2024-10-12T02:47:43.500Z (over 1 year ago)
- Language: Shell
- Size: 102 KB
- Stars: 25
- Watchers: 4
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# JuiceFS Setup
Installing [JuiceFS](https://juicefs.com/docs/community/introduction/) high performanced POSIX compatible shared file system on Centmin Mod LEMP stack using [JuiceFS caching](https://juicefs.com/docs/community/cache_management) with [Cloudflare R2](https://blog.cloudflare.com/r2-open-beta/) - S3 compatible object storage and local sqlite3 Metadata Engine. Check out JuiceFS Github discussion forum https://github.com/juicedata/juicefs/discussions if you have questions.
JuiceFS implements an architecture that seperates "data" and "metadata" storage. When using JuiceFS to store data, the data itself is persisted in [object storage](https://juicefs.com/docs/community/how_to_setup_object_storage/) (e.g., Amazon S3, OpenStack Swift, Ceph, Azure Blob or MinIO), and the corresponding metadata can be persisted in various databases ([Metadata Engines](https://juicefs.com/docs/community/databases_for_metadata/)) such as Redis, Amazon MemoryDB, MariaDB, MySQL, TiKV, etcd, SQLite, KeyDB, PostgreSQL, BadgerDB, or FoundationDB.
From https://juicefs.com/en/blog/usage-tips/juicefs-24-qas-for-beginners
**How is the performance of JuiceFS?**
JuiceFS is a distributed file system. The latency of metadata is determined by 1 to 2 network round trip(s) between the mount point and metadata service (generally 1-3 ms), and the latency of data depends on the object storage latency (generally 20-100ms). The throughput of sequential read and write can reachup to 2800 MiB/s (see Benchmark with fio), depending on the network bandwidth and whether the data can be easily compressed.
JuiceFS has a built-in multi-level cache (invalidated automatically). Once the cache is warmed up, latency and throughput can be very close to a local file system (although the use of FUSE may bring a small amount of overhead).
# Table Of Contents
* [Install JuiceFS binary](#install-juicefs-binary)
* [Upgrade JuiceFS binary](#upgrade-juicefs-binary)
* [Setup JuiceFS logrotation](#setup-juicefs-logrotation)
* [Format Cloudflare R2 S3 Storage](#format-cloudflare-r2-s3-storage)
* [Mount the JuiceFS Formatted R2 S3 Storage](#mount-the-juicefs-formatted-r2-s3-storage)
* [Manual Mount](#manual-mount)
* [systemd service Mount](#systemd-service-mount)
* [Setup JuiceFS S3 Gateway](#setup-juicefs-s3-gateway)
* [Manually Starting JuiceFS S3 Gateway](#manually-starting-juicefs-s3-gateway)
* [systemd service Starting JuiceFS S3 Gateway](#systemd-service-starting-juicefs-s3-gateway)
* [Working With Cloudflare R2 S3 Mounted Directory and JuiceFS S3 Gateway](#working-with-cloudflare-r2-s3-mounted-directory-and-juicefs-s3-gateway)
* [Mount Info](#mount-info)
* [Inspecting JuiceFS metadata engine status](#inspecting-juicefs-metadata-engine-status)
* [Warmup Local Cache](#warmup-local-cache)
* [Check Disk Size](#check-disk-size)
* [JuiceFS Benchmarks](#juicefs-benchmarks)
* [Redis Metadata Cache + Sharded R2 Mount On Intel Xeon E-2276G 6C/12T, 32GB memory and 2x 960GB NVMe raid 1](#redis-metadata-cache--sharded-r2-mount-on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1)
* [JuiceFS Benchmarks 100x R2 Sharded Mount + Redis Metadata Caching](#juicefs-benchmarks-100x-r2-sharded-mount--redis-metadata-caching)
* [JuiceFS Benchmarks 61x R2 Sharded Mount + Redis Metadata Caching](#juicefs-benchmarks-61x-r2-sharded-mount--redis-metadata-caching)
* [JuiceFS Benchmarks 21x R2 Sharded Mount + Redis Metadata Caching](#juicefs-benchmarks-21x-r2-sharded-mount--redis-metadata-caching)
* [JuiceFS Benchmarks 10x R2 Sharded Mount + Redis Metadata Caching](#juicefs-benchmarks-10x-r2-sharded-mount--redis-metadata-caching)
* [Sharded R2 Mount On Intel Xeon E-2276G 6C/12T, 32GB memory and 2x 960GB NVMe raid 1](#sharded-r2-mount-on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1)
* [10x Cloudflare R2 sharded JuiceFS mount](#10x-r2-sharded-juicefs-mount)
* [5x Cloudflare R2 sharded JuiceFS mount](#5x-r2-sharded-juicefs-mount)
* [On Intel Xeon E-2276G 6C/12T, 32GB memory and 2x 960GB NVMe raid 1](#on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1)
* [with R2 bucket created with location hint North American East](#with-r2-bucket-created-with-location-hint-north-american-east)
* [with R2 bucket created with location hint North American West](#with-r2-bucket-created-with-location-hint-north-american-west)
* [with R2 bucket created on server](#with-r2-bucket-created-on-server)
* [File copy tests](#file-copy-tests)
* [fio test for E-2276G server](#fio-test-for-e-2276g-server)
* [On Intel Core i7 4790K 4C/8T, 32GB memory and 2x 240GB SSD raid 1](#on-intel-core-i7-4790k-4c8t-32gb-memory-and-2x-240gb-ssd-raid-1)
* [fio tests](#fio-test)
* [Destroying JuiceFS Filesystem](#destroying-juicefs-filesystem)
* [Backup JuiceFS Metadata Script](#backup-juicefs-metadata-script)
* [JuiceFS Backup Metadata Cronjob](#juicefs-backup-metadata-cronjob)
# Install JuiceFS binary
```
cd /svr-setup
JFS_LATEST_TAG=$(curl -s https://api.github.com/repos/juicedata/juicefs/releases/latest | grep 'tag_name' | cut -d '"' -f 4 | tr -d 'v')
wget "https://github.com/juicedata/juicefs/releases/download/v${JFS_LATEST_TAG}/juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" -O juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz
tar -zxf "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz"
install juicefs /usr/local/bin
\cp -af /usr/local/bin/juicefs /sbin/mount.juicefs
```
```
juicefs -V
juicefs version 1.0.0-beta3+2022-05-05.0fb9155
```
```
juicefs --help
NAME:
juicefs - A POSIX file system built on Redis and object storage.
USAGE:
juicefs [global options] command [command options] [arguments...]
VERSION:
1.0.0-beta3+2022-05-05.0fb9155
COMMANDS:
ADMIN:
format Format a volume
config Change configuration of a volume
destroy Destroy an existing volume
gc Garbage collector of objects in data storage
fsck Check consistency of a volume
dump Dump metadata into a JSON file
load Load metadata from a previously dumped JSON file
INSPECTOR:
status Show status of a volume
stats Show real time performance statistics of JuiceFS
profile Show profiling of operations completed in JuiceFS
info Show internal information of a path or inode
SERVICE:
mount Mount a volume
umount Unmount a volume
gateway Start an S3-compatible gateway
webdav Start a WebDAV server
TOOL:
bench Run benchmark on a path
warmup Build cache for target directories/files
rmr Remove directories recursively
sync Sync between two storages
GLOBAL OPTIONS:
--verbose, --debug, -v enable debug log (default: false)
--quiet, -q only warning and errors (default: false)
--trace enable trace log (default: false)
--no-agent disable pprof (:6060) and gops (:6070) agent (default: false)
--no-color disable colors (default: false)
--help, -h show help (default: false)
--version, -V print only the version (default: false)
COPYRIGHT:
Apache License 2.0
```
# Upgrade JuiceFS Binary
Following instructions for upgrading JuiceFS client [here](https://github.com/juicedata/juicefs/blob/main/docs/en/faq.md#how-to-upgrade-juicefs-client) involves:
1. Unmounting the JuiceFS mount. If you setup using [systemd JuiceFS service file](#mount-the-juicefs-formatted-r2-s3-storage), then it's just a service stop for it and the [JuiceFS S3 Gateway service](#systemd-service-starting-juicefs-s3-gateway).
Upgrading to [JuiceFS v1.0.0](https://github.com/juicedata/juicefs/releases/tag/v1.0.0):
```
systemctl stop juicefs.service juicefs-gateway.service
```
2. Updating JuiceFS binary
```
cd /svr-setup
JFS_LATEST_TAG=$(curl -s https://api.github.com/repos/juicedata/juicefs/releases/latest | grep 'tag_name' | cut -d '"' -f 4 | tr -d 'v')
wget "https://github.com/juicedata/juicefs/releases/download/v${JFS_LATEST_TAG}/juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" -O juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz
tar -zxf "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz"
install juicefs /usr/local/bin
\cp -af /usr/local/bin/juicefs /sbin/mount.juicefs
```
3. Starting JuiceFS and JuiceFS S3 Gateway services
```
systemctl start juicefs.service juicefs-gateway.service
systemctl status juicefs.service juicefs-gateway.service --no-pager
```
4. Checking updated JuiceFS binary and mount.
```
juicefs -V
juicefs version 1.0.4+2023-04-06.f1c475d
df -hT /home/juicefs_mount
Filesystem Type Size Used Avail Use% Mounted on
JuiceFS:myjuicefs fuse.juicefs 1.0P 0 1.0P 0% /home/juicefs_mount
```
```
mkdir -p /home/juicefs
cd /home/juicefs
juicefs status sqlite3:///home/juicefs/myjuicefs.db
2022/06/21 13:54:45.570232 juicefs[28472] : Meta address: sqlite3:///home/juicefs/myjuicefs.db [interface.go:397]
{
"Setting": {
"Name": "myjuicefs",
"UUID": "2109366a-5f4f-4449-8723-dfec21f48e8f",
"Storage": "s3",
"Bucket": "https://juicefs.cfaccountid.r2.cloudflarestorage.com",
"AccessKey": "cfaccesskey",
"SecretKey": "removed",
"BlockSize": 4096,
"Compression": "none",
"Shards": 0,
"HashPrefix": false,
"Capacity": 0,
"Inodes": 0,
"KeyEncrypted": true,
"TrashDays": 0,
"MetaVersion": 1,
"MinClientVersion": "",
"MaxClientVersion": ""
},
"Sessions": [
{
"Sid": 19,
"Expire": "2022-08-12T12:58:32Z",
"Version": "1.0.4+2023-04-06.f1c475d",
"HostName": "host.domain.com",
"MountPoint": "/home/juicefs_mount",
"ProcessID": 28376
},
{
"Sid": 20,
"Expire": "2022-08-12T12:58:32Z",
"Version": "1.0.4+2023-04-06.f1c475d",
"HostName": "host.domain.com",
"MountPoint": "s3gateway",
"ProcessID": 28387
}
]
}
```
## Setup JuiceFS logrotation
```
cat > "/etc/logrotate.d/juicefs" <= 524288000 are rotated earlier, old logs are removed
considering log /var/log/juicefs.log
log does not need rotating (log has been rotated at 2022-5-25 3:0, that is not day ago yet)
```
# Format Cloudflare R2 S3 Storage
Fill in variables for your Cloudflare account id, R2 bucket access key and secret key and the R2 bucket name - create the R2 bucket before hand. The sqlite3 database will be saved at `/home/juicefs/myjuicefs.db`.
* JuiceFS supports compression algorithms which can be enabled via `--compress` option which have 3 available options - lz4, zstd or none (default).
* `--trash-days` - number of days after which removed files will be permanently deleted. Default = 1.
* `--block-size` - size of block in KiB
* Other various format options listed at https://juicefs.com/docs/community/command_reference#options.
```
cfaccountid='CF_ACCOUNT_ID'
cfaccesskey=''
cfsecretkey=''
cfbucketname='juicefs'
mkdir -p /home/juicefs
cd /home/juicefs
juicefs format --storage s3 \
--bucket https://${cfbucketname}.${cfaccountid}.r2.cloudflarestorage.com \
--access-key $cfaccesskey \
--secret-key $cfsecretkey \
--compress none \
--trash-days 0 \
--block-size 4096 \
sqlite3:///home/juicefs/myjuicefs.db myjuicefs
```
# Mount the JuiceFS Formatted R2 S3 Storage
Create the mount directory and cache directories.
```
mkdir -p /home/juicefs_mount /home/juicefs_cache
```
## Manual Mount
There are additional JuiceFS mounting options outlined at https://juicefs.com/docs/community/command_reference#options-1
Manually mount the R2 S3 storage at `/home/juicefs_mount`. Note for Cloudflare R2, you need to [disable automatica metadata backups](https://juicefs.com/docs/community/how_to_setup_object_storage/#r2) with `--backup-meta 0`. Otherwise, for non-R2 S3 providers, you can set `--backup-meta 1h`.
```
juicefs mount sqlite3:///home/juicefs/myjuicefs.db /home/juicefs_mount \
--cache-dir /home/juicefs_cache \
--cache-size 102400 \
--buffer-size 2048 \
--open-cache 0 \
--attr-cache 1 \
--entry-cache 1 \
--dir-entry-cache 1 \
--cache-partial-only false \
--free-space-ratio 0.1 \
--writeback \
--no-usage-report \
--max-uploads 20 \
--max-deletes 10 \
--backup-meta 0 \
--log /var/log/juicefs.log \
--get-timeout 300 \
--put-timeout 900 \
--io-retries 90 \
--prefetch 1 -d
```
Note: As `--backup-meta 0` is set for Cloudflare R2 to disable automatic metadata backups, you can manually run the backup command to backup to a file i.e. `meta-dump.json`:
```
juicefs dump sqlite3:///home/juicefs/myjuicefs.db meta-dump.json
```
## systemd service Mount
Or instead of manually mounting, setup systemd service file to manage mounting and unmounting the directory
`/usr/lib/systemd/system/juicefs.service`
```
[Unit]
Description=JuiceFS
AssertPathIsDirectory=/home/juicefs_mount
After=network-online.target
[Service]
Type=simple
WorkingDirectory=/home/juicefs
ExecStart=/usr/local/bin/juicefs mount \
"sqlite3:///home/juicefs/myjuicefs.db" \
/home/juicefs_mount \
--no-usage-report \
--writeback \
--cache-size 102400 \
--cache-dir /home/juicefs_cache \
--buffer-size 2048 \
--open-cache 0 \
--attr-cache 1 \
--entry-cache 1 \
--dir-entry-cache 1 \
--cache-partial-only false \
--free-space-ratio 0.1 \
--max-uploads 20 \
--max-deletes 10 \
--backup-meta 0 \
--log /var/log/juicefs.log \
--get-timeout 300 \
--put-timeout 900 \
--io-retries 90 \
--prefetch 1
ExecStop=/usr/local/bin/juicefs umount /home/juicefs_mount
Restart=always
RestartSec=5
[Install]
WantedBy=default.target
```
```
mkdir -p /etc/systemd/system/juicefs.service.d
cat > "/etc/systemd/system/juicefs.service.d/openfileslimit.conf" <: Meta address: sqlite3:///home/juicefs/myjuicefs.db [interface.go:385]
May 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.126772 juicefs[26947] : Data use s3://juicefs/myjuicefs/ [mount.go:289]
May 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.127088 juicefs[26947] : Disk cache (/home/juicefs_cache/3c874e07-a62c-42a9-ae67-5865491dd4a8/): capacity (102400 MB), free ratio (10%), max pending pages (51) [disk_cache.go:90]
May 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.138212 juicefs[26947] : create session 1 OK [base.go:185]
May 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.138802 juicefs[26947] : Prometheus metrics listening on 127.0.0.1:9567 [mount.go:157]
May 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.138890 juicefs[26947] : Mounting volume myjuicefs at /home/juicefs_mount ... [mount_unix.go:177]
May 25 04:26:33 hostname juicefs[26947]: 2022/05/25 04:26:33.628570 juicefs[26947] : OK, myjuicefs is ready at /home/juicefs_mount [mount_unix.go:45]
May 25 04:32:19 hostname juicefs[26947]: 2022/05/25 04:32:19.284310 juicefs[26947] : Secret key is removed for the sake of safety [sql.go:2770]
May 25 04:32:20 hostname juicefs[26947]: 2022/05/25 04:32:20.804652 juicefs[26947] : backup metadata succeed, used 1.527736137s [backup.go:69]
```
Using AWS CLI profile for r2 user to check underlying JuiceFS metadata:
```
cfbucketname='juicefs'
aws s3 ls --recursive --profile r2 --endpoint-url=$url s3://$cfbucketname/
2022-05-25 04:26:25 36 myjuicefs/juicefs_uuid
2022-05-25 04:32:20 598 myjuicefs/meta/dump-2022-05-25-043219.json.gz
```
# Setup JuiceFS S3 Gateway
Setup [JuiceFS S3 Gateway](https://juicefs.com/docs/community/s3_gateway#use-the-aws-cli) and setup AWS CLI profile `juicefs` using my [awscli-get.sh](https://awscli-get.centminmod.com/) script to configure the profile.
Install `awscli-get.sh`:
```
curl -4s https://awscli-get.centminmod.com/awscli-get.sh -o awscli-get.sh
chmod +x awscli-get.sh
```
Change `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD` variable values to your descired S3 Gateway access and secret keys. Make sure they're different from your Cloudflare R2 access and secret key credentials.
Setup AWS CLI profile using `awscli-get.sh`:
```
export MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE
export MINIO_ROOT_PASSWORD=12345678
# https://awscli-get.centminmod.com/
export AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER
export AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD
export AWS_DEFAULT_REGION=auto
export AWS_DEFAULT_OUTPUT=text
./awscli-get.sh install juicefs
```
Example output from [awscli-get.sh](https://awscli-get.centminmod.com/) script installing AWS CLI profile named `juicefs`:
```
./awscli-get.sh install juicefs
existing config file detected: /root/.aws/config
existing credential file detected: /root/.aws/credentials
configure aws-cli profile: juicefs
configure aws cli for Cloudflare R2
aws configure set s3.max_concurrent_requests 2 --profile juicefs
aws configure set s3.multipart_threshold 50MB --profile juicefs
aws configure set s3.multipart_chunksize 50MB --profile juicefs
aws configure set s3.addressing_style path --profile juicefs
aws-cli profile: juicefs set:
aws_access_key_id: AKIAIOSFODNN7EXAMPLE
aws_secret_access_key: 12345678
default.region: auto
default output format: text
list aws-cli profiles:
default
r2
juicefs
```
## Manually Starting JuiceFS S3 Gateway
Manually starting created JuiceFS S3 Gateway. Note for Cloudflare R2, you need to [disable automatica metadata backups](https://juicefs.com/docs/community/how_to_setup_object_storage/#r2) with `--backup-meta 0`. Otherwise, for non-R2 S3 providers, you can set `--backup-meta 1h`.
Private local access only:
```
# local private access
juicefs gateway \
--cache-dir /home/juicefs_cache \
--cache-size 102400 \
--attr-cache 1 \
--entry-cache 0 \
--dir-entry-cache 1 \
--prefetch 1 \
--free-space-ratio 0.1 \
--writeback \
--backup-meta 0 \
--no-usage-report \
--buffer-size 2048 sqlite3:///home/juicefs/myjuicefs.db localhost:3777
```
Public net accessible mode:
```
# public access
juicefs gateway \
--cache-dir /home/juicefs_cache \
--cache-size 102400 \
--attr-cache 1 \
--entry-cache 0 \
--dir-entry-cache 1 \
--prefetch 1 \
--free-space-ratio 0.1 \
--writeback \
--backup-meta 0 \
--no-usage-report \
--buffer-size 2048 sqlite3:///home/juicefs/myjuicefs.db 0.0.0.0:3777
```
## systemd service Starting JuiceFS S3 Gateway
Or instead of manually creating JuiceFS S3 Gateway, use systemd service file.
Below is using private local access only.
`/usr/lib/systemd/system/juicefs-gateway.service`
```
[Unit]
Description=JuiceFS Gateway
After=network-online.target
[Service]
Environment='MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE'
Environment='MINIO_ROOT_PASSWORD=12345678'
Type=simple
WorkingDirectory=/home/juicefs
ExecStart=/usr/local/bin/juicefs gateway \
--no-usage-report \
--writeback \
--cache-size 102400 \
--cache-dir /home/juicefs_cache \
--attr-cache 1 \
--entry-cache 0 \
--dir-entry-cache 1 \
--prefetch 1 \
--free-space-ratio 0.1 \
--max-uploads 20 \
--max-deletes 10 \
--backup-meta 0 \
--get-timeout 300 \
--put-timeout 900 \
--io-retries 90 \
--buffer-size 2048 \
"sqlite3:///home/juicefs/myjuicefs.db" \
localhost:3777
Restart=always
RestartSec=5
[Install]
WantedBy=default.target
```
```
mkdir -p /etc/systemd/system/juicefs-gateway.service.d
cat > "/etc/systemd/system/juicefs-gateway.service.d/openfileslimit.conf" <: Prometheus metrics listening on 127.0.0.1:10037 [mount.go:157]
May 25 04:26:33 hostname juicefs[26957]: Endpoint: http://localhost:3777
May 25 04:26:33 hostname juicefs[26957]: Browser Access:
May 25 04:26:33 hostname juicefs[26957]: http://localhost:3777
May 25 04:26:33 hostname juicefs[26957]: Object API (Amazon S3 compatible):
May 25 04:26:33 hostname juicefs[26957]: Go: https://docs.min.io/docs/golang-client-quickstart-guide
May 25 04:26:33 hostname juicefs[26957]: Java: https://docs.min.io/docs/java-client-quickstart-guide
May 25 04:26:33 hostname juicefs[26957]: Python: https://docs.min.io/docs/python-client-quickstart-guide
May 25 04:26:33 hostname juicefs[26957]: JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
May 25 04:26:33 hostname juicefs[26957]: .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
```
# Working With Cloudflare R2 S3 Mounted Directory and JuiceFS S3 Gateway
Using AWS CLI `r2` profile to inspect underlying JuiceFS metadata engine data.
```
url=${cfaccountid}.r2.cloudflarestorage.com
echo 1 > /home/juicefs_mount/file1.txt
aws s3 ls --recursive --profile r2 --endpoint-url=$url s3://$cfbucketname/
2022-05-25 04:48:46 2 myjuicefs/chunks/0/0/1_0_2
2022-05-25 04:26:25 36 myjuicefs/juicefs_uuid
2022-05-25 04:32:20 598 myjuicefs/meta/dump-2022-05-25-043219.json.gz
```
Using AWS CLI `juicefs` profile to inspect the JuiceFS S3 Gateway.
```
aws --endpoint-url http://localhost:3777 s3 ls --recursive myjuicefs
2022-05-25 04:48:45 2 file1.txt
```
## Mount Info
```
juicefs info /home/juicefs_mount/
/home/juicefs_mount/ :
inode: 1
files: 1
dirs: 1
length: 2
size: 8192
```
## Inspecting JuiceFS metadata engine status
```
juicefs status sqlite3:///home/juicefs/myjuicefs.db
2022/05/25 04:50:06.356669 juicefs[33155] : Meta address: sqlite3:///home/juicefs/myjuicefs.db [interface.go:385]
{
"Setting": {
"Name": "myjuicefs",
"UUID": "3c874e07-a62c-42a9-ae67-5865491dd4a8",
"Storage": "s3",
"Bucket": "https://juicefs.cfaccountid.r2.cloudflarestorage.com",
"AccessKey": "cfaccesskey",
"SecretKey": "removed",
"BlockSize": 4096,
"Compression": "none",
"Shards": 0,
"HashPrefix": false,
"Capacity": 0,
"Inodes": 0,
"KeyEncrypted": true,
"TrashDays": 1,
"MetaVersion": 1,
"MinClientVersion": "",
"MaxClientVersion": ""
},
"Sessions": [
{
"Sid": 1,
"Expire": "2022-05-25T04:50:59Z",
"Version": "1.0.0-beta3+2022-05-05.0fb9155",
"HostName": "host.domain.com",
"MountPoint": "/home/juicefs_mount",
"ProcessID": 26947
},
{
"Sid": 2,
"Expire": "2022-05-25T04:51:03Z",
"Version": "1.0.0-beta3+2022-05-05.0fb9155",
"HostName": "host.domain.com",
"MountPoint": "s3gateway",
"ProcessID": 26957
}
]
}
```
## Warmup Local Cache
```
juicefs warmup -p 2 /home/juicefs_mount
Warmed up paths count: 1 / 1 [==============================================================] done
2022/05/25 05:29:18.497915 juicefs[43684] : Successfully warmed up 1 paths [warmup.go:209]
```
## Check Disk Size
```
df -hT /home/juicefs_mount
Filesystem Type Size Used Avail Use% Mounted on
JuiceFS:myjuicefs fuse.juicefs 1.0P 4.0K 1.0P 1% /home/juicefs_mount
```
## Metrics
```
curl -s http://localhost:9567/metrics
```
checking blockcache metrics
```
curl -s http://localhost:9567/metrics | grep blockcache | egrep -v '\#|hist'
juicefs_blockcache_blocks{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_blockcache_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_blockcache_drops{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_blockcache_evicts{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_blockcache_hit_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.62144e+07
juicefs_blockcache_hits{mp="/home/juicefs_mount",vol_name="myjuicefs"} 200
juicefs_blockcache_miss{mp="/home/juicefs_mount",vol_name="myjuicefs"} 647
juicefs_blockcache_miss_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.680160256e+09
juicefs_blockcache_write_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.173698048e+09
juicefs_blockcache_writes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 712
```
filtered metrics
```
curl -s http://localhost:9567/metrics | egrep -v '\#|hist|bucket'
juicefs_blockcache_blocks{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_blockcache_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_blockcache_drops{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_blockcache_evicts{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_blockcache_hit_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.62144e+07
juicefs_blockcache_hits{mp="/home/juicefs_mount",vol_name="myjuicefs"} 200
juicefs_blockcache_miss{mp="/home/juicefs_mount",vol_name="myjuicefs"} 647
juicefs_blockcache_miss_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.680160256e+09
juicefs_blockcache_write_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.173698048e+09
juicefs_blockcache_writes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 712
juicefs_cpu_usage{mp="/home/juicefs_mount",vol_name="myjuicefs"} 21.072261
juicefs_fuse_open_handlers{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_fuse_read_size_bytes_sum{mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.173698048e+09
juicefs_fuse_read_size_bytes_count{mp="/home/juicefs_mount",vol_name="myjuicefs"} 16584
juicefs_fuse_written_size_bytes_sum{mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.173698048e+09
juicefs_fuse_written_size_bytes_count{mp="/home/juicefs_mount",vol_name="myjuicefs"} 16584
juicefs_go_build_info{checksum="",mp="/home/juicefs_mount",path="github.com/juicedata/juicefs",version="(devel)",vol_name="myjuicefs"} 1
juicefs_go_gc_duration_seconds{mp="/home/juicefs_mount",vol_name="myjuicefs",quantile="0"} 2.4418e-05
juicefs_go_gc_duration_seconds{mp="/home/juicefs_mount",vol_name="myjuicefs",quantile="0.25"} 4.3148e-05
juicefs_go_gc_duration_seconds{mp="/home/juicefs_mount",vol_name="myjuicefs",quantile="0.5"} 5.6996e-05
juicefs_go_gc_duration_seconds{mp="/home/juicefs_mount",vol_name="myjuicefs",quantile="0.75"} 0.000106379
juicefs_go_gc_duration_seconds{mp="/home/juicefs_mount",vol_name="myjuicefs",quantile="1"} 0.000342952
juicefs_go_gc_duration_seconds_sum{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0.001999786
juicefs_go_gc_duration_seconds_count{mp="/home/juicefs_mount",vol_name="myjuicefs"} 22
juicefs_go_goroutines{mp="/home/juicefs_mount",vol_name="myjuicefs"} 62
juicefs_go_info{mp="/home/juicefs_mount",version="go1.17.8",vol_name="myjuicefs"} 1
juicefs_go_memstats_alloc_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.8662952e+07
juicefs_go_memstats_alloc_bytes_total{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.377878736e+09
juicefs_go_memstats_buck_hash_sys_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.537716e+06
juicefs_go_memstats_frees_total{mp="/home/juicefs_mount",vol_name="myjuicefs"} 4.703242e+06
juicefs_go_memstats_gc_cpu_fraction{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.1818653907586683e-05
juicefs_go_memstats_gc_sys_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 4.8828976e+07
juicefs_go_memstats_heap_alloc_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.8662952e+07
juicefs_go_memstats_heap_idle_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.28196608e+09
juicefs_go_memstats_heap_inuse_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 3.3079296e+07
juicefs_go_memstats_heap_objects{mp="/home/juicefs_mount",vol_name="myjuicefs"} 53970
juicefs_go_memstats_heap_released_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.278754816e+09
juicefs_go_memstats_heap_sys_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.315045376e+09
juicefs_go_memstats_last_gc_time_seconds{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.6535426808430629e+09
juicefs_go_memstats_lookups_total{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_go_memstats_mallocs_total{mp="/home/juicefs_mount",vol_name="myjuicefs"} 4.757212e+06
juicefs_go_memstats_mcache_inuse_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 9600
juicefs_go_memstats_mcache_sys_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 16384
juicefs_go_memstats_mspan_inuse_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 312256
juicefs_go_memstats_mspan_sys_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.736128e+06
juicefs_go_memstats_next_gc_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 5.738088e+07
juicefs_go_memstats_other_sys_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.769556e+06
juicefs_go_memstats_stack_inuse_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.0354688e+07
juicefs_go_memstats_stack_sys_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.0354688e+07
juicefs_go_memstats_sys_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.380288824e+09
juicefs_go_threads{mp="/home/juicefs_mount",vol_name="myjuicefs"} 271
juicefs_memory{mp="/home/juicefs_mount",vol_name="myjuicefs"} 9.64608e+07
juicefs_object_request_data_bytes{method="GET",mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.147483648e+09
juicefs_object_request_data_bytes{method="PUT",mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.205155328e+09
juicefs_object_request_errors{mp="/home/juicefs_mount",vol_name="myjuicefs"} 337
juicefs_process_cpu_seconds_total{mp="/home/juicefs_mount",vol_name="myjuicefs"} 21.06
juicefs_process_max_fds{mp="/home/juicefs_mount",vol_name="myjuicefs"} 524288
juicefs_process_open_fds{mp="/home/juicefs_mount",vol_name="myjuicefs"} 23
juicefs_process_resident_memory_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 9.64608e+07
juicefs_process_start_time_seconds{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.65354147984e+09
juicefs_process_virtual_memory_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 2.159013888e+09
juicefs_process_virtual_memory_max_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1.8446744073709552e+19
juicefs_staging_block_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_staging_blocks{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_store_cache_size_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_transaction_restart{mp="/home/juicefs_mount",vol_name="myjuicefs"} 368
juicefs_uptime{mp="/home/juicefs_mount",vol_name="myjuicefs"} 1246.457965465
juicefs_used_buffer_size_bytes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 7.471104e+06
juicefs_used_inodes{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
juicefs_used_space{mp="/home/juicefs_mount",vol_name="myjuicefs"} 0
```
# JuiceFS Benchmarks
## Sharded R2 Mount On Intel Xeon E-2276G 6C/12T, 32GB memory and 2x 960GB NVMe raid 1
The server runs on 2x mismatched 960GB NVMe drives in raid 1 so take note of that for the potential peak read and write performance of the resulting benchmarks:
* Samsung SSD PM983 960GB 2.5 U.2 Gen 3.0 x4 PCIe NVMe
* Up to 3,000MB/s Read, 1,050MB/s Write
* 4K random read/write 400,000/40,000 IOPS
* 1366 TBW / 1.3 DWPD
* Power: 4 Watt (idle) 8.6 Watt (read) 8.1 Watt (write)
* Kingston DC1500M U.2 Enterprise SSD Gen 3.0 x4 PCIe NVME
* Up to 3,100MB/s Read, 1,700MB/s Write
* Steady-state 4k read/write 440,000/150,000 IOPS
* 1681 TBW (1 DWPD/5yrs) (1.6 DWPD/3yrs)
* Power: Idle: 6.30W Average read: 6.21W Average write: 11.40W Max read: 6.60W Max write: 12.24W
The table below shows comparison between [10x Cloudflare R2 sharded JuiceFS mount](#10x-r2-sharded-juicefs-mount) vs [5x Cloudflare R2 sharded JuiceFS mount](#5x-r2-sharded-juicefs-mount) vs [1x Cloudflare JuiceFS mount (default)](#on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1). All R2 storage locations are with location hint North American East.
For 1024MB big file size
| ITEM | VALUE (10x R2 Sharded) | COST (10x R2 Sharded) | VALUE (5x R2 Sharded) | COST (5x R2 Sharded) | VALUE (1x R2 Default) | COST (1x R2 Default) |
| --- | --- | --- | --- | --- | --- | --- |
| Write big file | 906.04 MiB/s | 4.52 s/file | 960.47 MiB/s | 4.26 s/file | 1374.08 MiB/s | 2.98 s/file |
| Read big file | 223.19 MiB/s | 18.35 s/file | 174.17 MiB/s | 23.52 s/file | 152.23 MiB/s | 26.91 s/file |
| Write small file | 701.2 files/s | 5.70 ms/file | 777.4 files/s | 5.15 ms/file | 780.3 files/s | 5.13 ms/file |
| Read small file | 6378.3 files/s | 0.63 ms/file | 7940.0 files/s | 0.50 ms/file | 8000.9 files/s | 0.50 ms/file |
| Stat file | 21123.7 files/s | 0.19 ms/file | 29344.7 files/s | 0.14 ms/file | 27902.2 files/s | 0.14 ms/file |
| FUSE operation | 71555 operations | 2.16 ms/op | 71597 operations | 2.67 ms/op | 71649 operations | 3.06 ms/op |
| Update meta | 6271 operations | 9.01 ms/op | 6041 operations | 4.09 ms/op | 6057 operations | 2.50 ms/op |
| Put object | 1152 operations | 403.23 ms/op | 1136 operations | 428.27 ms/op | 1106 operations | 547.32 ms/op |
| Get object | 1034 operations | 278.61 ms/op | 1049 operations | 299.50 ms/op | 1030 operations | 301.80 ms/op |
| Delete object | 316 operations | 124.32 ms/op | 60 operations | 120.73 ms/op | 29 operations | 234.02 ms/op |
| Write into cache | 1424 operations | 24.92 ms/op | 1424 operations | 83.12 ms/op | 1424 operations | 12.91 ms/op |
| Read from cache | 400 operations | 0.05 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.04 ms/op |
For 1MB big file size
| ITEM | VALUE (10x R2 Sharded) | COST (10x R2 Sharded) | VALUE (5x R2 Sharded) | COST (5x R2 Sharded) | VALUE (1x R2 Default) | COST (1x R2 Default) |
| --- | --- | --- | --- | --- | --- | --- |
| Write big file | 452.66 MiB/s | 0.01 s/file | 448.20 MiB/s | 0.01 s/file | 230.82 MiB/s | 0.02 s/file |
| Read big file | 1545.95 MiB/s | 0.00 s/file | 1376.38 MiB/s | 0.00 s/file | 1276.38 MiB/s | 0.00 s/file |
| Write small file | 682.8 files/s | 5.86 ms/file | 792.5 files/s | 5.05 ms/file | 675.7 files/s | 5.92 ms/file |
| Read small file | 6299.4 files/s | 0.63 ms/file | 7827.1 files/s | 0.51 ms/file | 7833.1 files/s | 0.51 ms/file |
| Stat file | 21365.2 files/s | 0.19 ms/file | 24308.1 files/s | 0.16 ms/file | 28226.1 files/s | 0.14 ms/file |
| FUSE operation | 5757 operations | 0.42 ms/op | 5750 operations | 0.38 ms/op | 5756 operations | 0.41 ms/op |
| Update meta | 5814 operations | 0.72 ms/op | 5740 operations | 0.74 ms/op | 5770 operations | 0.70 ms/op |
| Put object | 107 operations | 282.68 ms/op | 94 operations | 286.35 ms/op | 118 operations | 242.35 ms/op |
| Get object | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op |
| Delete object | 133 operations | 116.84 ms/op | 59 operations | 117.93 ms/op | 95 operations | 83.94 ms/op |
| Write into cache | 404 operations | 0.12 ms/op | 404 operations | 0.12 ms/op | 404 operations | 0.14 ms/op |
| Read from cache | 408 operations | 0.06 ms/op | 408 operations | 0.05 ms/op | 408 operations | 0.06 ms/op |
### 10x R2 Sharded JuiceFS Mount
Benchmark with [`--shard`](https://juicefs.com/docs/community/how_to_setup_object_storage#enable-data-sharding) mount option for [sharded Cloudflare R2 mounted JuiceFS](https://juicefs.com/docs/community/how_to_setup_object_storage#enable-data-sharding) over 10x sharded R2 object storage locations - `juicefs-shard-0`,`juicefs-shard-`,`juicefs-shard-1`,`juicefs-shard-3`, `juicefs-shard-4`, `juicefs-shard-5`, `juicefs-shard-6`, `juicefs-shard-7`, `juicefs-shard-8`, `juicefs-shard-9` with location hint North American East.
```
cfaccountid='CF_ACCOUNT_ID'
cfaccesskey=''
cfsecretkey=''
cfbucketname='juicefs-shard'
mkdir -p /home/juicefs
cd /home/juicefs
juicefs format --storage s3 \
--shards 10 \
--bucket https://${cfbucketname}-%d.${cfaccountid}.r2.cloudflarestorage.com \
--access-key $cfaccesskey \
--secret-key $cfsecretkey \
--compress none \
--trash-days 0 \
--block-size 4096 \
sqlite3:///home/juicefs/myjuicefs.db myjuicefs
```
JuiceFS 10x sharded Cloudflare R2 benchmark with location hint North American East and 1024MB big file size.
```
juicefs bench -p 4 /home/juicefs_mount/
Write big blocks count: 4096 / 4096 [===========================================================] done
Read big blocks count: 4096 / 4096 [===========================================================] done
Write small blocks count: 400 / 400 [=============================================================] done
Read small blocks count: 400 / 400 [=============================================================] done
Stat small files count: 400 / 400 [=============================================================] done
Benchmark finished!
BlockSize: 1 MiB, BigFileSize: 1024 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 4
Time used: 25.4 s, CPU: 127.8%, Memory: 1742.7 MiB
+------------------+------------------+--------------+
| ITEM | VALUE | COST |
+------------------+------------------+--------------+
| Write big file | 906.04 MiB/s | 4.52 s/file |
| Read big file | 223.19 MiB/s | 18.35 s/file |
| Write small file | 701.2 files/s | 5.70 ms/file |
| Read small file | 6378.3 files/s | 0.63 ms/file |
| Stat file | 21123.7 files/s | 0.19 ms/file |
| FUSE operation | 71555 operations | 2.16 ms/op |
| Update meta | 6271 operations | 9.01 ms/op |
| Put object | 1152 operations | 403.23 ms/op |
| Get object | 1034 operations | 278.61 ms/op |
| Delete object | 316 operations | 124.32 ms/op |
| Write into cache | 1424 operations | 24.92 ms/op |
| Read from cache | 400 operations | 0.05 ms/op |
+------------------+------------------+--------------+
```
JuiceFS 10x sharded Cloudflare R2 benchmark with location hint North American East and 1MB big file size.
```
juicefs bench -p 4 /home/juicefs_mount/ --big-file-size 1
Write big blocks count: 4 / 4 [==============================================================] done
Read big blocks count: 4 / 4 [==============================================================] done
Write small blocks count: 400 / 400 [=============================================================] done
Read small blocks count: 400 / 400 [=============================================================] done
Stat small files count: 400 / 400 [=============================================================] done
Benchmark finished!
BlockSize: 1 MiB, BigFileSize: 1 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 4
Time used: 1.7 s, CPU: 97.8%, Memory: 1946.3 MiB
+------------------+-----------------+--------------+
| ITEM | VALUE | COST |
+------------------+-----------------+--------------+
| Write big file | 452.66 MiB/s | 0.01 s/file |
| Read big file | 1545.95 MiB/s | 0.00 s/file |
| Write small file | 682.8 files/s | 5.86 ms/file |
| Read small file | 6299.4 files/s | 0.63 ms/file |
| Stat file | 21365.2 files/s | 0.19 ms/file |
| FUSE operation | 5757 operations | 0.42 ms/op |
| Update meta | 5814 operations | 0.72 ms/op |
| Put object | 107 operations | 282.68 ms/op |
| Get object | 0 operations | 0.00 ms/op |
| Delete object | 133 operations | 116.84 ms/op |
| Write into cache | 404 operations | 0.12 ms/op |
| Read from cache | 408 operations | 0.06 ms/op |
+------------------+-----------------+--------------+
```
Inspecting actual JuiceFS mount
```
ls -lah /home/juicefs_mount
total 9.5K
drwxrwxrwx 2 root root 4.0K May 24 21:09 .
drwxr-xr-x. 13 root root 4.0K May 21 18:58 ..
-r-------- 1 root root 0 May 24 21:04 .accesslog
-r-------- 1 root root 1.3K May 24 21:04 .config
-r--r--r-- 1 root root 0 May 24 21:04 .stats
dr-xr-xr-x 2 root root 0 May 24 21:04 .trash
```
JuiceFS mount stats
```
cat /home/juicefs_mount/.stats
juicefs_blockcache_blocks 0
juicefs_blockcache_bytes 0
juicefs_blockcache_drops 0
juicefs_blockcache_evicts 0
juicefs_blockcache_hit_bytes 436207616
juicefs_blockcache_hits 3232
juicefs_blockcache_miss 4241
juicefs_blockcache_miss_bytes 17720934400
juicefs_blockcache_read_hist_seconds_total 3232
juicefs_blockcache_read_hist_seconds_sum 0.16809886599999965
juicefs_blockcache_write_bytes 17616076800
juicefs_blockcache_write_hist_seconds_total 7312
juicefs_blockcache_write_hist_seconds_sum 391.8615498160001
juicefs_blockcache_writes 7312
juicefs_compact_size_histogram_bytes_total 0
juicefs_compact_size_histogram_bytes_sum 0
juicefs_cpu_usage 141.95404200000002
juicefs_fuse_open_handlers 1
juicefs_fuse_ops_durations_histogram_seconds_total 310446
juicefs_fuse_ops_durations_histogram_seconds_sum 897.3332114739511
juicefs_fuse_read_size_bytes_total 134400
juicefs_fuse_read_size_bytes_sum 17616076800
juicefs_fuse_written_size_bytes_total 134400
juicefs_fuse_written_size_bytes_sum 17616076800
juicefs_go_build_info__github.com/juicedata/juicefs_(devel) 1
juicefs_go_goroutines 79
juicefs_go_info_go1.19.2 1
juicefs_go_memstats_alloc_bytes 32336136
juicefs_go_memstats_alloc_bytes_total 2939952080
juicefs_go_memstats_buck_hash_sys_bytes 1805266
juicefs_go_memstats_frees_total 35324415
juicefs_go_memstats_gc_cpu_fraction 0.000014170426577562373
juicefs_go_memstats_gc_sys_bytes 84006704
juicefs_go_memstats_heap_alloc_bytes 32336136
juicefs_go_memstats_heap_idle_bytes 1917329408
juicefs_go_memstats_heap_inuse_bytes 37642240
juicefs_go_memstats_heap_objects 61603
juicefs_go_memstats_heap_released_bytes 1916534784
juicefs_go_memstats_heap_sys_bytes 1954971648
juicefs_go_memstats_last_gc_time_seconds 1684981564.030393
juicefs_go_memstats_lookups_total 0
juicefs_go_memstats_mallocs_total 35386018
juicefs_go_memstats_mcache_inuse_bytes 14400
juicefs_go_memstats_mcache_sys_bytes 15600
juicefs_go_memstats_mspan_inuse_bytes 357136
juicefs_go_memstats_mspan_sys_bytes 10918080
juicefs_go_memstats_next_gc_bytes 64394520
juicefs_go_memstats_other_sys_bytes 2965030
juicefs_go_memstats_stack_inuse_bytes 12156928
juicefs_go_memstats_stack_sys_bytes 12156928
juicefs_go_memstats_sys_bytes 2066839256
juicefs_go_threads 278
juicefs_memory 154210304
juicefs_meta_ops_durations_histogram_seconds_total 33621
juicefs_meta_ops_durations_histogram_seconds_sum 304.82784600500264
juicefs_object_request_data_bytes_GET 17234395136
juicefs_object_request_data_bytes_PUT 17573871616
juicefs_object_request_durations_histogram_seconds_DELETE_total 7312
juicefs_object_request_durations_histogram_seconds_DELETE_sum 894.7742219350012
juicefs_object_request_durations_histogram_seconds_GET_total 4225
juicefs_object_request_durations_histogram_seconds_GET_sum 1226.6492657670003
juicefs_object_request_durations_histogram_seconds_PUT_total 6866
juicefs_object_request_durations_histogram_seconds_PUT_sum 2722.8829050600116
juicefs_object_request_errors 499
juicefs_process_cpu_seconds_total 141.95
juicefs_process_max_fds 524288
juicefs_process_open_fds 15
juicefs_process_resident_memory_bytes 154210304
juicefs_process_start_time_seconds 1684980243.38
juicefs_process_virtual_memory_bytes 2880843776
juicefs_process_virtual_memory_max_bytes 18446744073709552000
juicefs_staging_block_bytes 0
juicefs_staging_blocks 0
juicefs_store_cache_size_bytes 0
juicefs_transaction_durations_histogram_seconds_total 51442
juicefs_transaction_durations_histogram_seconds_sum 140.4373622320012
juicefs_transaction_restart 0
juicefs_uptime 1416.787316108
juicefs_used_buffer_size_bytes 8388608
juicefs_used_inodes 0
juicefs_used_space 0
```
### 5x R2 Sharded JuiceFS Mount
Benchmark with [`--shard`](https://juicefs.com/docs/community/how_to_setup_object_storage#enable-data-sharding) mount option for [sharded Cloudflare R2 mounted JuiceFS](https://juicefs.com/docs/community/how_to_setup_object_storage#enable-data-sharding) over 5x sharded R2 object storage locations - `juicefs-shard-0`,`juicefs-shard-`,`juicefs-shard-1`,`juicefs-shard-3`, and `juicefs-shard-4` with location hint North American East.
```
cfaccountid='CF_ACCOUNT_ID'
cfaccesskey=''
cfsecretkey=''
cfbucketname='juicefs-shard'
mkdir -p /home/juicefs
cd /home/juicefs
juicefs format --storage s3 \
--shards 5 \
--bucket https://${cfbucketname}-%d.${cfaccountid}.r2.cloudflarestorage.com \
--access-key $cfaccesskey \
--secret-key $cfsecretkey \
--compress none \
--trash-days 0 \
--block-size 4096 \
sqlite3:///home/juicefs/myjuicefs.db myjuicefs
```
output
```
2023/05/24 17:45:14.116161 juicefs[3701895] : Meta address: sqlite3:///home/juicefs/myjuicefs.db [interface.go:401]
2023/05/24 17:45:14.117248 juicefs[3701895] : Data use shard5://s3://juicefs-shard-0/myjuicefs/ [format.go:434]
2023/05/24 17:45:18.423901 juicefs[3701895] : Can't list s3://juicefs-shard-0/: InvalidMaxKeys: MaxKeys params must be positive integer <= 1000.
status code: 400, request id: , host id: [sharding.go:85]
2023/05/24 17:45:18.423955 juicefs[3701895] : List storage shard5://s3://juicefs-shard-0/myjuicefs/ failed: list s3://juicefs-shard-0/: InvalidMaxKeys: MaxKeys params must be positive integer <= 1000.
status code: 400, request id: , host id: [format.go:452]
2023/05/24 17:45:18.709793 juicefs[3701895] : Volume is formatted as {
"Name": "myjuicefs",
"UUID": "UUID-UUID-UUID-UUID",
"Storage": "s3",
"Bucket": "https://juicefs-shard-%d.CF_ACCOUNT_ID.r2.cloudflarestorage.com",
"AccessKey": "CF_ACCESS_KEY",
"SecretKey": "removed",
"BlockSize": 4096,
"Compression": "none",
"Shards": 5,
"KeyEncrypted": true,
"MetaVersion": 1
} [format.go:471]
```
JuiceFS mount info
```
juicefs info /home/juicefs_mount/
/home/juicefs_mount/ :
inode: 1
files: 0
dirs: 1
length: 0 Bytes
size: 4.00 KiB (4096 Bytes)
path: /
```
JuiceFS 5x sharded Cloudflare R2 benchmark with location hint North American East and 1024MB big file size.
```
juicefs bench -p 4 /home/juicefs_mount/
Write big blocks count: 4096 / 4096 [===========================================================] done
Read big blocks count: 4096 / 4096 [===========================================================] done
Write small blocks count: 400 / 400 [=============================================================] done
Read small blocks count: 400 / 400 [=============================================================] done
Stat small files count: 400 / 400 [=============================================================] done
Benchmark finished!
BlockSize: 1 MiB, BigFileSize: 1024 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 4
Time used: 30.2 s, CPU: 103.5%, Memory: 1364.5 MiB
+------------------+------------------+--------------+
| ITEM | VALUE | COST |
+------------------+------------------+--------------+
| Write big file | 960.47 MiB/s | 4.26 s/file |
| Read big file | 174.17 MiB/s | 23.52 s/file |
| Write small file | 777.4 files/s | 5.15 ms/file |
| Read small file | 7940.0 files/s | 0.50 ms/file |
| Stat file | 29344.7 files/s | 0.14 ms/file |
| FUSE operation | 71597 operations | 2.67 ms/op |
| Update meta | 6041 operations | 4.09 ms/op |
| Put object | 1136 operations | 428.27 ms/op |
| Get object | 1049 operations | 299.50 ms/op |
| Delete object | 60 operations | 120.73 ms/op |
| Write into cache | 1424 operations | 83.12 ms/op |
| Read from cache | 400 operations | 0.05 ms/op |
+------------------+------------------+--------------+
```
JuiceFS sharded Cloudflare R2 benchmark with location hint North American East and 1MB big file size.
```
juicefs bench -p 4 /home/juicefs_mount/ --big-file-size 1
Write big blocks count: 4 / 4 [==============================================================] done
Read big blocks count: 4 / 4 [==============================================================] done
Write small blocks count: 400 / 400 [=============================================================] done
Read small blocks count: 400 / 400 [=============================================================] done
Stat small files count: 400 / 400 [=============================================================] done
Benchmark finished!
BlockSize: 1 MiB, BigFileSize: 1 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 4
Time used: 1.6 s, CPU: 102.4%, Memory: 164.9 MiB
+------------------+-----------------+--------------+
| ITEM | VALUE | COST |
+------------------+-----------------+--------------+
| Write big file | 448.20 MiB/s | 0.01 s/file |
| Read big file | 1376.38 MiB/s | 0.00 s/file |
| Write small file | 792.5 files/s | 5.05 ms/file |
| Read small file | 7827.1 files/s | 0.51 ms/file |
| Stat file | 24308.1 files/s | 0.16 ms/file |
| FUSE operation | 5750 operations | 0.38 ms/op |
| Update meta | 5740 operations | 0.74 ms/op |
| Put object | 94 operations | 286.35 ms/op |
| Get object | 0 operations | 0.00 ms/op |
| Delete object | 59 operations | 117.93 ms/op |
| Write into cache | 404 operations | 0.12 ms/op |
| Read from cache | 408 operations | 0.05 ms/op |
+------------------+-----------------+--------------+
```
Inspecting Cloudflare R2 sharded storage buckets after JuiceFS benchmark run with location hint North American East
```
aws s3 ls --recursive --profile r2 --endpoint-url=$url s3://juicefs-shard-0
2023-05-24 18:46:01 131072 myjuicefs/chunks/0/0/980_0_131072
2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1146_0_131072
2023-05-24 18:46:30 131072 myjuicefs/chunks/0/1/1540_0_131072
aws s3 ls --recursive --profile r2 --endpoint-url=$url s3://juicefs-shard-1
2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1154_0_131072
2023-05-24 18:46:29 131072 myjuicefs/chunks/0/1/1386_0_131072
2023-05-24 18:46:31 131072 myjuicefs/chunks/0/1/1688_0_131072
2023-05-24 17:45:18 36 myjuicefs/juicefs_uuid
aws s3 ls --recursive --profile r2 --endpoint-url=$url s3://juicefs-shard-2
2023-05-24 17:52:09 131072 myjuicefs/chunks/0/0/574_0_131072
2023-05-24 18:46:01 131072 myjuicefs/chunks/0/1/1000_0_131072
2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1142_0_131072
aws s3 ls --recursive --profile r2 --endpoint-url=$url s3://juicefs-shard-3
2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1130_0_131072
2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1150_0_131072
2023-05-24 18:46:05 131072 myjuicefs/chunks/0/1/1226_0_131072
2023-05-24 18:46:28 131072 myjuicefs/chunks/0/1/1382_0_131072
2023-05-24 18:46:30 131072 myjuicefs/chunks/0/1/1532_0_131072
2023-05-24 18:46:30 131072 myjuicefs/chunks/0/1/1552_0_131072
2023-05-24 18:46:31 131072 myjuicefs/chunks/0/1/1560_0_131072
2023-05-24 18:46:30 131072 myjuicefs/chunks/0/1/1564_0_131072
2023-05-24 18:46:31 131072 myjuicefs/chunks/0/1/1568_0_131072
2023-05-24 18:46:32 131072 myjuicefs/chunks/0/1/1728_0_131072
2023-05-24 17:53:44 581 myjuicefs/meta/dump-2023-05-24-225343.json.gz
aws s3 ls --recursive --profile r2 --endpoint-url=$url s3://juicefs-shard-4
2023-05-24 18:46:01 131072 myjuicefs/chunks/0/0/988_0_131072
2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1134_0_131072
2023-05-24 18:46:03 131072 myjuicefs/chunks/0/1/1138_0_131072
2023-05-24 18:46:28 131072 myjuicefs/chunks/0/1/1390_0_131072
2023-05-24 18:46:28 131072 myjuicefs/chunks/0/1/1394_0_131072
2023-05-24 18:46:30 131072 myjuicefs/chunks/0/1/1556_0_131072
```
### fio sharded Cloudflare R2 test for E-2276G server with location hint North American East
fio Sequential Write
```
mkdir -p /home/juicefs_mount/fio
fio --name=sequential-write --directory=/home/juicefs_mount/fio --rw=write --refill_buffers --bs=4M --size=1G --end_fsync=1
sequential-write: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.19
Starting 1 process
sequential-write: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1)
sequential-write: (groupid=0, jobs=1): err= 0: pid=3704701: Wed May 24 19:01:25 2023
write: IOPS=279, BW=1119MiB/s (1173MB/s)(1024MiB/915msec); 0 zone resets
clat (usec): min=2221, max=7356, avg=2961.60, stdev=807.86
lat (usec): min=2222, max=7357, avg=2962.43, stdev=808.05
clat percentiles (usec):
| 1.00th=[ 2245], 5.00th=[ 2311], 10.00th=[ 2376], 20.00th=[ 2442],
| 30.00th=[ 2540], 40.00th=[ 2638], 50.00th=[ 2704], 60.00th=[ 2802],
| 70.00th=[ 2966], 80.00th=[ 3163], 90.00th=[ 4424], 95.00th=[ 4948],
| 99.00th=[ 5735], 99.50th=[ 6718], 99.90th=[ 7373], 99.95th=[ 7373],
| 99.99th=[ 7373]
bw ( MiB/s): min= 1067, max= 1067, per=95.35%, avg=1067.08, stdev= 0.00, samples=1
iops : min= 266, max= 266, avg=266.00, stdev= 0.00, samples=1
lat (msec) : 4=89.84%, 10=10.16%
cpu : usr=16.19%, sys=38.95%, ctx=8195, majf=0, minf=9
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,256,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=1119MiB/s (1173MB/s), 1119MiB/s-1119MiB/s (1173MB/s-1173MB/s), io=1024MiB (1074MB), run=915-915msec
```
```
ls -lah /home/juicefs_mount/fio
total 1.1G
drwxr-xr-x 2 root root 4.0K May 24 19:01 .
drwxrwxrwx 3 root root 4.0K May 24 19:01 ..
-rw-r--r-- 1 root root 1.0G May 24 19:01 sequential-write.0.0
```
```
juicefs warmup -p 4 /home/juicefs_mount/fio
Warming up count: 5 0.06/s
Warming up bytes: 5.00 GiB (5368709120 Bytes) 57.32 MiB/s
2023/05/24 19:37:02.236625 juicefs[3705549] : Successfully warmed up 5 files (5368709120 bytes) [warmup.go:233]
```
fio Sequential Read
```
fio --name=sequential-read --directory=/home/juicefs_mount/fio --rw=read --refill_buffers --bs=4M --size=1G --numjobs=4
sequential-read: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
...
fio-3.19
Starting 4 processes
Jobs: 4 (f=4): [R(4)][-.-%][r=2270MiB/s][r=567 IOPS][eta 00m:00s]
sequential-read: (groupid=0, jobs=1): err= 0: pid=3705616: Wed May 24 19:37:25 2023
read: IOPS=132, BW=532MiB/s (557MB/s)(1024MiB/1926msec)
clat (usec): min=2368, max=15013, avg=7167.80, stdev=1697.61
lat (usec): min=2368, max=15013, avg=7169.52, stdev=1697.67
clat percentiles (usec):
| 1.00th=[ 2540], 5.00th=[ 5473], 10.00th=[ 5735], 20.00th=[ 6063],
| 30.00th=[ 6390], 40.00th=[ 6652], 50.00th=[ 6915], 60.00th=[ 7242],
| 70.00th=[ 7504], 80.00th=[ 7898], 90.00th=[ 9110], 95.00th=[10421],
| 99.00th=[13304], 99.50th=[13829], 99.90th=[15008], 99.95th=[15008],
| 99.99th=[15008]
bw ( KiB/s): min=457227, max=573440, per=24.57%, avg=534320.67, stdev=66767.53, samples=3
iops : min= 111, max= 140, avg=130.00, stdev=16.46, samples=3
lat (msec) : 4=2.34%, 10=92.19%, 20=5.47%
cpu : usr=0.52%, sys=62.55%, ctx=3056, majf=0, minf=1036
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=256,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
sequential-read: (groupid=0, jobs=1): err= 0: pid=3705617: Wed May 24 19:37:25 2023
read: IOPS=132, BW=531MiB/s (557MB/s)(1024MiB/1929msec)
clat (usec): min=1536, max=18497, avg=7181.80, stdev=1753.73
lat (usec): min=1536, max=18500, avg=7183.40, stdev=1753.80
clat percentiles (usec):
| 1.00th=[ 2343], 5.00th=[ 5211], 10.00th=[ 5669], 20.00th=[ 6063],
| 30.00th=[ 6456], 40.00th=[ 6718], 50.00th=[ 7046], 60.00th=[ 7373],
| 70.00th=[ 7701], 80.00th=[ 8225], 90.00th=[ 8979], 95.00th=[10552],
| 99.00th=[12518], 99.50th=[12649], 99.90th=[18482], 99.95th=[18482],
| 99.99th=[18482]
bw ( KiB/s): min=450877, max=572295, per=24.23%, avg=526742.67, stdev=66141.94, samples=3
iops : min= 110, max= 139, avg=128.33, stdev=15.95, samples=3
lat (msec) : 2=0.78%, 4=2.34%, 10=91.41%, 20=5.47%
cpu : usr=0.47%, sys=62.14%, ctx=3051, majf=0, minf=1037
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=256,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
sequential-read: (groupid=0, jobs=1): err= 0: pid=3705618: Wed May 24 19:37:25 2023
read: IOPS=133, BW=536MiB/s (562MB/s)(1024MiB/1911msec)
clat (usec): min=4751, max=13813, avg=7109.46, stdev=1330.79
lat (usec): min=4754, max=13815, avg=7111.26, stdev=1330.78
clat percentiles (usec):
| 1.00th=[ 5014], 5.00th=[ 5342], 10.00th=[ 5800], 20.00th=[ 6128],
| 30.00th=[ 6390], 40.00th=[ 6652], 50.00th=[ 6849], 60.00th=[ 7111],
| 70.00th=[ 7439], 80.00th=[ 7832], 90.00th=[ 8586], 95.00th=[ 9503],
| 99.00th=[12125], 99.50th=[12518], 99.90th=[13829], 99.95th=[13829],
| 99.99th=[13829]
bw ( KiB/s): min=476279, max=589824, per=25.24%, avg=548858.00, stdev=63028.99, samples=3
iops : min= 116, max= 144, avg=133.67, stdev=15.37, samples=3
lat (msec) : 10=96.48%, 20=3.52%
cpu : usr=0.63%, sys=64.08%, ctx=3023, majf=0, minf=1036
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=256,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
sequential-read: (groupid=0, jobs=1): err= 0: pid=3705619: Wed May 24 19:37:25 2023
read: IOPS=134, BW=536MiB/s (562MB/s)(1024MiB/1910msec)
clat (usec): min=4812, max=13160, avg=7107.62, stdev=1252.07
lat (usec): min=4814, max=13163, avg=7109.17, stdev=1252.09
clat percentiles (usec):
| 1.00th=[ 4883], 5.00th=[ 5473], 10.00th=[ 5669], 20.00th=[ 6063],
| 30.00th=[ 6456], 40.00th=[ 6652], 50.00th=[ 6980], 60.00th=[ 7242],
| 70.00th=[ 7635], 80.00th=[ 7963], 90.00th=[ 8586], 95.00th=[ 9503],
| 99.00th=[11469], 99.50th=[11731], 99.90th=[13173], 99.95th=[13173],
| 99.99th=[13173]
bw ( KiB/s): min=476279, max=598016, per=25.24%, avg=548863.33, stdev=64161.96, samples=3
iops : min= 116, max= 146, avg=133.67, stdev=15.70, samples=3
lat (msec) : 10=96.88%, 20=3.12%
cpu : usr=0.31%, sys=63.75%, ctx=3115, majf=0, minf=1036
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=256,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=2123MiB/s (2227MB/s), 531MiB/s-536MiB/s (557MB/s-562MB/s), io=4096MiB (4295MB), run=1910-1929msec
```
directory fio test
```
ls -lah /home/juicefs_mount/fio
total 5.1G
drwxr-xr-x 2 root root 4.0K May 24 19:08 .
drwxrwxrwx 3 root root 4.0K May 24 19:01 ..
-rw-r--r-- 1 root root 1.0G May 24 19:08 sequential-read.0.0
-rw-r--r-- 1 root root 1.0G May 24 19:08 sequential-read.1.0
-rw-r--r-- 1 root root 1.0G May 24 19:08 sequential-read.2.0
-rw-r--r-- 1 root root 1.0G May 24 19:08 sequential-read.3.0
-rw-r--r-- 1 root root 1.0G May 24 19:01 sequential-write.0.0
```
Checking the S3 Gateway
```
aws --endpoint-url http://localhost:3777 s3 ls --recursive myjuicefs
2023-05-24 19:08:04 1073741824 fio/sequential-read.0.0
2023-05-24 19:08:09 1073741824 fio/sequential-read.1.0
2023-05-24 19:08:07 1073741824 fio/sequential-read.2.0
2023-05-24 19:08:05 1073741824 fio/sequential-read.3.0
2023-05-24 19:01:24 1073741824 fio/sequential-write.0.0
```
## Redis Metadata Cache + Sharded R2 Mount On Intel Xeon E-2276G 6C/12T, 32GB memory and 2x 960GB NVMe raid 1
Switched from local [sqlite3 metadata caching](https://juicefs.com/docs/community/databases_for_metadata#sqlite) to [Redis server metadata caching](https://juicefs.com/docs/community/databases_for_metadata#redis) with 10x sharded R2 object storage buckets and location hint North American East.
On Centmin Mod LEMP stack server rather than use default Redis server on port 6379 which maybe used for other applications in a caching capacity, setup a separate Redis server on port 6479 using [`redis-generator.sh`](https://github.com/centminmod/centminmod-redis).
```
pushd /root/tools
git clone https://github.com/centminmod/centminmod-redis
cd centminmod-redis
```
`redis-generator.sh` options
```
./redis-generator.sh
* Usage: where X equal postive integer for number of redis
servers to create with incrementing TCP redis ports
starting at STARTPORT=6479.
* prep - standalone prep command installs redis-cluster-tool
* prepupdate - standalone prep update command updates redis-cluster-tool
* multi X - no. of standalone redis instances to create
* multi-cache X - no. of standalone redis instances + disable ondisk persistence
* clusterprep X - no. of cluster enabled config instances
* clustermake 6 - to enable cluster mode + create cluster
* clustermake 9 - flag to enable cluster mode + create cluster
* replication X - create redis replication
* replication X 6579 - create replication with custom start port 6579
* replication-cache X - create redis replication + disable ondisk persistence
* replication-cache X 6579 - create replication with custom start port 6579
* delete X - no. of redis instances to delete
* delete X 6579 - no. of redis instances to delete + custom start port 6579
./redis-generator.sh prep
./redis-generator.sh prepupdate
./redis-generator.sh multi X
./redis-generator.sh multi-cache X
./redis-generator.sh clusterprep X
./redis-generator.sh clustermake 6
./redis-generator.sh clustermake 9
./redis-generator.sh replication X
./redis-generator.sh replication X 6579
./redis-generator.sh replication-cache X
./redis-generator.sh replication-cache X 6579
./redis-generator.sh delete X
./redis-generator.sh delete X 6579
```
Create 2x Redis servers on ports 6479 and 6480 by editing script first with `DEBUG_REDISGEN='n'` and then running the multi command:
```
./redis-generator.sh multi 2
```
Outputs the following where:
* /etc/redis6479/redis6479.conf config file
* /etc/redis6480/redis6480.conf config file
```
./redis-generator.sh multi 2
Creating redis servers starting at TCP = 6479...
-------------------------------------------------------
creating redis server: redis6479.service [increment value: 0]
redis TCP port: 6479
create systemd redis6479.service
cp -a /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis6479.service
create /etc/redis6479/redis6479.conf config file
mkdir -p /etc/redis6479
cp -a /etc/redis/redis.conf /etc/redis6479/redis6479.conf
-rw-r----- 1 redis root 92K Apr 18 00:42 /etc/redis6479/redis6479.conf
-rw-r--r-- 1 root root 474 May 25 02:32 /usr/lib/systemd/system/redis6479.service
Created symlink /etc/systemd/system/multi-user.target.wants/redis6479.service → /usr/lib/systemd/system/redis6479.service.
Note: Forwarding request to 'systemctl enable redis6479.service'.
## Redis TCP 6479 Info ##
redis_version:6.2.12
redis_mode:standalone
process_id:3723969
tcp_port:6479
uptime_in_seconds:0
uptime_in_days:0
executable:/etc/redis6479/redis-server
config_file:/etc/redis6479/redis6479.conf
-------------------------------------------------------
creating redis server: redis6480.service [increment value: 1]
redis TCP port: 6480
create systemd redis6480.service
cp -a /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis6480.service
create /etc/redis6480/redis6480.conf config file
mkdir -p /etc/redis6480
cp -a /etc/redis/redis.conf /etc/redis6480/redis6480.conf
-rw-r----- 1 redis root 92K Apr 18 00:42 /etc/redis6480/redis6480.conf
-rw-r--r-- 1 root root 474 May 25 02:32 /usr/lib/systemd/system/redis6480.service
Created symlink /etc/systemd/system/multi-user.target.wants/redis6480.service → /usr/lib/systemd/system/redis6480.service.
Note: Forwarding request to 'systemctl enable redis6480.service'.
## Redis TCP 6480 Info ##
redis_version:6.2.12
redis_mode:standalone
process_id:3724058
tcp_port:6480
uptime_in_seconds:0
uptime_in_days:0
executable:/etc/redis6480/redis-server
config_file:/etc/redis6480/redis6480.conf
```
```
systemctl status redis6479 redis6480
● redis6479.service - Redis persistent key-value database
Loaded: loaded (/usr/lib/systemd/system/redis6479.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/redis6479.service.d
└─limit.conf, user.conf
Active: active (running) since Thu 2023-05-25 02:32:52 CDT; 2min 4s ago
Main PID: 3723969 (redis-server)
Status: "Ready to accept connections"
Tasks: 5 (limit: 203337)
Memory: 2.9M
CGroup: /system.slice/redis6479.service
└─3723969 /etc/redis6479/redis-server 127.0.0.1:6479
May 25 02:32:52 hostname systemd[1]: Starting Redis persistent key-value database...
May 25 02:32:52 hostname systemd[1]: Started Redis persistent key-value database.
● redis6480.service - Redis persistent key-value database
Loaded: loaded (/usr/lib/systemd/system/redis6480.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/redis6480.service.d
└─limit.conf, user.conf
Active: active (running) since Thu 2023-05-25 02:32:52 CDT; 2min 3s ago
Main PID: 3724058 (redis-server)
Status: "Ready to accept connections"
Tasks: 5 (limit: 203337)
Memory: 2.6M
CGroup: /system.slice/redis6480.service
└─3724058 /etc/redis6480/redis-server 127.0.0.1:6480
May 25 02:32:52 hostname systemd[1]: Starting Redis persistent key-value database...
May 25 02:32:52 hostname systemd[1]: Started Redis persistent key-value database.
```
For JuiceFS Redis metadata caching the references in JuiceFS mount formatting command will be for either:
* `redis://:password@localhost:6479/1` or without password `redis://:localhost:6479/1`
* `redis://:password@localhost:6480/1` or without password `redis://:localhost:6480/1`
Set `/etc/redis6479/redis6479.conf` and `/etc/redis6480/redis6480.conf` config files password for Redis and set `maxmemory-policy noeviction` and `appendonly yes`
```
requirepass password
maxmemory-policy noeviction
appendonly yes
```
And set JuiceFS best practices for Redis metadata caching outlined at https://juicefs.com/docs/community/redis_best_practices.
```
grep -C1 -i '^appendfsync' /etc/redis6479/redis6479.conf
# appendfsync always
appendfsync everysec
# appendfsync no
```
Restart Redis servers
```
systemctl restart redis6479 redis6480
```
Format JuiceFS sharded mount to use Redis metadata caching via `redis://:password@localhost:6479/1`
```
cfaccountid='CF_ACCOUNT_ID'
cfaccesskey=''
cfsecretkey=''
cfbucketname='juicefs-shard'
mkdir -p /home/juicefs
cd /home/juicefs
juicefs format --storage s3 \
--shards 10 \
--bucket https://${cfbucketname}-%d.${cfaccountid}.r2.cloudflarestorage.com \
--access-key $cfaccesskey \
--secret-key $cfsecretkey \
--compress none \
--trash-days 0 \
--block-size 4096 \
redis://:password@localhost:6479/1 myjuicefs
```
Outputs
```
2023/05/25 03:12:14.146627 juicefs[3725353] : Meta address: redis://:****@localhost:6479/1 [interface.go:401]
2023/05/25 03:12:14.147407 juicefs[3725353] : Ping redis: 29.563µs [redis.go:2904]
2023/05/25 03:12:14.148376 juicefs[3725353] : Data use shard10://s3://juicefs-shard-0/myjuicefs/ [format.go:434]
2023/05/25 03:12:14.872195 juicefs[3725353] : Can't list s3://juicefs-shard-0/: InvalidMaxKeys: MaxKeys params must be positive integer <= 1000.
status code: 400, request id: , host id: [sharding.go:85]
2023/05/25 03:12:14.872252 juicefs[3725353] : List storage shard10://s3://juicefs-shard-0/myjuicefs/ failed: list s3://juicefs-shard-0/: InvalidMaxKeys: MaxKeys params must be positive integer <= 1000.
status code: 400, request id: , host id: [format.go:452]
2023/05/25 03:12:15.367557 juicefs[3725353] : Volume is formatted as {
"Name": "myjuicefs",
"UUID": "UUID-UUID-UUID-UUID-UUID",
"Storage": "s3",
"Bucket": "https://juicefs-shard-%d.CF_ACCOUNT_ID.r2.cloudflarestorage.com",
"AccessKey": "cfaccesskey",
"SecretKey": "removed",
"BlockSize": 4096,
"Compression": "none",
"Shards": 10,
"KeyEncrypted": true,
"MetaVersion": 1
} [format.go:471]
```
Edit `/usr/lib/systemd/system/juicefs.service`. Note for Cloudflare R2, you need to [disable automatica metadata backups](https://juicefs.com/docs/community/how_to_setup_object_storage/#r2) with `--backup-meta 0`. Otherwise, for non-R2 S3 providers, you can set `--backup-meta 1h`.
```
[Unit]
Description=JuiceFS
AssertPathIsDirectory=/home/juicefs_mount
After=network-online.target
[Service]
Type=simple
WorkingDirectory=/home/juicefs
ExecStart=/usr/local/bin/juicefs mount \
"redis://:password@localhost:6479/1" \
/home/juicefs_mount \
--no-usage-report \
--writeback \
--cache-size 102400 \
--cache-dir /home/juicefs_cache \
--buffer-size 2048 \
--open-cache 0 \
--attr-cache 1 \
--entry-cache 1 \
--dir-entry-cache 1 \
--cache-partial-only false \
--free-space-ratio 0.1 \
--max-uploads 20 \
--max-deletes 10 \
--backup-meta 0 \
--log /var/log/juicefs.log \
--get-timeout 300 \
--put-timeout 900 \
--io-retries 90 \
--prefetch 1
ExecStop=/usr/local/bin/juicefs umount /home/juicefs_mount
Restart=always
RestartSec=5
[Install]
WantedBy=default.target
```
Edit `/usr/lib/systemd/system/juicefs-gateway.service`
```
[Unit]
Description=JuiceFS Gateway
After=network-online.target
[Service]
Environment='MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE'
Environment='MINIO_ROOT_PASSWORD=12345678'
Type=simple
WorkingDirectory=/home/juicefs
ExecStart=/usr/local/bin/juicefs gateway \
--no-usage-report \
--writeback \
--cache-size 102400 \
--cache-dir /home/juicefs_cache \
--attr-cache 1 \
--entry-cache 0 \
--dir-entry-cache 1 \
--prefetch 1 \
--free-space-ratio 0.1 \
--max-uploads 20 \
--max-deletes 10 \
--backup-meta 0 \
--get-timeout 300 \
--put-timeout 900 \
--io-retries 90 \
--buffer-size 2048 \
"redis://:password@localhost:6479/1" \
localhost:3777
Restart=always
RestartSec=5
[Install]
WantedBy=default.target
```
Restart Redis servers
```
systemctl restart juicefs.service juicefs-gateway.service
```
Note: As `--backup-meta 0` is set for Cloudflare R2 to disable automatic metadata backups, you can manually run the backup command to backup to a file i.e. `meta-dump.json`:
```
juicefs dump redis://:password@localhost:6479/1 meta-dump.json
```
```
juicefs dump redis://:password@localhost:6479/1 meta-dump.json
2023/05/26 07:19:58.883775 juicefs[3791823] : Meta address: redis://:****@localhost:6479/1 [interface.go:401]
2023/05/26 07:19:58.884482 juicefs[3791823] : Ping redis: 19.157µs [redis.go:2904]
2023/05/26 07:19:58.884709 juicefs[3791823] : Secret key is removed for the sake of safety [redis.go:3236]
Dumped entries count: 5 / 5 [==============================================================] done
2023/05/26 07:19:58.885830 juicefs[3791823] : Dump metadata into meta-dump.json succeed [dump.go:76]
```
### JuiceFS Benchmarks 100x R2 Sharded Mount + Redis Metadata Caching
Format JuiceFS sharded mount to use Redis metadata caching via `redis://:password@localhost:6479/1`
```
cfaccountid='CF_ACCOUNT_ID'
cfaccesskey=''
cfsecretkey=''
cfbucketname='juicefs-shard'
mkdir -p /home/juicefs
cd /home/juicefs
juicefs format --storage s3 \
--shards 100 \
--bucket https://${cfbucketname}-%d.${cfaccountid}.r2.cloudflarestorage.com \
--access-key $cfaccesskey \
--secret-key $cfsecretkey \
--compress none \
--trash-days 0 \
--block-size 4096 \
redis://:password@localhost:6479/1 myjuicefs
```
Edit `/usr/lib/systemd/system/juicefs.service` raise to `--max-uploads 800` and `--max-deletes 100`. Note for Cloudflare R2, you need to [disable automatica metadata backups](https://juicefs.com/docs/community/how_to_setup_object_storage/#r2) with `--backup-meta 0`. Otherwise, for non-R2 S3 providers, you can set `--backup-meta 1h`.
```
[Unit]
Description=JuiceFS
AssertPathIsDirectory=/home/juicefs_mount
After=network-online.target
[Service]
Type=simple
WorkingDirectory=/home/juicefs
ExecStart=/usr/local/bin/juicefs mount \
"redis://:password@localhost:6479/1" \
/home/juicefs_mount \
--no-usage-report \
--writeback \
--cache-size 102400 \
--cache-dir /home/juicefs_cache \
--buffer-size 3072 \
--open-cache 0 \
--attr-cache 1 \
--entry-cache 1 \
--dir-entry-cache 1 \
--cache-partial-only false \
--free-space-ratio 0.1 \
--max-uploads 800 \
--max-deletes 100 \
--backup-meta 0 \
--log /var/log/juicefs.log \
--get-timeout 300 \
--put-timeout 900 \
--io-retries 90 \
--prefetch 2
ExecStop=/usr/local/bin/juicefs umount /home/juicefs_mount
Restart=always
RestartSec=5
[Install]
WantedBy=default.target
```
The table below shows comparison between [100x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-100x-r2-sharded-mount--redis-metadata-caching) vs [61x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-61x-r2-sharded-mount--redis-metadata-caching) vs [21x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-21x-r2-sharded-mount--redis-metadata-caching) vs [10x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-10x-r2-sharded-mount--redis-metadata-caching) vs [10x Cloudflare R2 sharded JuiceFS mount](#10x-r2-sharded-juicefs-mount) vs [1x Cloudflare JuiceFS mount (default)](#on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1). All R2 storage locations are with location hint North American East.
Default 1024MB big file.
| ITEM | VALUE (100x R2 Sharded + Redis) | COST (100x R2 Sharded + Redis) | VALUE (61x R2 Sharded + Redis) | COST (61x R2 Sharded + Redis) | VALUE (21x R2 Sharded + Redis) | COST (21x R2 Sharded + Redis) | VALUE (10x R2 Sharded + Redis) | COST (10x R2 Sharded + Redis) | VALUE (10x R2 Sharded) | COST (10x R2 Sharded) | VALUE (1x R2 Default) | COST (1x R2 Default) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Write big file | 1103.70 MiB/s | 7.42 s/file | 1778.16 MiB/s | 2.30 s/file | 1774.18 MiB/s | 2.31 s/file | 1904.61 MiB/s | 2.15 s/file | 906.04 MiB/s | 4.52 s/file | 1374.08 MiB/s | 2.98 s/file |
| Read big file | 342.66 MiB/s | 23.91 s/file | 231.92 MiB/s | 17.66 s/file | 162.36 MiB/s | 25.23 s/file | 201.00 MiB/s | 20.38 s/file | 223.19 MiB/s | 18.35 s/file | 152.23 MiB/s | 26.91 s/file |
| Write small file | 2505.3 files/s | 3.19 ms/file | 2449.2 files/s | 1.63 ms/file | 2333.5 files/s | 1.71 ms/file | 1319.8 files/s | 3.03 ms/file | 701.2 files/s | 5.70 ms/file | 780.3 files/s | 5.13 ms/file |
| Read small file | 13321.4 files/s | 0.60 ms/file | 5997.6 files/s | 0.67 ms/file | 10382.7 files/s | 0.39 ms/file | 10279.8 files/s | 0.39 ms/file | 6378.3 files/s | 0.63 ms/file | 8000.9 files/s | 0.50 ms/file |
| Stat file | 16229.5 files/s | 0.49 ms/file | 38302.2 files/s | 0.10 ms/file | 15955.7 files/s | 0.25 ms/file | 15890.1 files/s | 0.25 ms/file | 21123.7 files/s | 0.19 ms/file | 27902.2 files/s | 0.14 ms/file |
| FUSE operation | 142585 operations | 2.51 ms/op | 71292 operations | 1.80 ms/op | 71319 operations | 2.79 ms/op | 71338 operations | 2.23 ms/op | 71555 operations | 2.16 ms/op | 71649 operations | 3.06 ms/op |
| Update meta | 3473 operations | 1.98 ms/op | 1740 operations | 0.25 ms/op | 1739 operations | 0.25 ms/op | 1740 operations | 0.27 ms/op | 6271 operations | 9.01 ms/op | 6057 operations | 2.50 ms/op |
| Put object | 2847 operations | 470.88 ms/op | 1087 operations | 466.15 ms/op | 1055 operations | 514.85 ms/op | 1083 operations | 390.88 ms/op | 1152 operations | 403.23 ms/op | 1106 operations | 547.32 ms/op |
| Get object | 2048 operations | 402.89 ms/op | 1024 operations | 319.02 ms/op | 1027 operations | 346.44 ms/op | 1024 operations | 294.63 ms/op | 1034 operations | 278.61 ms/op | 1030 operations | 301.80 ms/op |
| Delete object | 440 operations | 174.48 ms/op | 215 operations | 201.12 ms/op | 736 operations | 195.40 ms/op | 754 operations | 125.28 ms/op | 316 operations | 124.32 ms/op | 29 operations | 234.02 ms/op |
| Write into cache | 2848 operations | 13.31 ms/op | 1424 operations | 5.36 ms/op | 1424 operations | 7.19 ms/op | 1424 operations | 4.85 ms/op | 1424 operations | 24 ms/op | 1424 operations | 7.19 ms/op |
| Read from cache | 800 operations | 0.09 ms/op | 400 operations | 0.07 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.04 ms/op |
Default 1MB big file.
| ITEM | VALUE (100x R2 Sharded + Redis) | COST (100x R2 Sharded + Redis) | VALUE (61x R2 Sharded + Redis) | COST (61x R2 Sharded + Redis) | VALUE (21x R2 Sharded + Redis) | COST (21x R2 Sharded + Redis) | VALUE (10x R2 Sharded + Redis) | COST (10x R2 Sharded + Redis) | VALUE (1x R2 Default) | COST (1x R2 Default) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Write big file | 637.16 MiB/s | 0.01 s/file | 617.15 MiB/s | 0.01 s/file | 600.01 MiB/s | 0.01 s/file | 530.10 MiB/s | 0.01 s/file | 230.82 MiB/s | 0.02 s/file |
| Read big file | 1764.73 MiB/s | 0.00 s/file | 1600.85 MiB/s | 0.00 s/file | 1300.69 MiB/s | 0.00 s/file | 1914.40 MiB/s | 0.00 s/file | 1276.38 MiB/s | 0.00 s/file |
| Write small file | 2666.9 files/s | 3.00 ms/file | 2808.3 files/s | 1.42 ms/file | 2648.3 files/s | 1.51 ms/file | 2715.4 files/s | 1.47 ms/file | 675.7 files/s | 5.92 ms/file |
| Read small file | 10905.8 files/s | 0.73 ms/file | 10154.0 files/s | 0.39 ms/file | 10442.4 files/s | 0.38 ms/file | 10069.0 files/s | 0.40 ms/file | 7833.1 files/s | 0.51 ms/file |
| Stat file | 22475.0 files/s | 0.36 ms/file | 15935.2 files/s | 0.25 ms/file | 16277.5 files/s | 0.25 ms/file | 16545.3 files/s | 0.24 ms/file | 28226.1 files/s | 0.14 ms/file |
| FUSE operation | 11485 operations | 0.12 ms/op | 5761 operations | 0.09 ms/op | 5765 operations | 0.09 ms/op | 5767 operations | 0.09 ms/op | 5756 operations | 0.41 ms/op |
| Update meta | 3233 operations | 0.19 ms/op | 1617 operations | 0.19 ms/op | 1617 operations | 0.18 ms/op | 1617 operations | 0.19 ms/op | 5770 operations | 0.70 ms/op |
| Put object | 735 operations | 344.93 ms/op | 32 operations | 377.01 ms/op | 30 operations | 369.65 ms/op | 37 operations | 290.94 ms/op | 118 operations | 242.35 ms/op |
| Get object | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op |
| Delete object | 117 operations | 123.92 ms/op | 76 operations | 189.67 ms/op | 22 operations | 268.03 ms/op | 48 operations | 103.83 ms/op | 95 operations | 83.94 ms/op |
| Write into cache | 808 operations | 0.13 ms/op | 404 operations | 0.11 ms/op | 404 operations | 0.11 ms/op | 404 operations | 0.11 ms/op | 404 operations | 0.14 ms/op |
| Read from cache | 816 operations | 0.13 ms/op | 408 operations | 0.07 ms/op | 408 operations | 0.08 ms/op | 408 operations | 0.06 ms/op | 408 operations | 0.06 ms/op |
100x R2 sharded JuiceFS mount with Redis metadata caching with location hint North American East
Default 1024MB big file.
```
juicefs bench -p 8 /home/juicefs_mount/
Write big blocks count: 8192 / 8192 [===========================================================] done
Read big blocks count: 8192 / 8192 [===========================================================] done
Write small blocks count: 800 / 800 [=============================================================] done
Read small blocks count: 800 / 800 [=============================================================] done
Stat small files count: 800 / 800 [=============================================================] done
Benchmark finished!
BlockSize: 1 MiB, BigFileSize: 1024 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 8
Time used: 33.5 s, CPU: 238.9%, Memory: 1687.5 MiB
+------------------+-------------------+--------------+
| ITEM | VALUE | COST |
+------------------+-------------------+--------------+
| Write big file | 1103.70 MiB/s | 7.42 s/file |
| Read big file | 342.66 MiB/s | 23.91 s/file |
| Write small file | 2505.3 files/s | 3.19 ms/file |
| Read small file | 13321.4 files/s | 0.60 ms/file |
| Stat file | 16229.5 files/s | 0.49 ms/file |
| FUSE operation | 142585 operations | 2.51 ms/op |
| Update meta | 3473 operations | 1.98 ms/op |
| Put object | 2847 operations | 470.88 ms/op |
| Get object | 2048 operations | 402.89 ms/op |
| Delete object | 440 operations | 174.48 ms/op |
| Write into cache | 2848 operations | 13.31 ms/op |
| Read from cache | 800 operations | 0.09 ms/op |
+------------------+-------------------+--------------+
```
Default 1MB big file.
```
juicefs bench -p 8 /home/juicefs_mount/ --big-file-size 1
Write big blocks count: 8 / 8 [==============================================================] done
Read big blocks count: 8 / 8 [==============================================================] done
Write small blocks count: 800 / 800 [=============================================================] done
Read small blocks count: 800 / 800 [=============================================================] done
Stat small files count: 800 / 800 [=============================================================] done
Benchmark finished!
BlockSize: 1 MiB, BigFileSize: 1 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 8
Time used: 0.8 s, CPU: 210.4%, Memory: 605.2 MiB
+------------------+------------------+--------------+
| ITEM | VALUE | COST |
+------------------+------------------+--------------+
| Write big file | 637.16 MiB/s | 0.01 s/file |
| Read big file | 1764.73 MiB/s | 0.00 s/file |
| Write small file | 2666.9 files/s | 3.00 ms/file |
| Read small file | 10905.8 files/s | 0.73 ms/file |
| Stat file | 22475.0 files/s | 0.36 ms/file |
| FUSE operation | 11485 operations | 0.12 ms/op |
| Update meta | 3233 operations | 0.19 ms/op |
| Put object | 735 operations | 344.93 ms/op |
| Get object | 0 operations | 0.00 ms/op |
| Delete object | 117 operations | 123.92 ms/op |
| Write into cache | 808 operations | 0.13 ms/op |
| Read from cache | 816 operations | 0.13 ms/op |
+------------------+------------------+--------------+
```
100x Cloudflare R2 buckets info using `sharded-buckets-info-r2.sh` passing AWS CLI profile `r2` with R2 bucket prefix `juicefs-shard-` for `100` sharded buckets and `https://cfaccountid.r2.cloudflarestorage.com` R2 endpoint.
```
./sharded-buckets-info-r2.sh r2 juicefs-shard- 100 https://cfaccountid.r2.cloudflarestorage.com
Bucket: juicefs-shard-0, Total Files: 34, Total Size: 4456448
Bucket: juicefs-shard-1, Total Files: 22, Total Size: 2627642
Bucket: juicefs-shard-2, Total Files: 28, Total Size: 3670016
Bucket: juicefs-shard-3, Total Files: 31, Total Size: 3935716
Bucket: juicefs-shard-4, Total Files: 28, Total Size: 3670016
Bucket: juicefs-shard-5, Total Files: 39, Total Size: 5111808
Bucket: juicefs-shard-6, Total Files: 35, Total Size: 4459354
Bucket: juicefs-shard-7, Total Files: 36, Total Size: 5506235
Bucket: juicefs-shard-8, Total Files: 140, Total Size: 20185088
Bucket: juicefs-shard-9, Total Files: 59, Total Size: 7733248
Bucket: juicefs-shard-10, Total Files: 50, Total Size: 6553600
Bucket: juicefs-shard-11, Total Files: 40, Total Size: 5242880
Bucket: juicefs-shard-12, Total Files: 63, Total Size: 10092544
Bucket: juicefs-shard-13, Total Files: 28, Total Size: 3670016
Bucket: juicefs-shard-14, Total Files: 31, Total Size: 4063232
Bucket: juicefs-shard-15, Total Files: 72, Total Size: 9437184
Bucket: juicefs-shard-16, Total Files: 37, Total Size: 4849664
Bucket: juicefs-shard-17, Total Files: 41, Total Size: 5373952
Bucket: juicefs-shard-18, Total Files: 123, Total Size: 18874368
Bucket: juicefs-shard-19, Total Files: 39, Total Size: 6029312
Bucket: juicefs-shard-20, Total Files: 30, Total Size: 3932160
Bucket: juicefs-shard-21, Total Files: 30, Total Size: 3932160
Bucket: juicefs-shard-22, Total Files: 42, Total Size: 5505024
Bucket: juicefs-shard-23, Total Files: 173, Total Size: 23592960
Bucket: juicefs-shard-24, Total Files: 42, Total Size: 5505024
Bucket: juicefs-shard-25, Total Files: 41, Total Size: 5373952
Bucket: juicefs-shard-26, Total Files: 33, Total Size: 4198517
Bucket: juicefs-shard-27, Total Files: 35, Total Size: 6422528
Bucket: juicefs-shard-28, Total Files: 62, Total Size: 8126464
Bucket: juicefs-shard-29, Total Files: 34, Total Size: 4327957
Bucket: juicefs-shard-30, Total Files: 28, Total Size: 3670016
Bucket: juicefs-shard-31, Total Files: 40, Total Size: 5242880
Bucket: juicefs-shard-32, Total Files: 32, Total Size: 4194304
Bucket: juicefs-shard-33, Total Files: 76, Total Size: 9961472
Bucket: juicefs-shard-34, Total Files: 31, Total Size: 4063232
Bucket: juicefs-shard-35, Total Files: 45, Total Size: 6685553
Bucket: juicefs-shard-36, Total Files: 41, Total Size: 5242916
Bucket: juicefs-shard-37, Total Files: 29, Total Size: 3801088
Bucket: juicefs-shard-38, Total Files: 123, Total Size: 16911519
Bucket: juicefs-shard-39, Total Files: 47, Total Size: 6160384
Bucket: juicefs-shard-40, Total Files: 27, Total Size: 3538944
Bucket: juicefs-shard-41, Total Files: 46, Total Size: 5899407
Bucket: juicefs-shard-42, Total Files: 34, Total Size: 4326157
Bucket: juicefs-shard-43, Total Files: 34, Total Size: 4456448
Bucket: juicefs-shard-44, Total Files: 42, Total Size: 6422528
Bucket: juicefs-shard-45, Total Files: 159, Total Size: 21757952
Bucket: juicefs-shard-46, Total Files: 45, Total Size: 5898240
Bucket: juicefs-shard-47, Total Files: 56, Total Size: 8257536
Bucket: juicefs-shard-48, Total Files: 40, Total Size: 5242880
Bucket: juicefs-shard-49, Total Files: 51, Total Size: 6684672
Bucket: juicefs-shard-50, Total Files: 29, Total Size: 3801088
Bucket: juicefs-shard-51, Total Files: 179, Total Size: 23461888
Bucket: juicefs-shard-52, Total Files: 40, Total Size: 5112935
Bucket: juicefs-shard-53, Total Files: 138, Total Size: 19795756
Bucket: juicefs-shard-54, Total Files: 62, Total Size: 8914826
Bucket: juicefs-shard-55, Total Files: 41, Total Size: 6291456
Bucket: juicefs-shard-56, Total Files: 72, Total Size: 9437184
Bucket: juicefs-shard-57, Total Files: 45, Total Size: 6815744
Bucket: juicefs-shard-58, Total Files: 32, Total Size: 4064159
Bucket: juicefs-shard-59, Total Files: 38, Total Size: 6685639
Bucket: juicefs-shard-60, Total Files: 34, Total Size: 4456448
Bucket: juicefs-shard-61, Total Files: 41, Total Size: 5373952
Bucket: juicefs-shard-62, Total Files: 34, Total Size: 4329917
Bucket: juicefs-shard-63, Total Files: 61, Total Size: 8912896
Bucket: juicefs-shard-64, Total Files: 50, Total Size: 7471104
Bucket: juicefs-shard-65, Total Files: 37, Total Size: 4849664
Bucket: juicefs-shard-66, Total Files: 38, Total Size: 4980736
Bucket: juicefs-shard-67, Total Files: 119, Total Size: 16515072
Bucket: juicefs-shard-68, Total Files: 115, Total Size: 15990784
Bucket: juicefs-shard-69, Total Files: 39, Total Size: 5111808
Bucket: juicefs-shard-70, Total Files: 63, Total Size: 9966336
Bucket: juicefs-shard-71, Total Files: 44, Total Size: 5767168
Bucket: juicefs-shard-72, Total Files: 31, Total Size: 4063232
Bucket: juicefs-shard-73, Total Files: 42, Total Size: 5244609
Bucket: juicefs-shard-74, Total Files: 25, Total Size: 3276800
Bucket: juicefs-shard-75, Total Files: 47, Total Size: 6030141
Bucket: juicefs-shard-76, Total Files: 69, Total Size: 9043968
Bucket: juicefs-shard-77, Total Files: 54, Total Size: 7995392
Bucket: juicefs-shard-78, Total Files: 128, Total Size: 16518412
Bucket: juicefs-shard-79, Total Files: 104, Total Size: 13631488
Bucket: juicefs-shard-80, Total Files: 77, Total Size: 10092544
Bucket: juicefs-shard-81, Total Files: 49, Total Size: 6422528
Bucket: juicefs-shard-82, Total Files: 54, Total Size: 7077888
Bucket: juicefs-shard-83, Total Files: 84, Total Size: 11010048
Bucket: juicefs-shard-84, Total Files: 96, Total Size: 12582912
Bucket: juicefs-shard-85, Total Files: 50, Total Size: 7471104
Bucket: juicefs-shard-86, Total Files: 32, Total Size: 4194304
Bucket: juicefs-shard-87, Total Files: 41, Total Size: 6291456
Bucket: juicefs-shard-88, Total Files: 81, Total Size: 10616832
Bucket: juicefs-shard-89, Total Files: 114, Total Size: 14942208
Bucket: juicefs-shard-90, Total Files: 37, Total Size: 5767168
Bucket: juicefs-shard-91, Total Files: 54, Total Size: 7077888
Bucket: juicefs-shard-92, Total Files: 62, Total Size: 8126464
Bucket: juicefs-shard-93, Total Files: 56, Total Size: 7340032
Bucket: juicefs-shard-94, Total Files: 35, Total Size: 4587520
Bucket: juicefs-shard-95, Total Files: 46, Total Size: 6946816
Bucket: juicefs-shard-96, Total Files: 41, Total Size: 6291456
Bucket: juicefs-shard-97, Total Files: 151, Total Size: 19791872
Bucket: juicefs-shard-98, Total Files: 92, Total Size: 12976128
Bucket: juicefs-shard-99, Total Files: 177, Total Size: 24117248
Total for all buckets, Total Files: 5804, Total Size: 792512215
```
### JuiceFS Benchmarks 61x R2 Sharded Mount + Redis Metadata Caching
Format JuiceFS sharded mount to use Redis metadata caching via `redis://:password@localhost:6479/1`
```
cfaccountid='CF_ACCOUNT_ID'
cfaccesskey=''
cfsecretkey=''
cfbucketname='juicefs-shard'
mkdir -p /home/juicefs
cd /home/juicefs
juicefs format --storage s3 \
--shards 61 \
--bucket https://${cfbucketname}-%d.${cfaccountid}.r2.cloudflarestorage.com \
--access-key $cfaccesskey \
--secret-key $cfsecretkey \
--compress none \
--trash-days 0 \
--block-size 4096 \
redis://:password@localhost:6479/1 myjuicefs
```
Edit `/usr/lib/systemd/system/juicefs.service` raise `--max-uploads 20` and `--max-deletes 10` values to `--max-uploads 30` and `--max-deletes 30`. Note for Cloudflare R2, you need to [disable automatica metadata backups](https://juicefs.com/docs/community/how_to_setup_object_storage/#r2) with `--backup-meta 0`. Otherwise, for non-R2 S3 providers, you can set `--backup-meta 1h`.
```
[Unit]
Description=JuiceFS
AssertPathIsDirectory=/home/juicefs_mount
After=network-online.target
[Service]
Type=simple
WorkingDirectory=/home/juicefs
ExecStart=/usr/local/bin/juicefs mount \
"redis://:password@localhost:6479/1" \
/home/juicefs_mount \
--no-usage-report \
--writeback \
--cache-size 102400 \
--cache-dir /home/juicefs_cache \
--buffer-size 2048 \
--open-cache 0 \
--attr-cache 1 \
--entry-cache 1 \
--dir-entry-cache 1 \
--cache-partial-only false \
--free-space-ratio 0.1 \
--max-uploads 30 \
--max-deletes 30 \
--backup-meta 0 \
--log /var/log/juicefs.log \
--get-timeout 300 \
--put-timeout 900 \
--io-retries 90 \
--prefetch 1
ExecStop=/usr/local/bin/juicefs umount /home/juicefs_mount
Restart=always
RestartSec=5
[Install]
WantedBy=default.target
```
The table below shows comparison between [61x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-61x-r2-sharded-mount--redis-metadata-caching) vs [21x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-21x-r2-sharded-mount--redis-metadata-caching) vs [10x Cloudflare R2 sharded JuiceFS mount + Redis metadata caching](#juicefs-benchmarks-10x-r2-sharded-mount--redis-metadata-caching) vs [10x Cloudflare R2 sharded JuiceFS mount](#10x-r2-sharded-juicefs-mount) vs [1x Cloudflare JuiceFS mount (default)](#on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1). All R2 storage locations are with location hint North American East.
Default 1024MB big file.
| ITEM | VALUE (61x R2 Sharded + Redis) | COST (61x R2 Sharded + Redis) | VALUE (21x R2 Sharded + Redis) | COST (21x R2 Sharded + Redis) | VALUE (10x R2 Sharded + Redis) | COST (10x R2 Sharded + Redis) | VALUE (10x R2 Sharded) | COST (10x R2 Sharded) | VALUE (1x R2 Default) | COST (1x R2 Default) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Write big file | 1778.16 MiB/s | 2.30 s/file | 1774.18 MiB/s | 2.31 s/file | 1904.61 MiB/s | 2.15 s/file | 906.04 MiB/s | 4.52 s/file | 1374.08 MiB/s | 2.98 s/file |
| Read big file | 231.92 MiB/s | 17.66 s/file | 162.36 MiB/s | 25.23 s/file | 201.00 MiB/s | 20.38 s/file | 223.19 MiB/s | 18.35 s/file | 152.23 MiB/s | 26.91 s/file |
| Write small file | 2449.2 files/s | 1.63 ms/file | 2333.5 files/s | 1.71 ms/file | 1319.8 files/s | 3.03 ms/file | 701.2 files/s | 5.70 ms/file | 780.3 files/s | 5.13 ms/file |
| Read small file | 5997.6 files/s | 0.67 ms/file | 10382.7 files/s | 0.39 ms/file | 10279.8 files/s | 0.39 ms/file | 6378.3 files/s | 0.63 ms/file | 8000.9 files/s | 0.50 ms/file |
| Stat file | 38302.2 files/s | 0.10 ms/file | 15955.7 files/s | 0.25 ms/file | 15890.1 files/s | 0.25 ms/file | 21123.7 files/s | 0.19 ms/file | 27902.2 files/s | 0.14 ms/file |
| FUSE operation | 71292 operations | 1.80 ms/op | 71319 operations | 2.79 ms/op | 71338 operations | 2.23 ms/op | 71555 operations | 2.16 ms/op | 71649 operations | 3.06 ms/op |
| Update meta | 1740 operations | 0.25 ms/op | 1739 operations | 0.25 ms/op | 1740 operations | 0.27 ms/op | 6271 operations | 9.01 ms/op | 6057 operations | 2.50 ms/op |
| Put object | 1087 operations | 466.15 ms/op | 1055 operations | 514.85 ms/op | 1083 operations | 390.88 ms/op | 1152 operations | 403.23 ms/op | 1106 operations | 547.32 ms/op |
| Get object | 1024 operations | 319.02 ms/op | 1027 operations | 346.44 ms/op | 1024 operations | 294.63 ms/op | 1034 operations | 278.61 ms/op | 1030 operations | 301.80 ms/op |
| Delete object | 215 operations | 201.12 ms/op | 736 operations | 195.40 ms/op | 754 operations | 125.28 ms/op | 316 operations | 124.32 ms/op | 29 operations | 234.02 ms/op |
| Write into cache | 1424 operations | 5.36 ms/op | 1424 operations | 7.19 ms/op | 1424 operations | 4.85 ms/op | 1424 operations | 24 ms/op | 1424 operations | 7.19 ms/op |
| Read from cache | 400 operations | 0.07 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.05 ms/op | 400 operations | 0.04 ms/op |
Default 1MB big file.
| ITEM | VALUE (61x R2 Sharded + Redis) | COST (61x R2 Sharded + Redis) | VALUE (21x R2 Sharded + Redis) | COST (21x R2 Sharded + Redis) | VALUE (10x R2 Sharded + Redis) | COST (10x R2 Sharded + Redis) | VALUE (1x R2 Default) | COST (1x R2 Default) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Write big file | 617.15 MiB/s | 0.01 s/file | 600.01 MiB/s | 0.01 s/file | 530.10 MiB/s | 0.01 s/file | 230.82 MiB/s | 0.02 s/file |
| Read big file | 1600.85 MiB/s | 0.00 s/file | 1300.69 MiB/s | 0.00 s/file | 1914.40 MiB/s | 0.00 s/file | 1276.38 MiB/s | 0.00 s/file |
| Write small file | 2808.3 files/s | 1.42 ms/file | 2648.3 files/s | 1.51 ms/file | 2715.4 files/s | 1.47 ms/file | 675.7 files/s | 5.92 ms/file |
| Read small file | 10154.0 files/s | 0.39 ms/file | 10442.4 files/s | 0.38 ms/file | 10069.0 files/s | 0.40 ms/file | 7833.1 files/s | 0.51 ms/file |
| Stat file | 15935.2 files/s | 0.25 ms/file | 16277.5 files/s | 0.25 ms/file | 16545.3 files/s | 0.24 ms/file | 28226.1 files/s | 0.14 ms/file |
| FUSE operation | 5761 operations | 0.09 ms/op | 5765 operations | 0.09 ms/op | 5767 operations | 0.09 ms/op | 5756 operations | 0.41 ms/op |
| Update meta | 1617 operations | 0.19 ms/op | 1617 operations | 0.18 ms/op | 1617 operations | 0.19 ms/op | 5770 operations | 0.70 ms/op |
| Put object | 32 operations | 377.01 ms/op | 30 operations | 369.65 ms/op | 37 operations | 290.94 ms/op | 118 operations | 242.35 ms/op |
| Get object | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op | 0 operations | 0.00 ms/op |
| Delete object | 76 operations | 189.67 ms/op | 22 operations | 268.03 ms/op | 48 operations | 103.83 ms/op | 95 operations | 83.94 ms/op |
| Write into cache | 404 operations | 0.11 ms/op | 404 operations | 0.11 ms/op | 404 operations | 0.11 ms/op | 404 operations | 0.14 ms/op |
| Read from cache | 408 operations | 0.07 ms/op | 408 operations | 0.08 ms/op | 408 operations | 0.06 ms/op | 408 operations | 0.06 ms/op |
61x R2 sharded JuiceFS mount with Redis metadata caching with location hint North American East
Default 1024MB big file.
```
juicefs bench -p 4 /home/juicefs_mount/
Write big blocks count: 4096 / 4096 [===========================================================] done
Read big blocks count: 4096 / 4096 [===========================================================] done
Write small blocks count: 400 / 400 [========================