{"id":13579367,"url":"https://github.com/45Drives/autotier","last_synced_at":"2025-04-05T20:34:00.162Z","repository":{"id":47327006,"uuid":"221750048","full_name":"45Drives/autotier","owner":"45Drives","description":"A passthrough FUSE filesystem that intelligently moves files between storage tiers based on frequency of use, file age, and tier fullness.","archived":false,"fork":false,"pushed_at":"2024-02-11T23:56:18.000Z","size":6457,"stargazers_count":264,"open_issues_count":11,"forks_count":13,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-11-05T17:49:39.372Z","etag":null,"topics":["data-storage","passthrough-fuse-filesystem","storage-tiers","tiered","tiered-file-system","tiered-storage","tiering"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/45Drives.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-14T17:18:21.000Z","updated_at":"2024-10-27T03:44:12.000Z","dependencies_parsed_at":"2024-01-16T20:28:04.960Z","dependency_job_id":"5eee053c-d212-4a81-b448-3469e40e76bf","html_url":"https://github.com/45Drives/autotier","commit_stats":null,"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/45Drives%2Fautotier","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/45Drives%2Fautotier/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/45Drives%2Fautotier/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/45Drives%2Fautotier/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/45Drives","download_url":"https://codeload.github.com/45Drives/autotier/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247399877,"owners_count":20932876,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-storage","passthrough-fuse-filesystem","storage-tiers","tiered","tiered-file-system","tiered-storage","tiering"],"created_at":"2024-08-01T15:01:38.733Z","updated_at":"2025-04-05T20:33:55.151Z","avatar_url":"https://github.com/45Drives.png","language":"C++","funding_links":[],"categories":["\u003ca name=\"cpp\"\u003e\u003c/a\u003eC++"],"sub_categories":[],"readme":"# autotier\nA passthrough FUSE filesystem that intelligently moves files between storage tiers based on frequency of use, file age, and tier fullness.\n\n## What it does\n`autotier` is a tiered FUSE filesystem which acts as a merging passthrough to any number of underlying filesystems. These underlying filesystems can be of any type. Behind the scenes, `autotier` moves files around such that the most often accessed files are kept in the highest tier. `autotier` fills each defined tier up to their configuration-defined quota, starting at the fastest tier with the highest priority files. If you do a lot of writing, set a lower quota for the highest tier to allow for more room. If you do mostly reading, set a higher watermark to allow for as much use as possible out of your available top tier storage.  \n![autotier example](doc/mounted_fs_status.png)\n\n## Quick Start\n1. [Install](#installation)\n1. [Configure](#configuration)\n1. If data is already on any tiers, run `autotier-init-dirs /path/to/tier1 /path/to/tier2 [ /path/to/tier3 ... ]` to clone directory structures across all tiers. ([Note on conflicting file paths between tiers](#file-path-conflicts))\n1. [Mount Filesystem](#mounting)\n\n## Installation\n### Ubuntu Focal\n1. Get deb: `$ wget https://github.com/45Drives/autotier/releases/download/v1.2.0/autotier_1.2.0-1focal_amd64.deb`\n1. Install deb: `# apt install ./autotier_1.2.0-1focal_amd64.deb`\n1. [Edit configuration file.](#configuration)\n1. [Mount filesystem.](#mounting)\n1. Optionally add user to `autotier` group to allow non-root users to run CLI commands:\n\t* `# usermod -aG autotier \u003cuser\u003e` (takes effect on next login)\n\n### Debian Bullseye\n1. Get deb: `$ wget https://github.com/45Drives/autotier/releases/download/v1.2.0/autotier_1.2.0-1bullseye_amd64.deb`\n1. Install deb: `# apt install ./autotier_1.2.0-1bullseye_amd64.deb`\n1. [Edit configuration file.](#configuration)\n1. [Mount filesystem.](#mounting)\n1. Optionally add user to `autotier` group to allow non-root users to run CLI commands:\n\t* `# usermod -aG autotier \u003cuser\u003e` (takes effect on next login)\n\n### EL8\n1. Install rpm: `# dnf install https://github.com/45Drives/autotier/releases/download/v1.2.0/autotier-1.2.0-1.el8.x86_64.rpm`\n1. [Edit configuration file.](#configuration)\n1. [Mount filesystem.](#mounting)\n1. Optionally add user to `autotier` group to allow non-root users to run CLI commands:\n\t* `# usermod -aG autotier \u003cuser\u003e` (takes effect on next login)\n\n### Installing from Source\n1. Install dependencies:  \n\t```# apt install libfuse3-dev libboost-system-dev libboost-filesystem-dev libboost-serialization-dev librocksdb-dev libtbb-dev```\n1. `$ git clone https://github.com/45drives/autotier`\n1. `$ cd autotier`\n1. `$ git checkout \u003cversion\u003e` (v1.2.0 is the latest tag)\n1. `$ make -j8` (or `make -j8 no-par-sort` to use c++11 instead of c++17)\n1. `# make install`\n1. [Edit configuration file.](#configuration)\n1. [Mount filesystem.](#mounting)\n1. Optionally add user to `autotier` group to allow non-root users to run CLI commands:\n\t* `# usermod -aG autotier \u003cuser\u003e` (takes effect on next login)\n\n### Uninstallation\nUbuntu: `# apt remove autotier` or `# apt purge autotier`  \nEL8: `# dnf remove autotier`\nFrom source: `# make uninstall` from root of cloned repo\n\n## Configuration\nSee `man autotier` after installing for full configuration details.  \nDefault configuration file location: `/etc/autotier.conf`.\n### Global Config\nFor global configuration of `autotier`, options are placed below the `[Global]` header. Example:\n```\n[Global]                       # global settings\nLog Level = 1                  # 0 = none, 1 = normal, 2 = debug\nTier Period = 100              # number of seconds between file move batches\nCopy Buffer Size = 1 MiB       # size of buffer for moving files between tiers\n```\nThe global config section can be placed before, after, or between tier definitions.\n### Tier Config\nThe layout of a single tier's configuration entry is as follows:\n```\n[Tier 1]                       # tier name (can be anything)\nPath =                         # full path to tier storage pool\nQuota =                        # absolute or % usage to keep tier under\n# Quota format: x (%|B|MB|MiB|KB|KiB|MB|MiB|...)\n# Example: Quota = 5.3 TiB\n```\nAs many tiers as desired can be defined in the configuration, however they must be in order of fastest to slowest. The tier's name can be whatever you want but it cannot be `global` or `Global`. Tier names are only used for config diagnostics and file pinning.  \nBelow is a complete example of a configuration file:\n```\n# autotier config\n[Global]                       # global settings\nLog Level = 1                  # 0 = none, 1 = normal, 2 = debug\nTier Period = 1000             # number of seconds between file move batches\nCopy Buffer Size = 1 MiB       # size of buffer for moving files between tiers\n\n[Tier 1]                       # tier name (can be anything)\nPath = /mnt/ssd_tier           # full path to tier storage pool\nQuota = 5 TiB                  # absolute or % usage to keep tier under\n\n[Tier 2]\nPath = /mnt/ssd_tier\nQuota = 90 %\n\n[Tier 3]\nPath = /mnt/cold_storage\nQuota = 100 %\n```\n\n## Usage\nSee `man autotier` after installing for full usage details.\n### Command Line Tool Usage\n```\nUsage:\n  autotier [\u003cflags\u003e] \u003ccommand\u003e [\u003carg1 arg2 ...\u003e]\nCommands:\n  config      - display current configuration values\n  help        - display this message\n  list-pins   - show all pinned files\n  list-popularity\n              - print list of all tier files sorted by frequency of use\n  oneshot     - execute tiering only once\n  pin \u003c\"tier name\"\u003e \u003c\"path/to/file\" \"path/to/file\" ...\u003e\n              - pin file(s) to tier using tier name in config file\n  status      - list info about defined tiers\n  unpin \u003c\"path/to/file\" \"path/to/file\" ...\u003e\n              - remove pin from file(s)\n  which-tier \u003c\"path/to/file\" \"path/to/file\" ...\u003e\n              - list which tier each argument is in\nFlags:\n  -c, --config \u003cpath/to/config\u003e\n              - override configuration file path (default /etc/autotier.conf)\n  -h, --help  - display this message and cancel current command\n  -q, --quiet - set log level to 0 (no output)\n  -v, --verbose\n              - set log level to 2 (debug output)\n  -V, --version\n              - print version and exit\n              - if log level \u003e= 1, logo will also print\n              - combine with -q to mute logo output\n```\nExamples:  \nTrigger tiering of files immediately:  \n`autotier oneshot`  \nShow status of configured tiers:  \n`autotier status`  \nPin a file to a tier with \\\u003cTier Name\\\u003e:  \n`autotier pin \"\u003cTier Name\u003e\" /path/to/file`  \nPin multiple files:  \n`autotier pin \"\u003cTier Name\u003e\" /path/to/file1 /path/to/dir/* /bash/expansion/**/*`  \n`find path/* -type f -print | xargs autotier pin \"\u003cTier Name\u003e\"`  \nRemove pins:  \n`autotier unpin path/to/file`  \n`find path/* -type f -print | xargs autotier unpin`  \nList pinned files:  \n`autotier list-pins`\n\n### Filesystem Usage\n```\nUsage:\n  autotierfs [\u003cflags\u003e] \u003cmountpoint\u003e [-o \u003cfuse,options,...\u003e]\nFlags:\n  -c, --config \u003cpath/to/config\u003e\n              - override configuration file path (default /etc/autotier.conf)\n  -h, --help  - display this message and cancel current command\n  -o, --fuse-options \u003ccomma,separated,list\u003e\n              - mount options to pass to fuse (see man mount.fuse)\n  -q, --quiet - set log level to 0 (no output)\n  -v, --verbose\n              - set log level to 2 (debug output)\n  -V, --version\n              - print version and exit\n              - if log level \u003e= 1, logo will also print\n              - combine with -q to mute logo output\n```\n#### Mounting\n##### fstab\n1. Ensure a mount point exists, e.g. `sudo mkdir /mnt/autotier`\n2. Edit `/etc/fstab` as root and add the following line:\n   ```\n   /usr/bin/autotierfs\t/mnt/autotier\tfuse\tallow_other,default_permissions 0 0\n   ```\n3. Apply with `sudo mount -a`\n##### Manually\n1. Ensure a mount point exists, e.g. `sudo mkdir /mnt/autotier`\n2. Mount filesystem with `sudo autotierfs /mnt/autotier -o allow_other,default_permissions`\n\n### Using cron  \nTo have `cron` schedule file tiering, first disable automatic tiering by setting `Tier Period = -1` in `/etc/autotier.conf`.\nThen in the cron entry, call `autotier oneshot`.  \n\n## Notes\n### File Path Conflicts\nIf autotier is set up with tiers that already contain files, there is a chance that you could have more than one file with the same path relative to the tier root. In the event of a collision during file movement while tiering, the file that is already in the tier is left untouched and the incoming file has `.autotier_conflict.\u003coriginal tier name\u003e` appended to the file name. `autotier status` will list any file conflicts found.\n### Development Documentation\nDevelopment documentation (generated by Doxygen) can be found [here](https://45drives.github.io/autotier/dev-doc/index.html)  \n\n\n---\n```\n   ┓\n└─ ┃ ├─\n└─ ┣ ├─\n└─ ┃ └─\n   ┛\n```\n[![45Drives Logo](https://www.45drives.com/img/45-drives-brand.png)](https://www.45drives.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F45Drives%2Fautotier","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F45Drives%2Fautotier","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F45Drives%2Fautotier/lists"}