{"id":13483274,"url":"https://github.com/s3ql/s3ql","last_synced_at":"2025-05-15T19:09:35.285Z","repository":{"id":1862876,"uuid":"38976977","full_name":"s3ql/s3ql","owner":"s3ql","description":"a full featured file system for online data storage","archived":false,"fork":false,"pushed_at":"2025-05-10T09:26:05.000Z","size":8008,"stargazers_count":1166,"open_issues_count":14,"forks_count":112,"subscribers_count":35,"default_branch":"master","last_synced_at":"2025-05-10T10:29:44.113Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/s3ql.png","metadata":{"files":{"readme":"README.rst","changelog":"ChangeLog.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-07-12T20:33:18.000Z","updated_at":"2025-05-10T09:38:43.000Z","dependencies_parsed_at":"2024-01-16T09:03:46.429Z","dependency_job_id":"84709eb8-b758-419b-b757-916ea5c74c52","html_url":"https://github.com/s3ql/s3ql","commit_stats":{"total_commits":3140,"total_committers":49,"mean_commits":64.08163265306122,"dds":0.25,"last_synced_commit":"4a3bc37b84b664eba2a101e7c2405268e2e942ca"},"previous_names":[],"tags_count":132,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3ql%2Fs3ql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3ql%2Fs3ql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3ql%2Fs3ql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3ql%2Fs3ql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/s3ql","download_url":"https://codeload.github.com/s3ql/s3ql/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254404357,"owners_count":22065641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T17:01:09.590Z","updated_at":"2025-05-15T19:09:35.264Z","avatar_url":"https://github.com/s3ql.png","language":"Python","readme":"..\n  NOTE: We cannot use sophisticated ReST syntax here because this\n  file is rendered by GitHub.\n\n======\n S3QL\n======\n\nS3QL is a file system that stores all its data online using storage\nservices like `Google Storage`_, `Amazon S3`_, or OpenStack_. S3QL\neffectively provides a virtual drive of dynamic, infinite capacity that\ncan be accessed from any computer with internet access.\n\nS3QL is a full featured UNIX file system that is conceptually indistinguishable from a\nlocal file system like ext4.  Furthermore, S3QL has additional features like compression\nencryption, data de-duplication, immutable trees and snapshotting which make it especially\nsuitable for online backup and archival.\n\nS3QL is designed to favor simplicity and elegance over performance and\nfeature-creep. Care has been taken to make the source code as\nreadable and serviceable as possible. Solid error detection and error\nhandling have been included from the very first line, and S3QL comes\nwith extensive automated test cases for all its components.\n\n.. _`Google Storage`: https://cloud.google.com/storage/docs\n.. _`Amazon S3`: https://aws.amazon.com/s3/\n.. _OpenStack: https://www.openstack.org/software/\n\n\nFeatures\n========\n\n* **Transparency.** Conceptually, S3QL is indistinguishable from a\n  local file system. For example, it supports hardlinks, symlinks,\n  standard unix permissions, extended attributes and file\n  sizes up to 2 TB.\n\n* **Dynamic Size.** The size of an S3QL file system grows and shrinks\n  dynamically as required.\n\n* **Compression.** Before storage, all data may be compressed with the\n  LZMA, bzip2 or deflate (gzip) algorithm.\n\n* **Encryption.** After compression (but before upload), all data can be\n  AES encrypted with a 256 bit key. An additional SHA256 HMAC checksum\n  is used to protect the data against manipulation.\n\n* **Data De-duplication.** If several files have identical contents,\n  the redundant data will be stored only once. This works across all\n  files stored in the file system, and also if only some parts of the\n  files are identical while other parts differ.\n\n* **Immutable Trees.** Directory trees can be made immutable, so that\n  their contents can no longer be changed in any way whatsoever. This\n  can be used to ensure that backups can not be modified after they\n  have been made.\n\n* **Copy-on-write snapshots.** S3QL can replicate entire directory\n  trees without using any additional storage space. Only if one of the\n  copies is modified, the part of the data that has been modified will\n  take up additional storage space. This can be used to create\n  intelligent snapshots that preserve the state of a directory at\n  different points in time using a minimum amount of space.\n\n* **Performance independent of network latency.** All operations\n  that do not write or read file contents (like creating directories\n  or moving, renaming, and changing permissions of files and\n  directories) are very fast because they are carried out without any\n  network transactions.\n\n  S3QL achieves this by saving the entire file and directory structure\n  in a database. This database is locally cached and the remote\n  copy updated asynchronously.\n\n* **Support for low bandwidth connections.** S3QL splits file contents\n  into smaller blocks and caches blocks locally. This minimizes both\n  the number of network transactions required for reading and writing\n  data, and the amount of data that has to be transferred when only\n  parts of a file are read or written.\n\n\nDevelopment Status\n==================\n\nS3QL is considered stable and suitable for production use.  Starting\nwith version 2.17.1, S3QL uses semantic versioning. This means that\nbackwards-incompatible versions (e.g., versions that require an\nupgrade of the file system revision) will be reflected in an increase\nof the major version number.\n\n\nSupported Platforms\n===================\n\nS3QL is developed and tested under Linux. Users have also reported\nrunning S3QL successfully on OS-X, FreeBSD and NetBSD. We try to\nmaintain compatibility with these systems, but (due to lack of\npre-release testers) we cannot guarantee that every release will run\non all non-Linux systems. Please report any bugs you find, and we will\ntry to fix them.\n\n\nTypical Usage\n=============\n\nBefore a file system can be mounted, the backend which will hold the\ndata has to be initialized. This is done with the *mkfs.s3ql*\ncommand. Here we are using the Amazon S3 backend, and\n*nikratio-s3ql-bucket* is the S3 bucket in which the file system will\nbe stored. ::\n\n  mkfs.s3ql s3://ap-south-1/nikratio-s3ql-bucket\n\nTo mount the S3QL file system stored in the S3 bucket\n*nikratio_s3ql_bucket* in the directory ``/mnt/s3ql``, enter::\n\n  mount.s3ql s3://ap-south-1/nikratio-s3ql-bucket /mnt/s3ql\n\nNow you can instruct your favorite backup program to run a backup into\nthe directory ``/mnt/s3ql`` and the data will be stored on Amazon\nS3. When you are done, the file system has to be unmounted with ::\n\n   umount.s3ql /mnt/s3ql\n\n\nNeed Help?\n==========\n\nThe following resources are available:\n\n* The `S3QL User's Guide`_.\n* The `S3QL Wiki`_\n* The `S3QL Mailing List`_. You\n  can subscribe by sending a mail to\n  `s3ql+subscribe@googlegroups.com \u003cmailto:s3ql+subscribe@googlegroups.com\u003e`_.\n\nPlease report any bugs you may encounter in the `GitHub Issue Tracker`_.\n\nContributing\n============\n\nThe S3QL source code is available on GitHub_.\n\n.. _`S3QL User's Guide`: https://www.rath.org/s3ql-docs/\n.. _`S3QL Wiki`: https://github.com/s3ql/s3ql/wiki\n.. _`S3QL Mailing List`: https://groups.google.com/g/s3ql\n.. _`GitHub Issue Tracker`: https://github.com/s3ql/s3ql/issues\n.. _GitHub: https://github.com/s3ql/s3ql\n","funding_links":[],"categories":["\u003ca id=\"tag-dev\" href=\"#tag-dev\"\u003eDev\u003c/a\u003e","Python"],"sub_categories":["\u003ca id=\"tag-dev.storage\" href=\"#tag-dev.storage\"\u003eStorage\u003c/a\u003e"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs3ql%2Fs3ql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fs3ql%2Fs3ql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs3ql%2Fs3ql/lists"}