An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with s3

A curated list of projects in awesome lists tagged with s3 .

https://github.com/minio/minio

MinIO is a high-performance, S3 compatible object store, open sourced under GNU AGPLv3 license.

amazon-s3 cloud cloudnative cloudstorage go k8s kubernetes multi-cloud multi-cloud-kubernetes objectstorage s3 storage

Last synced: 12 May 2025

https://github.com/rclone/rclone

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files

azure-blob azure-blob-storage azure-files backblaze-b2 cloud-storage dropbox encryption ftp fuse-filesystem go golang google-cloud-storage google-drive onedrive openstack-swift rclone s3 sftp sync webdav

Last synced: 12 May 2025

https://github.com/ncw/rclone

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files

azure-blob azure-blob-storage azure-files backblaze-b2 cloud-storage dropbox encryption ftp fuse-filesystem go golang google-cloud-storage google-drive onedrive openstack-swift rclone s3 sftp sync webdav

Last synced: 15 Apr 2025

https://github.com/siyuan-note/siyuan

A privacy-first, self-hosted, fully open source personal knowledge management software, written in typescript and golang.

anki chatgpt deepseek electron evernote knowledge-base local-first markdown note-taking notes-app notion obsidian ocr ollama openai pdf s3 self-hosted webdav

Last synced: 12 May 2025

https://github.com/seaweedfs/seaweedfs

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

blob-storage cloud-drive distributed-file-system distributed-storage distributed-systems erasure-coding fuse hadoop-hdfs hdfs kubernetes object-storage posix replication s3 s3-storage seaweedfs tiered-file-system

Last synced: 12 May 2025

https://github.com/airbytehq/airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

bigquery change-data-capture data data-analysis data-collection data-engineering data-integration data-pipeline elt etl java mssql mysql pipeline postgresql python redshift s3 self-hosted snowflake

Last synced: 12 May 2025

https://github.com/thanos-io/thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

cncf go google-cloud-storage hacktoberfest high-availability metrics monitoring observability prometheus prometheus-ha-pairs prometheus-setup s3 storage thanos

Last synced: 12 May 2025

https://github.com/benbjohnson/litestream

Streaming replication for SQLite.

replication s3 sqlite

Last synced: 12 May 2025

https://github.com/juicedata/juicefs

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

bigdata cloud-native distributed-systems filesystem go golang hdfs object-storage posix redis s3 storage

Last synced: 12 May 2025

https://github.com/mickael-kerjean/filestash

:file_folder: A file manager / web client for SFTP, S3, FTP, WebDAV, Git, Minio, LDAP, CalDAV, CardDAV, Mysql, Backblaze, ...

file-manager file-sharing filemanager ftp ftps git ldap nfs s3 sftp smb webclient webdav

Last synced: 12 May 2025

https://github.com/drakkan/sftpgo

Full-featured and highly configurable SFTP, HTTP/S, FTP/S and WebDAV server - S3, Google Cloud Storage, Azure Blob

azure-blob cloud-storage data-at-rest-encryption docker ftp ftp-server git go google-cloud-storage multi-factor-authentication portable prometheus proxy-protocol s3 scp sftp sftp-server webdav webdav-server

Last synced: 12 May 2025

https://github.com/s3fs-fuse/s3fs-fuse

FUSE-based file system backed by Amazon S3

aws-s3 cloud-storage filesystem fuse fuse-filesystem s3

Last synced: 14 Mar 2025

https://github.com/getmoto/moto

A library that allows you to easily mock out tests based on AWS infrastructure.

aws boto ec2 s3

Last synced: 12 May 2025

https://github.com/automq/automq

AutoMQ is a stateless Kafka on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.

ai apache-kafka aws azure cloud cloud-first cloud-native ebs gcp kafka llm messaging minio s3 serverless spot streaming

Last synced: 12 May 2025

https://github.com/awslabs/mountpoint-s3

A simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system.

aws filesystem fuse object-storage s3

Last synced: 14 May 2025

https://github.com/backup/backup

Easy full stack backup operations on UNIX-like systems.

backup database encryption mongodb mysql postgresql ruby s3 slack

Last synced: 14 May 2025

https://github.com/meskyanichi/backup

Easy full stack backup operations on UNIX-like systems.

backup database encryption mongodb mysql postgresql ruby s3 slack

Last synced: 05 Dec 2024

https://github.com/AutoMQ/automq

AutoMQ is a cloud-native alternative to Kafka by decoupling durability to cloud storage services like S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.

apache-kafka aws azure cloud cloud-economics cloud-first cloud-native ebs gcp kafka messaging minio s3 serverless spot storage streaming

Last synced: 31 Mar 2025

https://github.com/s3tools/s3cmd

Official s3cmd repo -- Command line tool for managing S3 compatible storage services (including Amazon S3 and CloudFront).

backup cli hacktoberfest python s3 s3cmd storage-service

Last synced: 12 May 2025

https://github.com/qovery/replibyte

Seed your development database with real data ⚡️

aws backup cloud cloudnative database mongodb mysql postgres postgresql rust rust-lang s3

Last synced: 14 May 2025

https://github.com/Qovery/Replibyte

Seed your development database with real data ⚡️

aws backup cloud cloudnative database mongodb mysql postgres postgresql rust rust-lang s3

Last synced: 13 Mar 2025

https://github.com/apache/opendal

Apache OpenDAL: One Layer, All Storage.

azblob gcs hacktoberfest redis rust s3 storage

Last synced: 12 May 2025

https://github.com/iterate-ch/cyberduck

Cyberduck is a libre FTP, SFTP, WebDAV, Amazon S3, Backblaze B2, Microsoft Azure & OneDrive and OpenStack Swift file transfer client for Mac and Windows.

azure backblaze-b2 cloudfront cryptomator cyberduck dracoon dropbox ftp google-drive google-storage hubic irods onedrive onedrive-for-business onedrive-storage openstack-swift s3 sftp sharepoint webdav

Last synced: 13 May 2025

https://github.com/piskvorky/smart_open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

boto bz2 file gzip-stream hacktoberfest hdfs python s3 streaming streaming-data webhdfs

Last synced: 13 May 2025

https://github.com/aws/aws-sdk-js-v3

Modularized AWS SDK for JavaScript.

aws-sdk aws-sdk-js dynamodb javascript lambda nodejs s3 sns typescript

Last synced: 13 May 2025

https://github.com/roapi/roapi

Create full-fledged APIs for slowly moving datasets without writing a single line of code.

analytics arrow blob-storage cloud-native columnar datafusion datasets delta-lake graphql in-memory-database parquet query query-frontends rest-api rust s3 sql static-datasets

Last synced: 27 Mar 2025

https://github.com/shrinerb/shrine

File Attachment toolkit for Ruby applications

attachment background-jobs direct-upload file-upload filesystem metadata orm rack ruby s3 storage

Last synced: 13 May 2025

https://github.com/RaRe-Technologies/smart_open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

boto bz2 file gzip-stream hacktoberfest hdfs python s3 streaming streaming-data webhdfs

Last synced: 31 Mar 2025

https://github.com/janko-m/shrine

File Attachment toolkit for Ruby applications

attachment background-jobs direct-upload file-upload filesystem metadata orm rack ruby s3 storage

Last synced: 04 Apr 2025

https://github.com/peak/s5cmd

Parallel S3 and local filesystem execution tool.

aws cli filesystem go s3 s5cmd storage

Last synced: 12 May 2025

https://github.com/winscp/winscp

WinSCP is a popular free file manager for Windows supporting SFTP, FTP, FTPS, SCP, S3, WebDAV and local-to-local file transfers. A powerful tool to enhance your productivity with a user-friendly interface and automation options like .NET assembly.

ftp ftp-client ftps s3 scp sftp ssh webdav windows winscp

Last synced: 13 May 2025

https://github.com/minio/mc

Unix like utilities for object store

aws aws-s3 coreutils filesystem go golang s3 s3-bucket storage tools

Last synced: 12 May 2025

https://github.com/sa7mon/s3scanner

Scan for misconfigured S3 buckets across S3-compatible APIs!

aws bugbounty gcp infosec s3 s3scanner

Last synced: 13 May 2025

https://github.com/sa7mon/S3Scanner

Scan for misconfigured S3 buckets across S3-compatible APIs!

aws bugbounty gcp infosec s3 s3scanner

Last synced: 24 Mar 2025

https://github.com/minio/minio-go

MinIO Go client SDK for S3 compatible object storage

aws-signature client-sdk cloud fileuploader go libraries minio s3

Last synced: 13 May 2025

https://github.com/offen/docker-volume-backup

Backup Docker volumes locally or to any S3, WebDAV, Azure Blob Storage, Dropbox or SSH compatible storage

backup docker docker-compose docker-swarm docker-volume s3 ssh webdav

Last synced: 14 May 2025

https://github.com/peerdb-io/peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

bigquery cdc clickhouse cloud-native distributed-systems etl eventhubs kafka postgres postgresql realtime rust s3 snowflake sql stream-processing

Last synced: 14 May 2025

https://github.com/ladjs/lad

Node.js framework made by a former @expressjs TC and @koajs team member. Built for @forwardemail, @spamscanner, @breejs, @cabinjs, and @lassjs.

api async-await ava aws boilerplate bootstrap cloudfront framework javascript koa mongodb mongoose mvc nodejs passport prettier react-native redis s3 xo

Last synced: 13 May 2025

https://github.com/minio/minio-Go

MinIO Go client SDK for S3 compatible object storage

aws-signature client-sdk cloud fileuploader go libraries minio s3

Last synced: 12 Mar 2025

https://github.com/PeerDB-io/peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

bigquery cdc clickhouse cloud-native distributed-systems etl eventhubs kafka postgres postgresql realtime rust s3 snowflake sql stream-processing

Last synced: 28 Mar 2025

https://github.com/ByConity/ByConity

ByConity is an open source cloud data warehouse

bytedance clickhouse clickhouse-database cloud kubernets lakehouse olap s3 snowflake sql tiktok

Last synced: 18 Mar 2025

https://github.com/byconity/byconity

ByConity is an open source cloud data warehouse

bytedance clickhouse clickhouse-database cloud kubernets lakehouse olap s3 snowflake sql tiktok

Last synced: 29 Apr 2025

https://github.com/notea-org/notea

📒 Self hosted note taking app stored on S3

docker markdown minio note-taking s3

Last synced: 14 Apr 2025

https://github.com/QingWei-Li/notea

📒 Self hosted note taking app stored on S3

docker markdown minio note-taking s3

Last synced: 15 Dec 2024

https://github.com/rilldata/rill

Rill is a tool for effortlessly transforming data sets into powerful, opinionated dashboards using SQL. BI-as-code.

bi business-analytics csv data data-analysis data-visualization dataviz duckdb gcs golang parquet parquet-tools parquet-viewer s3 sql sql-editor svelte sveltejs sveltekit

Last synced: 13 May 2025

https://github.com/jstarfish/javakeeper

✍️ Java 工程师必备架构体系知识总结:涵盖分布式、微服务、RPC等互联网公司常用架构,以及数据存储、缓存、搜索等必备技能

algorithms elasticsearch java jvm kafka linux mysql redis s3 zookeeper

Last synced: 14 May 2025

https://github.com/humanmade/s3-uploads

The WordPress Plugin to Store Uploads on Amazon S3

s3 wordpress wordpress-plugin

Last synced: 11 May 2025

https://github.com/humanmade/S3-Uploads

The WordPress Plugin to Store Uploads on Amazon S3

s3 wordpress wordpress-plugin

Last synced: 01 Apr 2025

https://github.com/Jstarfish/JavaKeeper

✍️ Java 工程师必备架构体系知识总结:涵盖分布式、微服务、RPC等互联网公司常用架构,以及数据存储、缓存、搜索等必备技能

algorithms elasticsearch java jvm kafka linux mysql redis s3 zookeeper

Last synced: 24 Mar 2025

https://github.com/gaul/s3proxy

Access other storage backends via the S3 API

atmos aws-s3 azure backblaze-b2 google-cloud-storage openstack-swift proxy s3

Last synced: 16 May 2025

https://github.com/saltbo/zpan

A self-hosted cloud disk base on the cloud storage./ 一个基于云存储的网盘系统,用于自建私人网盘或企业网盘。

cloud cloud-disk cloud-storage disk oss qiniuyun s3 upyun

Last synced: 14 May 2025

https://github.com/rilldata/rill-developer

Rill is a tool for effortlessly transforming data sets into powerful, opinionated dashboards using SQL. BI-as-code.

bi business-analytics csv data data-analysis data-visualization dataviz duckdb gcs golang parquet parquet-tools parquet-viewer s3 sql sql-editor svelte sveltejs sveltekit

Last synced: 08 Mar 2025

https://github.com/dromara/x-file-storage

一行代码将文件存储到 本地、FTP、SFTP、WebDAV、谷歌云存储、阿里云OSS、华为云OBS、七牛云Kodo、腾讯云COS、百度云 BOS、又拍云USS、MinIO、 AWS S3、FastDFS、 Azure Blob Storage、金山云 KS3、美团云 MSS、京东云 OSS、天翼云 OOS、移动云 EOS、沃云 OSS、 网易数帆 NOS、Ucloud US3、青云 QingStor、平安云 OBS、首云 OSS、IBM COS、其它兼容 S3 协议的平台。后续即将支持 Samba、NFS

bos file-storage file-upload ftp java kodo minio oss s3 sftp spring uss webdav

Last synced: 15 May 2025

https://github.com/cloudkj/scar

Deploy static websites in seconds - with HTTPS, a global CDN, and custom domains.

aws cloudformation cloudfront route53 s3 serverless static-site

Last synced: 08 Apr 2025

https://github.com/MarSeventh/CloudFlare-ImgBed

基于 CloudFlare Pages 的开源文件托管解决方案(图床/文件床/网盘)

cloudflare-pages cloudflare-r2 img imgbed netdisk oss s3 telegram-bot

Last synced: 24 Mar 2025

https://github.com/aws/aws-toolkit-vscode

Amazon Q, CodeCatalyst, Local Lambda debug, SAM/CFN syntax, ECS Terminal, AWS resources

amazon amazon-q aws cdk cloudformation codecatalyst codewhisperer lambda s3 serverless vscode vscode-extension

Last synced: 14 May 2025

https://github.com/eduardolat/pgbackweb

🐘 Effortless PostgreSQL backups with a user-friendly web interface! 🌐💾

backup backups database postgres postgresql restore s3

Last synced: 13 Mar 2025

https://github.com/altinity/clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.

azblob backup clickhouse clickhouse-backup clickhousedump dump ftp gcs s3 sftp

Last synced: 13 May 2025

https://github.com/Altinity/clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.

azblob backup clickhouse clickhouse-backup clickhousedump dump ftp gcs s3 sftp

Last synced: 04 Apr 2025

https://github.com/AlexAkulov/clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.

azblob backup clickhouse clickhouse-backup clickhousedump dump ftp gcs s3 sftp

Last synced: 11 Dec 2024

https://github.com/swoodford/aws

A collection of bash shell scripts for automating various tasks with Amazon Web Services using the AWS CLI and jq.

amazon aws aws-cli awscli bash cidr cloudfront cloudwatch ec2 elb iam infrastructure-as-code jq json route53 s3 shell terraform vpc waf

Last synced: 14 May 2025

https://github.com/nhost/hasura-backend-plus

🔑Auth and 📦Storage for Hasura. The quickest way to get Auth and Storage working for your next app based on Hasura.

authentication docker graphql hasura jwt s3 storage typescript

Last synced: 07 Apr 2025

https://github.com/stavro/arc

:paperclip: Flexible file upload and attachment library for Elixir

attachment elixir s3 s3-storage storage

Last synced: 09 Apr 2025

https://github.com/algotraders/stock-analysis-engine

Backtest 1000s of minute-by-minute trading algorithms for training AI with automated pricing data from: IEX, Tradier and FinViz. Datasets and trading performance automatically published to S3 for building AI training datasets for teaching DNNs how to trade. Runs on Kubernetes and docker-compose. >150 million trading history rows generated from +5000 algorithms. Heads up: Yahoo's Finance API was disabled on 2019-01-03 https://developer.yahoo.com/yql/

algorithmic-trading backtesting deep-learning deep-learning-tutorial deep-neural-networks docker helm helm-charts iex iexcloud jupyter keras kubernetes minio options redis s3 stocks tensorflow tradier

Last synced: 15 May 2025

https://github.com/AlgoTraders/stock-analysis-engine

Backtest 1000s of minute-by-minute trading algorithms for training AI with automated pricing data from: IEX, Tradier and FinViz. Datasets and trading performance automatically published to S3 for building AI training datasets for teaching DNNs how to trade. Runs on Kubernetes and docker-compose. >150 million trading history rows generated from +5000 algorithms. Heads up: Yahoo's Finance API was disabled on 2019-01-03 https://developer.yahoo.com/yql/

algorithmic-trading backtesting deep-learning deep-learning-tutorial deep-neural-networks docker helm helm-charts iex iexcloud jupyter keras kubernetes minio options redis s3 stocks tensorflow tradier

Last synced: 30 Mar 2025

https://github.com/littlstar/s3-lambda

Lambda functions over S3 objects with concurrency control (each, map, reduce, filter)

concurrency lambda-functions node s3

Last synced: 16 May 2025

https://github.com/bareos/bareos

Bareos is a cross-network Open Source backup solution (licensed under AGPLv3) which preserves, archives, and recovers data from all major operating systems.

archiving backup backup-solution backup-utility bareos ceph compression cross-platform disaster-recovery encrypt gluster mysql postgresql python recover restore s3 security vmware

Last synced: 13 May 2025

https://github.com/jakejarvis/s3-sync-action

🔄 GitHub Action to sync a directory with a remote S3 bucket 🧺

actions aws aws-s3 ci deployment github-actions s3

Last synced: 17 Jan 2025

https://github.com/lensesio/stream-reactor

A collection of open source Apache 2.0 Kafka Connector maintained by Lenses.io.

awss3 azure-datalake cassandra connector cosmosdb elasticsearch eventhubs gcp http influxdb jms kafka kafka-connect mongodb mqtt pubsub s3

Last synced: 14 May 2025

https://github.com/SnellerInc/sneller

World's fastest log analysis: λ + SQL + JSON + S3

avx512 go high-performance indexless json log query-engine s3 schemaless serverless simd sql vectorized

Last synced: 23 Mar 2025

https://github.com/sambecker/exif-photo-blog

Photo blog, reporting 🤓 EXIF camera details (aperture, shutter speed, ISO) for each image.

ai approuter blob-storage exif nextjs ogimage photography postgresql r2 s3 vercel

Last synced: 14 May 2025

https://github.com/supabase/storage

S3 compatible object storage service that stores metadata in Postgres

gotrue object-storage postgres s3

Last synced: 14 May 2025

https://github.com/adobe/s3mock

A simple mock implementation of the AWS S3 API startable as Docker image, TestContainer, JUnit 4 rule, JUnit Jupiter extension or TestNG listener

amd64 arm64 aws aws-s3 docker java junit kotlin mock s3 testcontainers testing

Last synced: 13 May 2025

https://github.com/ysde/grafana-backup-tool

A Python-based application to backup Grafana settings by using the Grafana API

backup docker gcs grafana s3

Last synced: 14 May 2025

https://github.com/apache/ozone

Scalable, reliable, distributed storage system optimized for data analytics and object store workloads.

big-data hadoop kubernetes object-store s3 storage

Last synced: 11 Apr 2025

https://github.com/dvassallo/s3-benchmark

Measure Amazon S3's performance from any location.

aws benchmark performance-analysis s3

Last synced: 08 Apr 2025

https://github.com/adobe/S3Mock

A simple mock implementation of the AWS S3 API startable as Docker image, TestContainer, JUnit 4 rule, JUnit Jupiter extension or TestNG listener

amd64 arm64 aws aws-s3 docker java junit kotlin mock s3 testcontainers testing

Last synced: 08 Apr 2025

https://github.com/ctrox/csi-s3

A Container Storage Interface for S3

csi golang k8s kubernetes s3

Last synced: 28 Mar 2025

https://github.com/tmspzz/Rome

Carthage cache for S3, Minio, Ceph, Google Storage, Artifactory and many others

aws aws-s3 cache carthage ceph cocoapods fastlane fastlane-plugin framework hacktoberfest ios macos minio package-manager s3 swift tool tvos watchos

Last synced: 07 Dec 2024

https://github.com/fclairamb/ftpserver

Golang based autonomous FTP server with SFTP, S3, Dropbox, and Google Drive connectors.

afero ftp ftp-server go golang google-drive s3

Last synced: 14 May 2025