{"id":13671003,"url":"https://github.com/senrok/yadal","last_synced_at":"2026-01-12T16:20:07.837Z","repository":{"id":61438641,"uuid":"550204985","full_name":"senrok/yadal","owner":"senrok","description":"Yet Another Data Access Layer: Accessing S3, POSIX in the same way. Deeply inspired by Databend's OpenDAL","archived":false,"fork":false,"pushed_at":"2022-12-09T15:39:41.000Z","size":70,"stargazers_count":18,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-06T10:44:26.317Z","etag":null,"topics":["cloud-native","data","go","minio","s3","storage"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/senrok/yadal","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/senrok.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-12T11:22:07.000Z","updated_at":"2024-11-01T18:18:18.000Z","dependencies_parsed_at":"2023-01-25T14:30:42.375Z","dependency_job_id":null,"html_url":"https://github.com/senrok/yadal","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/senrok%2Fyadal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/senrok%2Fyadal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/senrok%2Fyadal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/senrok%2Fyadal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/senrok","download_url":"https://codeload.github.com/senrok/yadal/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251145833,"owners_count":21543105,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-native","data","go","minio","s3","storage"],"created_at":"2024-08-02T09:00:55.541Z","updated_at":"2026-01-12T16:20:07.791Z","avatar_url":"https://github.com/senrok.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"./assets/sqaure-logo.png\" width=\"370\"\u003e\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003cb\u003eA project of SENROK Open Source\u003c/b\u003e\n\u003c/p\u003e\n\n\n# YaDAL \n\n\u003cp\u003e\n\u003ca href=\"https://goreportcard.com/report/github.com/senrok/yadal\"\u003e\n\u003cimg src=\"https://goreportcard.com/badge/github.com/senrok/yadal\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://godoc.org/github.com/senrok/yadal\"\u003e\n\u003cimg src=\"https://godoc.org/github.com/senrok/yadal?status.svg\" alt=\"GoDoc\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://github.com/senrok/yadal/actions/workflows/service_test_s3.yml\"\u003e\n\u003cimg src=\"https://github.com/senrok/yadal/actions/workflows/service_test_s3.yml/badge.svg\"/\u003e\n\u003c/a\u003e\n\u003ca href=\"https://github.com/senrok/yadal/actions/workflows/service_test_fs.yml\"\u003e\n\u003cimg src=\"https://github.com/senrok/yadal/actions/workflows/service_test_fs.yml/badge.svg\"/\u003e\n\u003c/a\u003e\n\u003c/p\u003e\n\n\n**Y**et **A**nother **D**ata **A**ccess **L**ayer: Access data freely, efficiently, without the tears 😢\n\ninspired by [Databend's OpenDAL](https://github.com/datafuselabs/opendal)\n\n## Table of contents\n\n- [Features](#features)\n- [Installation](#installation)\n- [Get started](#get-started)\n- [Documentation](#documentation)\n  - [Object](#object)\n    - [Handler](#handler)\n    - [IsExist](#isexist)\n    - [Metadata](#metadata)\n    - [Create a dir or a file](#create-a-dir-or-a-object)\n    - [Read](#read)\n    - [Range Read](#range-read)\n    - [Write](#write)\n    - [Delete](#delete)\n    - [List current directory](#list-current-directory)\n  - [Layers](#layers)\n    - [Retry](#retry)\n    - [Logging](#logging)\n  \n- [License](#license)\n\n## Features\n\n**Freely**\n- [x] Access different storage services in the same way\n- [ ] Behavior tests for all services\n  - [x] S3 and S3 compatible services\n  - [x] fs: POSIX compatible filesystem\n\n**Without the tears 😢**\n- [x] Powerful Layer Middlewares\n  - [x] Auto Retry (Backoff)\n  - [x] Logging Layer\n  - [ ] Tracing Layer\n  - [ ] Metrics Layer\n- [ ] Compress/Decompress \n- [ ] Service-side encryption\n\n**Efficiently**\n- Zero cost: mapping to underlying API calls directly\n- Auto metadata reuse: avoid extra metadata calls\n\n\n## Installation\n\n```bash\ngo get -u github.com/senrok/yadal\n```\n\n## Get started\n\n```go\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"github.com/senrok/yadal\"\n\t\"github.com/senrok/yadal/providers/s3\"\n\t\"os\"\n)\n\nfunc main() {\n\tacc, _ := s3.NewDriver(context.Background(), s3.Options{\n\t\tBucket:    os.Getenv(\"Bucket\"),\n\t\tEndpoint:  os.Getenv(\"Endpoint\"),\n\t\tRoot:      os.Getenv(\"Root\"),\n\t\tRegion:    os.Getenv(\"Region\"),\n\t\tAccessKey: os.Getenv(\"AccessKey\"),\n\t\tSecretKey: os.Getenv(\"SecretKey\"),\n\t})\n\top := yadal.NewOperatorFromAccessor(acc)\n\t// Create object handler\n\to := op.Object(\"test_file\")\n\n\t// Write data\n\tif err := o.Write(context.Background(), []byte(\"Hello,World!\")); err != nil {\n\t\treturn\n\t}\n\n\t// Read data\n\tbs, _ := o.Read(context.Background())\n\tfmt.Println(bs)\n\tname := o.Name()\n\tfmt.Println(name)\n\tpath := o.Path()\n\tfmt.Println(path)\n\tmeta, _ := o.Metadata(context.Background())\n\n\t_ = meta.ETag()\n\t_ = meta.ContentLength()\n\t_ = meta.ContentMD5()\n\t_ = meta.Mode()\n\n\t// Delete\n\t_ = o.Delete(context.Background())\n\n\t// Read Dir\n\tds := op.Object(\"test-dir/\")\n\titer, _ := ds.List(context.Background())\n\tfor iter.HasNext() {\n\t\tentry, _ := iter.Next(context.Background())\n\t\tentry.Path()\n\t\tentry.Metadata()\n\t}\n}\n\n```\n\nSee the [Documentation](https://godoc.org/github.com/senrok/yadal) or explore more [examples](examples)\n\n## Documentation\n\n\u003ca href=\"https://godoc.org/github.com/senrok/yadal\"\u003e\n\u003cimg src=\"https://godoc.org/github.com/senrok/yadal?status.svg\" alt=\"GoDoc\"\u003e\n\u003c/a\u003e\n\n### Object\n#### Handler\n```go\nfunc ExampleOperator_Object() {\n\tacc, _ := newS3Accessor()\n\top := NewOperatorFromAccessor(acc)\n\t// it returns a object.Object handler \n\tobject := op.Object(\"test\")\n\tfmt.Println(object.ID())\n\tfmt.Println(object.Path())\n\tfmt.Println(object.Name())\n}\n```\n#### Create a dir or a object\nIt creates an empty object, like using the following linux commands:\n- `touch path/to/file`\n- `mkdir path/to/dir/`\n\nThe behaviors: \n- create on existing dir will succeed.\n- create on existing file will overwrite and truncate it.\n\na dir:\n```go\nfunc ExampleOperator_Object_create() {\n\tacc, _ := newS3Accessor()\n\top := NewOperatorFromAccessor(acc)\n\tobject := op.Object(\"test/\")\n\t_ = object.Create(context.TODO())\n}\n```\n\na object:\n```go\nfunc ExampleOperator_Object_create() {\n\tacc, _ := newS3Accessor()\n\top := NewOperatorFromAccessor(acc)\n\tobject := op.Object(\"test\")\n\t_ = object.Create(context.TODO())\n}\n```\n\n#### IsExist\n\n```go\nfunc ExampleOperator_Object_isExist() {\n\tacc, _ := newS3Accessor()\n\top := NewOperatorFromAccessor(acc)\n\tobject := op.Object(\"test\")\n\tfmt.Println(object.IsExist(context.TODO()))\n}\n```\n\n#### Metadata\n\n```go\nfunc ExampleOperator_Object_metadata() {\n\tacc, _ := newS3Accessor()\n\top := NewOperatorFromAccessor(acc)\n\tobject := op.Object(\"test\")\n\tmeta, err := object.Metadata(context.TODO())\n\tif err == errors.ErrNotFound {\n\t\tfmt.Println(\"not found\")\n\t\treturn\n\t}\n\tfmt.Println(meta.LastModified())\n\tfmt.Println(meta.ETag())\n\tfmt.Println(meta.ContentLength())\n\tfmt.Println(meta.ContentMD5())\n\tfmt.Println(meta.Mode())\n}\n```\n\n#### Read\n\nIt returns a io.ReadCloser holds the whole object.\n\n```go\nfunc ExampleOperator_Object_read() {\n\tacc, _ := newS3Accessor()\n\top := NewOperatorFromAccessor(acc)\n\tobject := op.Object(\"test\")\n\t_ = object.Write(context.TODO(), []byte(\"Hello,World!\"))\n\treader, _ := object.Read(context.TODO())\n\n\t_, _ = io.ReadAll(reader)\n}\n```\n\n\n#### Range Read\n\nIt returns a io.ReadCloser holds specified range of object .\n\n```go\nfunc ExampleOperator_Object_rangeRead() {\n\tacc, _ := newS3Accessor()\n\top := NewOperatorFromAccessor(acc)\n\tobject := op.Object(\"test\")\n\t_ = object.Write(context.TODO(), []byte(\"Hello,World!\"))\n\treader, _ := object.RangeRead(context.TODO(), options.NewRangeBounds(options.Range(0, 11)))\n\t_, _ = object.RangeRead(context.TODO(), options.NewRangeBounds(options.Start(2)))\n\t_, _ = object.RangeRead(context.TODO(), options.NewRangeBounds(options.End(11)))\n\n\t_, _ = io.ReadAll(reader)\n}\n```\n\n#### Write\n\nIt writes bytes into object.\n\n```go\nfunc ExampleOperator_Object_write() {\n\tacc, _ := newS3Accessor()\n\top := NewOperatorFromAccessor(acc)\n\tobject := op.Object(\"test\")\n\t_ = object.Write(context.TODO(), []byte(\"Hello,World!\"))\n}\n```\n\n#### Delete\n\nIt deletes object.\n\n```go\nfunc ExampleOperator_Object_delete() {\n\tacc, _ := newS3Accessor()\n\top := NewOperatorFromAccessor(acc)\n\tobject := op.Object(\"test\")\n\t_ = object.Delete(context.TODO())\n}\n```\n\n#### List current directory\n\nIt returns a [interfaces.ObjectStream](./interfaces/stream.go).\n\n```go\nfunc ExampleOperator_Object_list() {\n\tacc, _ := newS3Accessor()\n\top := NewOperatorFromAccessor(acc)\n\tobject := op.Object(\"dir/\")\n\tstream, _ := object.List(context.TODO())\n\tfor stream.HasNext() {\n\t\tentry, _ := stream.Next(context.TODO())\n\t\tif entry != nil {\n\t\t\tfmt.Println(entry.Path())\n\t\t\tfmt.Println(entry.Metadata().LastModified())\n\t\t\tfmt.Println(entry.Metadata().ContentLength())\n\t\t}\n\t}\n}\n```\n\n### Layers\n#### Retry\n```go\nfunc ExampleOperator_Layer_retry() {\n\tacc, _ := newS3Accessor()\n\top := NewOperatorFromAccessor(acc)\n\n\tseed := time.Now().UnixNano()\n\trandom := rand.New(rand.NewSource(seed))\n\n\t// retry layer\n\tretryLayer := layers.NewRetryLayer(\n\t\tlayers.SetStrategy(\n\t\t\tstrategy.Limit(5),\n\t\t\tstrategy.BackoffWithJitter(\n\t\t\t\tbackoff.BinaryExponential(10*time.Millisecond),\n\t\t\t\tjitter.Deviation(random, 0.5),\n\t\t\t),\n\t\t),\n\t)\n\n\top.Layer(retryLayer)\n}\n```\n\nSee more backoff [strategies](https://github.com/Rican7/retry)\n\n\n#### Logging\n\n```go\nfunc ExampleOperator_Layer_logging() {\n\tacc, _ := newS3Accessor()\n\top := NewOperatorFromAccessor(acc)\n\n\t// logger\n\tlogger, _ := zap.NewProduction()\n\ts := logger.Sugar()\n\t\n\t// logging layer\n\tloggingLayer := layers.NewLoggingLayer(\n\t\tlayers.SetLogger(\n\t\t\tlayers.NewLo\n\t\t\tggerAdapter(s.Info, s.Infof),\n\t\t),\n\t)\n\n\top.Layer(loggingLayer)\n}\n```\n\n\n\n## License\n\nThe Project is licensed under the [Apache License, Version 2.0](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsenrok%2Fyadal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsenrok%2Fyadal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsenrok%2Fyadal/lists"}