{"id":15095910,"url":"https://github.com/qinguoyi/osproxy","last_synced_at":"2025-04-06T00:09:38.533Z","repository":{"id":174598102,"uuid":"649745962","full_name":"qinguoyi/osproxy","owner":"qinguoyi","description":"对象存储分布式代理(object storage distrbuted proxy)，支持Docker一键部署","archived":false,"fork":false,"pushed_at":"2024-09-27T13:50:01.000Z","size":6220,"stargazers_count":275,"open_issues_count":3,"forks_count":47,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-27T09:11:20.587Z","etag":null,"topics":["gin","object-storage","proxy","s3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/qinguoyi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-05T14:41:33.000Z","updated_at":"2025-03-25T14:18:38.000Z","dependencies_parsed_at":"2024-01-06T14:22:45.424Z","dependency_job_id":"c2262d12-a81c-44e0-977b-e3d53e43d4df","html_url":"https://github.com/qinguoyi/osproxy","commit_stats":{"total_commits":24,"total_committers":5,"mean_commits":4.8,"dds":0.5416666666666667,"last_synced_commit":"8108ac328bb363e964e9d6471ee40d5f9efe8978"},"previous_names":["qinguoyi/osproxy"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qinguoyi%2Fosproxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qinguoyi%2Fosproxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qinguoyi%2Fosproxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qinguoyi%2Fosproxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/qinguoyi","download_url":"https://codeload.github.com/qinguoyi/osproxy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247415967,"owners_count":20935387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gin","object-storage","proxy","s3"],"created_at":"2024-09-25T15:43:38.640Z","updated_at":"2025-04-06T00:09:38.515Z","avatar_url":"https://github.com/qinguoyi.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## osproxy\n\n`osproxy`是一个使用Go语言开发的对象存储分布式代理(object-storage-distributed-proxy)，可以作为文件存储微服务，文件会在服务中转处理后再对接到对象存储，包括但不限于以下功能：\n\n* 分布式uid及秒传，支持相同文件不同命名\n* 分片读写，大文件上传，merge接口不用等待数据合并，分片上传完直接下载\n* 异步任务，易扩展的event-handler，支持分片合并及其他文件处理任务\n* 统一封装，降低业务接入复杂度，业务侧只需要存储文件uid\n* 代理下载，不直接暴露底层存储厂商及格式\n* 支持集群部署，proxy模块处理不同机器的分片转发\n* 支持Local/MinIO/腾讯COS/阿里OSS等对象存储，易于扩展\n* 支持Docker一键部署\n\n\n\n本项目仅用作学习交流，如果你正在学习Go语言，并且该项目给你的学习带来了一些帮助，欢迎star，欢迎交流。\n\n\n## 目录\n\n| [架构](#架构) | [功能](#功能) | [API](#API文档) | [技术栈](#技术栈) | [更新日志](#更新日志) | [本地](#本地调试) | [单机](#单机部署) | [集群](#集群部署) | [扩展](#扩展存储) | [如何接入](#如何接入osproxy) |  [庖丁解牛](#庖丁解牛) | [grpc实现](#grpc实现) |[讨论群](#讨论群) |\n|:---------:|:---------:|:-------------:|:-----------:|:-------------:|:-----------:|:-----------:|:-----------:|:-----------:|:-------------:|:-----------------:|:-----------------:|:-----------------:|\n\n## 架构\n\n![img.png](image/img.png)\n\n## 功能\n\n![img1.png](image/img1.png)\n\n## API文档\n\n![img2.png](image/img2.png)\n\n## 技术栈\n\n\n* Golang\n* Gin\n* GORM\n* Redis\n* MySQL/PostgreSQL\n* MinIO\n* Swagger\n* Zap\n* Nginx\n* Docker\n\n## 更新日志\n- [X] 新增本地存储\n\n## 本地调试\n**注意： 请提前准备好golang和docker环境；服务启动会自动创建表，但不会创建库，需要自己创建库.**\n\n\n**注意：本地调试，不使用nginx做反向代理.**\n### 基础服务\n* 启动pg数据库\n\n```shell\ndocker run -d --restart=always -p 5432:5432 --name=postgresql -v `pwd`/pg:/var/lib/postgresql/data -e POSTGRES_PASSWORD=123456 postgres\n```\n\n* 启动redis\n\n```shell\ndocker run -d --restart=always --name myredis -p 6379:6379 redis --requirepass \"123456\"\n```\n\n* 启动minio\n\n```shell\n# 9000是api端口，9090是控制台端口\ndocker run -p 9000:9000 -p 9001:9001 --name minio -d --restart=always -e \"MINIO_ACCESS_KEY=minioadmin\" -e \"MINIO_SECRET_KEY=minioadmin\" -v `pwd`/minio/data:/data -v `pwd`/minio/config:/root/.minio minio/minio server /data --console-address \":9001\"\n```\n### 配置修改\n* 修改配置文件\n\n```shell\napp:\n  env: prod\n  port: 8888                # 服务端口\n  app_name: osproxy         # 服务名称\n  app_url: http://127.0.0.1\n\nlog:\n  level: info               # 日志等级\n  root_dir: ./storage/logs  # 日志根目录\n  filename: app.log         # 日志文件名称\n  format: json              # 写入格式 可选json\n  show_line: true           # 是否显示调用行\n  max_backups: 3            # 旧文件的最大个数\n  max_size: 500             # 日志文件最大大小（MB）\n  max_age: 28               # 旧文件的最大保留天数\n  compress: true            # 是否压缩\n  enable_file: false        # 是否启用日志文件\n\ndatabase:\n  - db_name: default\n    driver: postgres                # 数据库驱动\n    host: 127.0.0.1                 # 服务地址\n    port: 5432                      # 端口号\n    database: postgres              # 数据库名称\n    username: postgres              # 用户名\n    password: 123456                # 密码\n    charset: utf8mb4                # 编码格式\n    max_idle_conns: 10              # 空闲连接池中连接的最大数量\n    max_open_conns: 100             # 打开数据库连接的最大数量\n    log_mode: info                  # 日志级别\n    enable_lg_log: false            # 是否开启自定义日志\n    enable_file_log_writer: false   # 是否打印到日志文件\n    log_filename: sql.log           # 日志文件名称\n\nredis:\n  host: 127.0.0.1        # 服务地址\n  port: 6379             # 服务端口\n  db: 0                  # 库选择\n  password: 123456       # 密码\n\nminio:\n  endpoint: 127.0.0.1:9000      # 服务地址\n  access_key_id: minioadmin     # 用户名\n  secret_access_key: minioadmin # 密码\n  use_ssl: false                # 密码\n  enabled: true                 # 是否启用\n\ncos:\n  appid: 12***63195                              # 唯一标识\n  region: ap-nanjing                             # 地域\n  secret_id: AKIDc4GCrG*******p9GApIZdU9V        # 用户名\n  secret_key: IMDoC1uYw*******1pNTRFpBJOJzSm     # 密码\n  enabled: false                                 # 是否启用\n\noss:\n  endpoint: oss-cn-beijing.aliyuncs.com          # 服务地址\n  access_key_id: LTAI5t*******X1tHK              # 用户名\n  access_key_secret: bfCt*******HudARdfwfvaoS    # 密码\n  enabled: false                                 # 是否启用\n  \nlocal:\n  enabled: false                                 # 是否启用\n```\n\n### 服务启动\n\n```shell\n# 设置goproxy\nLinux: go env -w GOPROXY=https://goproxy.cn,direct\nWindows: $env:GOPROXY = \"https://goproxy.cn\"\n\n# 拉取依赖\ngo mod tidy\ngo install github.com/swaggo/swag/cmd/swag@v1.8.1\n\n# 修改本地变量\n\napp/pkg/utils/constant.go\nLocalStore变量, 更新成本地可访问的目录\n\n\n# 生成api文档\nswag fmt -g cmd/main.go -d ./\nswag init -g cmd/main.go\nhttp://127.0.0.1:8888/swagger/index.html#/\n```\n\n### 服务测试\n* 服务部署验证\n  ```shell\n  # 修改服务地址和上传文件\n  baseUrl := \"http://127.0.0.1:8888\"\n  uploadFilePath := \"./xxx.jpg\"\n  # 启动服务\n  go run cmd/main.go\n  # 测试\n  go run test/httptest.go\n  ```\n* 下载断点续传测试\n  ```shell\n  wget -c url\n  ```\n\n### 本地构建镜像\n\n```shell\n# 这里使用dockerhub作为镜像仓库，请提前创建好账户密码\n\n# 登录，username和passwd替换成有效的账户密码\necho passwd | docker login --username=username --password-stdin\n\n# 本地构建，version替换成自定义的有效字符，比如v0.1\ndocker build -t osproxy:version -f deploy/Dockerfile  .\n\n# 镜像重命名，qinguoyi/object-storage-proxy请替换成你的username/repo\ndocker tag osproxy:version qinguoyi/object-storage-proxy:version\n\n# 镜像上传\ndocker push qinguoyi/object-storage-proxy:tag\n```\n\n## 单机部署\n\n\n**注意：单机部署，使用nginx做反向代理；安装docker compose**\n\n### 配置修改\n* http\n  * [docker-compose.yaml](deploy/http/docker-compose.yaml)\n  * [nginx.conf](deploy/http/nginx.conf)\n* https\n  * [docker-compose.yaml](deploy/https/docker-compose.yaml)\n  * [nginx.conf](deploy/https/nginx.conf)\n  * 自签名证书，只需要pem和key后缀的文件即可\n    * [如何本地实现自签名证书](https://learnku.com/articles/73105)\n\n* 修改config.yaml\n  * 使用的是compose，config中的各服务的host，指向的是compose service的名字，比如db，redis等\n\n* 机器路径下创建必需文件\n  ```shell\n  [root@k8s-node1 rpc]# ls\n  conf  docker-compose.yml  nginx.conf  server.key  server.pem\n  # conf是目录，conf/config.yaml\n  ```\n\n\n### 启动服务\n```shell\ndocker compose up -d\n```\n\n### 服务测试\n* 服务部署验证\n  ```shell\n  # 修改服务地址和上传文件\n  baseUrl := \"https://124.222.198.8\"\n  uploadFilePath := \"./xxx.jpg\"\n  \n  go run test/httptest.go\n  ```\n* 下载断点续传测试\n  ```shell\n  wget -c url\n  ```\n\n### 停止服务\n\n```shell\ndocker compose down\n```\n\n## 集群部署\n\n* 如果是docker-swarm或者k8s部署，注意修改获取local ip的方法，直接使用`GetClientIp`即可，集群内可以互通；\n* 如果是腾讯云或者阿里云的服务器，直接部署，服务器间是不能互通内网ip的，需要使用`GetOutBoundIP`获取外网ip。\n\n```golang\n// GetClientIp 获取本地网卡ip\nfunc GetClientIp() (string, error) {\n\taddrs, err := net.InterfaceAddrs()\n\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\n\tfor _, address := range addrs {\n\t\t// 检查ip地址判断是否回环地址\n\t\tif ipnet, ok := address.(*net.IPNet); ok \u0026\u0026 !ipnet.IP.IsLoopback() {\n\t\t\tif ipnet.IP.To4() != nil {\n\t\t\t\treturn ipnet.IP.String(), nil\n\t\t\t}\n\n\t\t}\n\t}\n\n\treturn \"\", errors.New(\"can not find the client ip address\")\n}\n\n// GetOutBoundIP 获取外网ip\nfunc GetOutBoundIP() (string, error) {\n\t//向查询IP的网站发送GET请求\n\tresp, err := http.Get(\"http://myexternalip.com/raw\")\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tdefer resp.Body.Close()\n\n\t//读取响应的内容\n\tbody, err := ioutil.ReadAll(resp.Body)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\treturn string(body), nil\n}\n```\n\n### 服务测试\n\n* 服务部署验证\n  ```shell\n  # 修改服务地址和上传文件\n  baseUrl := \"https://124.222.198.8\"\n  uploadFilePath := \"./xxx.jpg\"\n  \n  go run test/httptest.go\n  ```\n* 下载断点续传测试\n  ```shell\n  wget -c url\n  ```\n\n\n## 扩展存储\n\n* 实现CustomStorage接口并注册，用于存取数据\n\n```go\n// CustomStorage 存储\ntype CustomStorage interface {\n\t// MakeBucket 创建存储桶\n    MakeBucket(string) error\n\n    // GetObject 获取存储对象\n    GetObject(string, string, int64, int64) ([]byte, error)\n\n    // PutObject 上传存储对象\n    PutObject(string, string, string, string) error\n}\n\n// InitStorage 注册启动顺序\nfunc InitStorage(conf *config.Configuration) {\n    var storageHandler CustomStorage\n    if conf.Minio.Enabled {\n        storageHandler = NewMinIOStorage()\n        bootstrap.NewLogger().Logger.Info(\"当前使用的对象存储：Minio\")\n    } else if conf.Cos.Enabled {\n        storageHandler = NewCosStorage()\n        bootstrap.NewLogger().Logger.Info(\"当前使用的对象存储：COS\")\n    } else if conf.Oss.Enabled {\n        storageHandler = NewOssStorage()\n        bootstrap.NewLogger().Logger.Info(\"当前使用的对象存储：OSS\")\n    } else {\n        panic(\"当前对象存储都未启用\")\n    }\n\n    lgStorage = \u0026LangGoStorage{\n        Mux:     \u0026sync.RWMutex{},\n        Storage: storageHandler,\n    }\n    for _, bucket := range []string{\"image\", \"video\", \"audio\", \"archive\", \"unknown\", \"doc\"} {\n        if err := storageHandler.MakeBucket(bucket); err != nil {\n            panic(err)\n        }\n    }\n}\n```\n\n* 实现Plugin接口，用于初始化和健康检查\n\n```golang\n// Plugin 插件接口\ntype Plugin interface {\n    // Flag 是否启动\n    Flag() bool\n    // Name 插件名称\n    Name() string\n    // New 初始化插件资源\n    New() interface{}\n    // Health 插件健康检查\n    Health()\n    // Close 释放插件资源\n    Close()\n}\n```\n\n* 增加config.yaml配置项\n\n```shell\n# 类似下面的存储\nminio:\n  endpoint: 127.0.0.1:9000      # 服务地址\n  access_key_id: minioadmin     # 用户名\n  secret_access_key: minioadmin # 密码\n  use_ssl: false                # 密码\n  enabled: true                 # 是否启用\n\ncos:\n  appid: 12***63195                              # 唯一标识\n  region: ap-nanjing                             # 地域\n  secret_id: AKIDc4GCrG*******p9GApIZdU9V        # 用户名\n  secret_key: IMDoC1uYw*******1pNTRFpBJOJzSm     # 密码\n  enabled: false                                 # 是否启用\n\noss:\n  endpoint: oss-cn-beijing.aliyuncs.com          # 服务地址\n  access_key_id: LTAI5t*******X1tHK              # 用户名\n  access_key_secret: bfCt*******HudARdfwfvaoS    # 密码\n  enabled: false  \n\nlocal:\n  enabled: false                                 # 是否启用\n```\n## 如何接入osproxy\n**客户端对接存储代理服务，业务侧仅存储文件uid.**\n* 上传\n  * 客户端对接存储代理，判断秒传/断点续传，获取上传链接后上传数据到存储，返回文件uid，上报到业务侧\n  ```mermaid\n  sequenceDiagram\n  Note left of 客户端(web/api):上传接入osproxy\n  客户端(web/api) -\u003e\u003e+ osproxy: 请求秒传\n  alt 数据已上传\n  osproxy--\u003e\u003e-客户端(web/api) : 文件uid\n  客户端(web/api) -\u003e\u003e+ 业务侧: 文件uid\n  业务侧--\u003e\u003e- 客户端(web/api) : 上报成功\n  else 数据不存在\n  osproxy--\u003e\u003e+客户端(web/api) : 文件不存在\n  客户端(web/api) -\u003e\u003e+ osproxy: 请求上传链接\n  osproxy--\u003e\u003e-客户端(web/api) : 文件链接及上传uid\n  客户端(web/api) -\u003e\u003e+ osproxy: 上传数据\n  osproxy--\u003e\u003e-客户端(web/api) : 成功，返回文件uid\n  客户端(web/api) -\u003e\u003e+ 业务侧: 文件uid\n  业务侧--\u003e\u003e- 客户端(web/api) : 上报成功\n  end\n  ```\n* 下载\n  * 客户端从业务侧获取文件uid，从存储代理获取下载链接，返回文件数据。\n  ```mermaid\n  sequenceDiagram\n  Note left of 客户端(web/api):下载接入osproxy\n  客户端(web/api) -\u003e\u003e+ 业务侧: 获取业务侧文件\n  业务侧--\u003e\u003e- 客户端(web/api) : 文件uid\n  客户端(web/api) -\u003e\u003e+ osproxy: 文件uid\n  osproxy--\u003e\u003e-客户端(web/api) : 加密下载链接\n  ```\n## 庖丁解牛\n\n敬请期待...\n\n## grpc实现\n\n\n[grpc+ssl](https://github.com/qinguoyi/osproxy-grpc)\n\n## 讨论群\n* 微信群失效，加我微信，备注osproxy\n\n|    微信群    |     作者微信      |\n|:---------:|:-------------:|\n| ![group](image/group.jpg) | ![author](image/author.jpg)|\n\n\n\n\n\n\n## License\n\n\nDistributed under the Apache 2.0 License. See LICENSE for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqinguoyi%2Fosproxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqinguoyi%2Fosproxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqinguoyi%2Fosproxy/lists"}