{"id":20986633,"url":"https://github.com/dpwgc/piefilemigrate","last_synced_at":"2026-05-17T18:03:05.821Z","repository":{"id":144485549,"uuid":"595708120","full_name":"dpwgc/PieFileMigrate","owner":"dpwgc","description":"一个小型文件定时增量/全量迁移脚本，可用于将服务器内的日志、图像、音频等小型文件批量迁移至对象存储服务或云盘服务中。","archived":false,"fork":false,"pushed_at":"2023-02-09T09:56:09.000Z","size":60,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-13T11:25:52.982Z","etag":null,"topics":["boltdb","file","ftp","golang","http","oss","redis","s3","sync"],"latest_commit_sha":null,"homepage":"https://gitee.com/dpwgc/PieFileMigrate","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpwgc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-31T16:43:38.000Z","updated_at":"2024-01-30T12:19:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"ffbd54b0-6a81-4b8d-ae1a-211b32f0b483","html_url":"https://github.com/dpwgc/PieFileMigrate","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dpwgc/PieFileMigrate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpwgc%2FPieFileMigrate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpwgc%2FPieFileMigrate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpwgc%2FPieFileMigrate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpwgc%2FPieFileMigrate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpwgc","download_url":"https://codeload.github.com/dpwgc/PieFileMigrate/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpwgc%2FPieFileMigrate/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265563007,"owners_count":23788628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["boltdb","file","ftp","golang","http","oss","redis","s3","sync"],"created_at":"2024-11-19T06:14:24.664Z","updated_at":"2026-05-17T18:03:00.771Z","avatar_url":"https://github.com/dpwgc.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PieFileMigrate \n## 一个基于Go开发的小型文件批量迁移脚本\n### `待完善`\n\n***\n\n### 简介\n\nPieFileMigrate是一个基于Go开发的小型文件批量迁移脚本，可将服务器内的一个或多个目录内的全部文件通过HTTP、S3（已实现，待优化）、FTP（未实现）的方式定时增量/全量同步上传至其他服务器。\n\n***\n\n### 说明\n\n* `文件迁移流程` ：定时扫描本地目录下的所有文件，将未被标记的文件上传至指定服务器中，如果上传成功，则标记该文件。\n* `聚合发送文件` : 为减少网络IO压力，使用HTTP方式推送文件时，可将多个文件写入一个HTTP请求里推送。\n* `如何标记文件` ：如果一个文件上传成功，则将它的完整路径及它的最近修改时间写入KV数据库中（可以选择用本地BoltDB存储或者用线上Redis存储），每次上传文件前都会检查该文件路径是否已经在KV数据库中，如果已存在就说明该文件已经被上传过了，无需重复上传。\n* `增量迁移方式` ：自行修改 config/application.yaml 配置文件中的 migrate-file-age-limit 配置项，迁移目录下最近一段时间内更新的文件。\n* `全量迁移方式` ：将 config/application.yaml 配置文件中的 migrate-file-age-limit 设成0，迁移目录下的所有文件。\n* `重新全量迁移` ：如果想重新全量迁移整个目录，可以先停用脚本，将标记用的KV数据库表删除/清空，重新运行脚本即可。\n\n***\n\n### 【HTTP迁移示例】以HTTP方式迁移文件至指定服务接口\n\n* 将本地 `./data` 目录下的文件增量迁移上传至 `http://127.0.0.1:8011/demo/uploadTargetBatchApi` 接口\n* 迁移周期为每60秒一次，每次仅迁移在当前时间前`172800`秒内新增/更新的文件，分配五个消费者并发推送文件，单次HTTP请求推送100个文件。\n\n#### 填写脚本应用配置 config/application.yaml\n\n```yaml\napplication:\n\n  # http服务端口（用于监控脚本运行情况）\n  server-port: 8012\n  # 文件迁移方式（http、ftp、s3）\n  migrate-mode: http\n  # 文件上传记录存储介质（boltdb、redis）\n  storage-media: boltdb\n\n  # 迁移工作者数组（可同时迁移多个目录）\n  workers:\n    # 需要进行迁移的源文件目录（两个工作者不能同时迁移一个目录）\n    - source-path: ./data\n      # 增量迁移文件时间限制（单位：秒，只迁移{migrate-file-time-limit}秒内更新的文件，小于等于0则全部迁移）\n      migrate-file-age-limit: 172800\n      # 定时迁移任务的cron表达式\n      job-cron: 0/60 * * * * ?\n\n  # 内置mq\n  mq:\n    # 内置mq容量大小\n    max-size: 10000000\n    # 内置mq单批次消费消息数量（必须大于0）\n    consume-batch: 100\n    # 内置mq消费者数量（文件迁移并发数）\n    consumer-num: 5\n\n  # 日志配置\n  log:\n    # 日志文件最长存活时间（单位：天）\n    maxAge: 7\n```\n\n#### 填写HTTP上传配置 config/http.yaml\n\n```yaml\nhttp:\n  # 文件上传的目标地址\n  target-url: http://127.0.0.1:8011/demo/uploadTargetBatchApi\n```\n\n#### 自定义文件接收接口示例\n\n* 该接口的 `Content-Type` 必须为 `multipart/form-data` ，需要接收5个参数\n\n| 参数名       | 参数类型     | 参数说明       |\n|-----------|----------|------------|\n| token     | string   | 权限校验令牌     |\n| fileNames | string[] | 文件名称列表     |\n| filePaths | string[] | 文件原路径列表    |\n| modTimes  | string[] | 文件最近修改时间列表 |\n| files     | file[]   | 文件列表       |\n\n* 需要自行编写接收逻辑，以下是示例接口代码\n\n```\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/gin-gonic/gin\"\n\t\"io/ioutil\"\n\t\"net/http\"\n)\n\nfunc main() {\n\tgin.SetMode(gin.ReleaseMode)\n\tgin.DefaultWriter = ioutil.Discard\n\t// 启动HTTP服务\n\tr := gin.Default()\n\tr.POST(\"/demo/uploadTargetBatchApi\", uploadTargetBatchApi)\n\t//加载端口号\n\t_ = r.Run(\":8011\")\n}\n\nfunc uploadTargetBatchApi(c *gin.Context) {\n\n\t// 身份校验token（自行处理校验逻辑）\n\ttoken := c.PostForm(\"token\")\n\tfmt.Println(\"token: \", token)\n\n\t// 文件名称列表\n\tfileNames := c.PostFormArray(\"fileNames\")\n\tfmt.Println(\"fileNames: \", fileNames)\n\n\t// 文件原路径列表\n\tfilePaths := c.PostFormArray(\"filePaths\")\n\tfmt.Println(\"filePaths: \", filePaths)\n\n\t// 文件最后修改时间列表\n\tmodTimes := c.PostFormArray(\"modTimes\")\n\tfmt.Println(\"modTimes: \", modTimes)\n\n\t// 文件列表\n\tform, _ := c.MultipartForm()\n\tfiles := form.File[\"files\"]\n\t\n\t// 遍历文件列表\n\tfor i := 0; i \u003c len(files); i++ {\n\t\t//保存文件到本地\n\t\tif err := c.SaveUploadedFile(files[i], \"./data/\"+fileNames[i]); err != nil {\n\t\t\tc.String(http.StatusBadRequest, \"fail\")\n\t\t\tfmt.Println(\"SaveUploadedFile: \", err)\n\t\t\treturn\n\t\t}\n\t}\n\n\t// 注意，必须要确保文件保存成功后才返回 code 200\n\tc.String(http.StatusOK, \"ok\")\n}\n```\n\n***\n\n### 【S3示例】以S3方式迁移文件至指定对象存储桶\n\n#### 填写脚本应用配置 config/application.yaml\n\n```yaml\n# 参考上文HTTP示例中的应用配置\nmigrate-mode: s3\n```\n\n#### 填写S3上传配置 config/s3.yaml\n\n```yaml\ns3:\n  endpoint: http://xxx.xxx.cn\n  region: default\n  bucket: test\n  access-key: xxxxxx\n  secret-key: xxxxxxxxx\n  path-style: true\n```\n\n***\n\n### 迁移工作者运行状态监控\n\n* 监控接口地址 `/worker/monitor`\n\n\u003e `GET` http://127.0.0.1:8012/worker/monitor\n\n* 接口返回数据\n\n```json\n{\n  \"./data\": {\n    \"lastMigrateStartTime\": \"2023-02-02T14:21:30.00088+08:00\",\n    \"lastMigrateEndTime\": \"2023-02-02T14:21:30.001168+08:00\"\n  }\n}\n```\n\n* 返回字段说明：`./data` 该工作者负责的目录 `lastMigrateStartTime` 最近一次迁移开始时间 `lastMigrateEndTime` 最近一次迁移结束时间","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpwgc%2Fpiefilemigrate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpwgc%2Fpiefilemigrate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpwgc%2Fpiefilemigrate/lists"}