{"id":24583461,"url":"https://github.com/itmisx/go-dfs","last_synced_at":"2025-12-15T21:56:02.630Z","repository":{"id":64302741,"uuid":"308537075","full_name":"itmisx/go-dfs","owner":"itmisx","description":"一款易用的分布式文件系统，免费、开源。","archived":false,"fork":false,"pushed_at":"2021-06-24T07:11:02.000Z","size":169,"stargazers_count":22,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T08:41:32.694Z","etag":null,"topics":["cloud","cluster","dfs","distributed-systems","fastdfs","go","godfs","storage"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/itmisx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-30T05:48:54.000Z","updated_at":"2024-12-13T14:43:49.000Z","dependencies_parsed_at":"2023-01-15T09:45:44.713Z","dependency_job_id":null,"html_url":"https://github.com/itmisx/go-dfs","commit_stats":null,"previous_names":["smally84/go-dfs"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itmisx%2Fgo-dfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itmisx%2Fgo-dfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itmisx%2Fgo-dfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itmisx%2Fgo-dfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/itmisx","download_url":"https://codeload.github.com/itmisx/go-dfs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251296530,"owners_count":21566623,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud","cluster","dfs","distributed-systems","fastdfs","go","godfs","storage"],"created_at":"2025-01-24T04:46:38.789Z","updated_at":"2025-12-15T21:55:57.578Z","avatar_url":"https://github.com/itmisx.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# go-dfs\n简单的分布式文件系统,该项目使用纯go语言编写，并完全开源，部署简单方便。\n\n[![LICENSE](https://raw.githubusercontent.com/smally84/go-dfs/4bffdfa2b020b98ccd8d618a53ac7c0294d26786/assets/mit.svg)](https://github.com/syndtr/goleveldb)\n[![LANGUAGE](https://raw.githubusercontent.com/smally84/go-dfs/28deed0c494ef141109a5e141c54b752c20923b0/assets/language.svg)]()\n[![DB](https://raw.githubusercontent.com/smally84/go-dfs/6851b5a0b570278c52135f77e812810278c2b898/assets/leveldb.svg)]()\n[![gopsutil](https://raw.githubusercontent.com/smally84/go-dfs/28deed0c494ef141109a5e141c54b752c20923b0/assets/gopsutil.svg)]()\n\n\n```\n客户端 ---\u003e tracker------\u003estorage1\n                 \\------\u003estorage2\n```\n\n# 项目说明\n本项目参考fastdfs逻辑进行简单实现，主要功能包括：\n- 文件上传\n- 临时文件上传\n- 文件删除\n- 文件下载\n- 文件多副本同步保存和删除\n- tracker自动容量均衡到不同的存储组\n\u003e 临时文件上传功能，需要二次确认，否则会被系统自动删除,\n\u003e 该功能需要使能配置: enable_temp_file\n\n# 使用说明\n\u003e特别说明，无论是tracker还是storage，使用的都是同个编译输出文件，仅仅配置不同而已\n## 一、源码安装\n- 1.clone源代码,编译出二进制文件\n```\ncd cmd\ngo build main.go -o dfs\n./dfs\n```\n- 2.配置文件\n将 configs/dsf.yml放到dfs可执行文件目录\n```\n#服务类型tracker or storage\nservice_type: \"storage\"\n#对外提供服务的ip和端口信息，用于storage上报自己的ip及端口信息\nservice_scheme: \"http\"\nservice_ip: 127.0.0.1\nservice_port: 9000\n#bind_port,服务运行的端口\nbind_port: 9000\n#默认语言\ndefault_lang: zh_cn\n#跟踪服务器的配置\ntracker:\n  node_id: 1\n  enable_temp_file: true\n# 存储服务器的配置    \nstorage:\n  #存储服务所属的group\n  group: group1 \n  #文件大小限制(byte)\n  file_size_limit: 10000000\n  #存储目录\n  storage_path: ./dfs\n  #跟踪服务器，可以有多个\n  trackers: \n    - http://127.0.0.1:9000\n```\n服务的类型：用server_type来定义。\n最小系统，要配置一个tracker，一个storage\n#### 配置参考\n- [tracker配置文件](https://github.com/smally84/easy-dfs/blob/main/docs/tracker.yml)\n- [storage1配置文件](https://github.com/smally84/easy-dfs/blob/main/docs/storage-1.yml)\n- [storage2配置文件](https://github.com/smally84/easy-dfs/blob/main/docs/storage-2.yml)\n## 二、docker安装\n参考docker-compose.yml配置\n```\nversion: \"3.7\"\nservices:\n  #tracker\n  tracker:\n    image: golang:1.15\n    container_name: \"dfs_tracker\"\n    expose:\n      - 9000\n    volumes:\n      - ./go-dfs/tracker:/app  \n    working_dir: /app\n    command: /app/tracker_app\n    restart: \"always\"\n    networks:\n      dfs: \n        ipv4_address: 172.20.0.2\n    logging:\n      options:\n        max-size: \"10M\"\n        max-file: \"5\"\n  storage_1:\n    image: golang:1.15\n    container_name: \"dfs_storage_1\"\n    expose:\n      - 9000\n    volumes:\n      - ./go-dfs/storage-1:/app\n    working_dir: /app\n    command: /app/storage_app\n    restart: \"always\"\n    networks:\n      dfs: \n        ipv4_address: 172.20.0.3\n    logging:\n      options:\n        max-size: \"10M\"\n        max-file: \"5\"\n  storage_2:\n    image: golang:1.15\n    container_name: \"dfs_storage_2\"\n    expose:\n      - 9000\n    volumes:\n      - ./go-dfs/storage-2:/app\n    working_dir: /app\n    command: /app/storage_app\n    restart: \"always\"\n    networks:\n      dfs: \n        ipv4_address: 172.20.0.4\n    logging:\n      options:\n        max-size: \"10M\"\n        max-file: \"5\"\n# 网络配置\nnetworks:\n  dfs:\n    name: dfs\n    ipam:\n      driver: default\n      config:\n        -\n          subnet: \"172.20.0.0/16\"\n```\n目录结构\n```\n-go-dfs\n |_ _tracker\n |      |_ _ configs\n |      |       |_ _ dfs.yml\n |      |_ _ tracker\n |_ _storage-1\n |      |_ _ configs\n |      |       |_ _ dfs.yml\n |      |_ _ storage\n |_ _storage-2\n        |_ _ configs\n        |       |_ _ dfs.yml\n        |_ _ storage\n```\ndfs.yml的配置请参考configs/dfs.yml\n其中：serverType要配置对应的服务类型，跟踪服务器为`tracker`,存储服务器为`storage`\n另外存储服务器时，要配置tracker服务器的host地址\n#### 配置参考\n- [tracker配置文件](https://github.com/smally84/easy-dfs/blob/main/docs/tracker.yml)\n- [storage1配置文件](https://github.com/smally84/easy-dfs/blob/main/docs/storage-1.yml)\n- [storage2配置文件](https://github.com/smally84/easy-dfs/blob/main/docs/storage-2.yml)\n# 接口说明\n统一返回格式：\n  ```\n  {\n    \"code\": 0,\n    \"data\": {},\n    \"msg\": \"\"\n  }\n  ```\n  code为0表示操作成功，非0表示有错误，msg为错误信息\n- 上传\n  - api: /upload\n  - headers:\n    - Content-Type:application/json\n  - method: post\n  - 参数:file(值为完整的文件路径)\n  - 返回结果示例：\n  ```\n  {\n    \"code\": 0,\n    \"data\": {\n        \"url\": \"/group1/2020/11/12/7/1326785062468919296.png\"\n    },\n    \"msg\": \"\"\n  }\n  ```\n- 上传确认\n  - api: /confirm\n  - headers:\n    - Content-Type:application/json\n  - method: post\n  - 参数:file(值为完整的文件路径)\n  - 备注：需要在配置文件中启用enable_temp_file:true\n- 下载\n  - api: /完整文件路径\n  - method: get\n- 删除\n  - api: /delete\n  - headers:\n    - Content-Type:application/json\n  - method: post\n  - 参数: file(值为完整的文件路径)\n  - 返回结果示例:\n  ```\n  {\n    \"code\": 0,\n    \"msg\": \"\"\n  }\n  ```\n# 测试用例  \n参考目录: easy-dfs/internal/app/dfs_test.go\n```go\nfunc TestDfs(t *testing.T) {\n\t// 启动tracker\n\tconfig1 := pkg.DsfConfigType{\n\t\tServiceType: \"tracker\",\n\t\tBindPort:    \"9000\",\n\t\tDefaultLang: \"zh_cn\",\n\t}\n\tconfig1.Tracker.NodeID = 1\n\tconfig1.Tracker.EnableTempFile = true\n\tgo Start(\u0026config1)\n\t// 启动storage\n\tconfig2 := pkg.DsfConfigType{\n\t\tServiceType:   \"storage\",\n\t\tServiceScheme: \"http\",\n\t\tServiceIP:     \"127.0.0.1\",\n\t\tServicePort:   \"9001\",\n\t\tBindPort:      \"9001\",\n\t\tDefaultLang:   \"zh_cn\",\n\t}\n\tconfig2.Storage.Group = \"group1\"\n\tconfig2.Storage.StoragePath = \"./dfs/1\"\n\tconfig2.Storage.Trackers = []string{\"http://127.0.0.1:9000\"}\n\tgo Start(\u0026config2)\n\n\t// 启动storage\n\tconfig3 := pkg.DsfConfigType{\n\t\tServiceType:   \"storage\",\n\t\tServiceScheme: \"http\",\n\t\tServiceIP:     \"127.0.0.1\",\n\t\tServicePort:   \"9002\",\n\t\tBindPort:      \"9002\",\n\t\tDefaultLang:   \"zh_cn\",\n\t}\n\tconfig3.Storage.Group = \"group1\"\n\tconfig3.Storage.StoragePath = \"./dfs/2\"\n\tconfig3.Storage.Trackers = []string{\"http://127.0.0.1:9000\"}\n\tgo Start(\u0026config3)\n\n\t\u003c-make(chan bool)\n}\n```\n# 项目工具\n- gin，高效的golang web框架\n- leveldb，基于golang的kv数据库\n\n# License\nUse of go-dfs is governed by the Mit License found at [LICENSE](https://github.com/smally84/easy-dfs/blob/main/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitmisx%2Fgo-dfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fitmisx%2Fgo-dfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitmisx%2Fgo-dfs/lists"}