{"id":13509871,"url":"https://github.com/sjqzhang/go-fastdfs","last_synced_at":"2026-01-12T02:48:46.943Z","repository":{"id":38154664,"uuid":"164362812","full_name":"sjqzhang/go-fastdfs","owner":"sjqzhang","description":"go-fastdfs 是一个简单的分布式文件系统(私有云存储)，具有无中心、高性能，高可靠，免维护等优点，支持断点续传，分块上传，小文件合并，自动同步，自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).","archived":false,"fork":false,"pushed_at":"2025-01-02T03:11:38.000Z","size":39555,"stargazers_count":4029,"open_issues_count":0,"forks_count":759,"subscribers_count":127,"default_branch":"master","last_synced_at":"2025-04-27T20:05:54.423Z","etag":null,"topics":["breakpoint-resume","cloud-storage","cloudnative","distributed-file-storage","distributed-file-system","fastdfs","file-system","fileserver","filesystem","go","go-fastdfs","godfs","object-storage-service","oss","rsync","small-file-merge","storage","storage-servers","sync","tus"],"latest_commit_sha":null,"homepage":"https://gitee.com/sjqzhang/go-fastdfs","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sjqzhang.png","metadata":{"files":{"readme":"README-en.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-07T01:35:37.000Z","updated_at":"2025-04-27T08:17:09.000Z","dependencies_parsed_at":"2023-02-02T05:46:31.615Z","dependency_job_id":"8a14e762-3123-44aa-bf61-c9ca4781f123","html_url":"https://github.com/sjqzhang/go-fastdfs","commit_stats":{"total_commits":415,"total_committers":24,"mean_commits":"17.291666666666668","dds":"0.12289156626506026","last_synced_commit":"9594d1686f6718a07beb4e223888772ab683245f"},"previous_names":[],"tags_count":48,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sjqzhang%2Fgo-fastdfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sjqzhang%2Fgo-fastdfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sjqzhang%2Fgo-fastdfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sjqzhang%2Fgo-fastdfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sjqzhang","download_url":"https://codeload.github.com/sjqzhang/go-fastdfs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254010822,"owners_count":21999003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["breakpoint-resume","cloud-storage","cloudnative","distributed-file-storage","distributed-file-system","fastdfs","file-system","fileserver","filesystem","go","go-fastdfs","godfs","object-storage-service","oss","rsync","small-file-merge","storage","storage-servers","sync","tus"],"created_at":"2024-08-01T02:01:15.507Z","updated_at":"2026-01-12T02:48:46.937Z","avatar_url":"https://github.com/sjqzhang.png","language":"Go","funding_links":[],"categories":["开源类库","Go","Open source library","Repositories","go"],"sub_categories":["文件/存储","Files/Storage"],"readme":"# [中文](README.md) [English](README-en.md)\n## Vision: Provide users with the most simple, reliable and efficient distributed file system.\n![logo](doc/logo.png)\n\n\n# go-fastdfs is a distributed file system based on http protocol. It is based on the design concept of avenue to simple. All the simple design makes its operation and expansion more simple. It has high performance, high reliability and no center. , maintenance-free and so on.\n\n### Everyone is worried about such a simple file system. Is it not reliable, can it be used in a production environment? The answer is yes, it is efficient because it is simple, and it is stable because it is simple. If you are worried about the function, then run the unit test, if you are worried about the performance, then run the stress test, the project comes with it, run more confident ^_^.\n\nNote: Please read this article carefully before using it, especially [wiki](https://github.com/sjqzhang/go-fastdfs/wiki)\n\n- Support curl command upload\n- Support browser upload\n- Support HTTP download\n- Support multi-machine automatic synchronization\n- Support breakpoint download\n- Support configuration automatic generation\n- Support small file automatic merge (reduce inode occupancy)\n- Support for second pass\n- Support for cross-domain access\n- Support one-click migration\n- Support for parallel experience\n- Support for breakpoint resuming ([tus](https://tus.io/))\n- Support for docker deployment\n- Support self-monitoring alarm\n- Support image zoom\n- Support google authentication code\n- Support for custom authentication\n- Support cluster file information viewing\n- Use the universal HTTP protocol\n- No need for a dedicated client (support wget, curl, etc.)\n- class fastdfs\n- High performance (using leveldb as a kv library)\n- High reliability (design is extremely simple, using mature components)\n- No center design (all nodes can read and write at the same time)\n\n# advantage\n\n- No dependencies (single file)\n- Automatic synchronization\n- Failure automatic repair\n- Convenient maintenance by talent directory\n- Support different scenarios\n- Automatic file deduplication\n- Support for directory customization\n- Support to retain the original file name\n- Support for automatic generation of unique file names\n- Support browser upload\n- Support for viewing cluster file information\n- Support cluster monitoring email alarm\n- Support small file automatic merge (reduce inode occupancy)\n- Support for second pass\n- Support image zoom\n- Support google authentication code\n- Support for custom authentication\n- Support for cross-domain access\n- Very low resource overhead\n- Support for breakpoint resuming ([tus](https://tus.io/))\n- Support for docker deployment\n- Support for one-click migration (migrated from other system file systems)\n- Support for parallel experience (parallel experience with existing file system, confirm OK and then one-click migration)\n- Support token download token=md5(file_md5+timestamp)\n- Easy operation and maintenance, only one role (unlike fastdfs has three roles Tracker Server, Storage Server, Client), the configuration is automatically generated\n- Peer-to-peer (simplified operation and maintenance)\n- All nodes can read and write simultaneously\n\n\n[![asciicast](https://asciinema.org/a/258926.svg)](https://asciinema.org/a/258926)\n# Start the server (compiled, [download](https://github.com/sjqzhang/fastdfs/releases) experience)\n```\n./fileserver\n```\n\n\n# Command upload\n\n`curl -F file=@http-index-fs http://10.1.xx.60:8080/upload`\n\n\n# WEB upload (browser open)\n\n`http://127.0.0.1:8080`\n\n# Code upload (options see browser upload)\n## python\n```python\nimport requests\nurl = 'http://127.0.0.1:8080/upload'\nfiles = {'file': open('report.xls', 'rb')}\noptions={'output':'json','path':'','scene':''} #See browser upload options\nr = requests.post(url, files=files)\nprint(r.text)\n```\n## golang\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/astaxie/beego/httplib\"\n)\n\nfunc main()  {\n\tvar obj interface{}\n\treq:=httplib.Post(\"http://10.1.5.9:8080/upload\")\n\treq.PostFile(\"file\",\"path/to/file\")\n\treq.Param(\"output\",\"json\")\n\treq.Param(\"scene\",\"\")\n\treq.Param(\"path\",\"\")\n\treq.ToJSON(\u0026obj)\n\tfmt.Print(obj)\n}\n````\n\n## java\n\n```xml\n\u003cdependency\u003e\n\t\u003cgroupId\u003ecn.hutool\u003c/groupId\u003e\n\t\u003cartifactId\u003ehutool-all\u003c/artifactId\u003e\n\t\u003cversion\u003e4.4.3\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n```java\npublic static void main(String[] args) {\n    File file = new File(\"D:\\\\git\\\\2.jpg\");\n    HashMap\u003cString, Object\u003e paramMap = new HashMap\u003c\u003e();\n    paramMap.put(\"file\", file);\n    paramMap.put(\"output\",\"json\");\n    paramMap.put(\"path\",\"image\");\n    paramMap.put(\"scene\",\"image\");\n    String result= HttpUtil.post(\"http://xxxxx:xxxx/upload\", paramMap);\n    System.out.println(result);\n}\n```\n\n# Breakpoint resume example\n## golang\n```go\npackage main\n\nimport (\n    \"os\"\n    \"fmt\"\n    \"github.com/eventials/go-tus\"\n)\n\nfunc main() {\n    f, err := os.Open(\"100m\")\n    if err != nil {\n        panic(err)\n    }\n    defer f.Close()\n    // create the tus client.\n    client, err := tus.NewClient(\"http://10.1.5.9:8080/big/upload/\", nil)\n    fmt.Println(err)\n    // create an upload from a file.\n    upload, err := tus.NewUploadFromFile(f)\n    fmt.Println(err)\n    // create the uploader.\n    uploader, err := client.CreateUpload(upload)\n    fmt.Println(err)\n    // start the uploading process.\n   fmt.Println( uploader.Upload())\n\n}\n````\n[more langue](doc/upload.md)\n\n![deploy](doc/go-fastdfs-deploy.png)\n\nUniversal file authentication timing diagram\n![Universal file authentication timing diagram](doc/authentication2.png)\n\nFile google authentication timing diagram\n![File google authentication timing diagram](doc/authentication.png)\n\n# Please click [Feedback](https://github.com/sjqzhang/go-fastdfs/issues/new)\n\n\n # \u003cspan id=\"qa\"\u003eQ\u0026A\u003c/span\u003e\n\n- Best practice?\n```\nFirst, if it is mass storage, do not open the file token authentication function to reduce performance.\nSecond, try to use the standard upload, upload the business to save the path, and then connect the domain name when the business is used (convenient migration extension, etc.).\nThird, if you use breakpoints to continue transmission, you must use the file id to replace the path storage after uploading (how to replace the QA/API document), to reduce performance for subsequent access.\nFourth, try to use physical server deployment, because the main pressure or performance comes from IO\nFifth, the online business should use the nginx+gofastdfs deployment architecture (the equalization algorithm uses ip_hash) to meet the later functional scalability (nginx+lua).\nSixth, the online environment is best not to use container deployment, the container is suitable for testing and functional verification.\nSummary: The path of the file saved by the business reduces the conversion of the later access path, and the file access permission is completed by the service, so that the performance is the best and the versatility is strong (can be directly connected to other web servers).\n\nImportant reminder: If the small file merge function is enabled, it is impossible to delete small files later.\nUpload result description\nPlease use md5, path, scene field, others are added to be compatible with the old online system, and may be removed in the future.\n\n```\n\n- Is there an API document?\n[API documentation](doc/api.md)\n\n- Is there a management background?\n```\nHttps://github.com/perfree/go-fastdfs-web\n```\n- Is there any instructions for using the breakpoint upload?\n```\nHttps://github.com/tus\n```\n\n- In the WeChat discussion group, everyone asked about the performance of go-fastdfs?\n```\nBecause there are too many people asking, answer here in unison.\nThe file location of go-fastdfs is different from other distributed systems. Its addressing is directly located without any components, so the approximate time complexity is o(1)[file path location]\nThere is basically no performance loss. The project also has a pressure test script. You can carry out the pressure test yourself. Don’t discuss the problem too much in the group. People reply to the same question every time.\nEveryone will also feel that this group is boring.\n```\n\n\n\n- Files that have been stored using fastdfs can be migrated to go fastdfs (other migrations can be handled in a similar manner, and the speed experience is similar)?\n```\nThe answer is yes, the problem you are worried about is the path change, go fastdfs considers this for you.\nstep:\nFirst, download the latest version of go-fastdfs\nSecond, copy the original fastdfs file directory to the files directory of go-fastdfs (if there are a lot of files, you can reverse it, copy the fileserver, but keep the fileserver directory structure)\nThird, set the enable_migrate to true\nNote: All files in the files directory will be scanned during the migration process.\nSlower, set enable_migrate to false after migration is complete\n\nNote: The directory of go-fastdfs can not be changed, related to the synchronization mechanism, many students in the group, my files directory can not be customized, the answer is no.\nAs for whether or not I can use the soft link, I have not tested it and can test it myself.\n\n```\n\n- What is a cluster, how to manage multiple clusters with Nginx?\n```\n1. In go-fastdfs, a cluster is a group.\n2, please refer to the deployment diagram\nNote: When the support_group_manage parameter in the configuration is set to true, group information is automatically added to all urls.\nFor example: http://10.1.5.9:8080/group/status\nDefault: http://10.1.5.9:8080/status\nThe difference: more group, corresponding to the group parameter in the configuration, so mainly to solve a Nginx reverse proxy multiple groups (cluster)\nPlease refer to the deployment diagram for details.\n\n```\n\n\n- How to build a cluster?\n```\nFirst, download the compiled executable file (with the latest version)\nSecond, run the executable file (generate configuration)\nThird, modify the configuration\nPeers: increase the peer's http address\n\tan examination:\nHost: Is the automatic generation correct?\nPeer_id: Is it unique within the cluster?\nFourth, re-run the server\n5. Is the verification service OK?\n```\n\n\n- Is it suitable for mass storage?\n```\nAnswer: Suitable for mass storage\n```\n\n- How to upload a folder?\n```\n DIR=log \u0026\u0026 ls $DIR |xargs -n 1 -I {} curl -s -F path=$DIR -F file=@$DIR/{} http://10.1.50.90:8080/upload\n The log of the above command is the directory to be uploaded. If the code upload is a simple loop upload, it will be OK.\n```\n\n- How to scale the picture?\n```\nAdd width parameter in the download url\nFor example: http://127.0.0.1:8080/group1/haystack/5/124,0,27344,.jpg?download=0\u0026width=100\u0026height=100\nThe special explanation is: if you want to scale, please set the width or height to 0.\n```\n\n- How to display images directly in the browser?\n```\nAdd the download=0 parameter to the download url.\nFor example: http://127.0.0.1:8080/group1/haystack/5/124,0,27344,.jpg?download=0\n```\n\n\n- How to implement custom authentication upload and download?\n```\nFirst, the use of the 1.2.6 version of the go-fastdfs\nSecond, set the auth_url parameter (provided by the application)\nThird, the application implements the authentication permission interface (that is, the url of the second step), the parameter is auth_toke, ok, the authentication is passed, and the others are not passed.\nFourth, after the certification is passed, you can upload or download\n```\n\n\n- Need to install nginx yet?\n```\nGo-fastdfs itself is a high-performance web server, you don't need to install nginx when developing or testing.\nBut go-fastdfs has a single function, such as caching or redirection or other extensions, nginx has mature components\nTherefore, it is recommended to add a layer of nginx on the line, and then use nginx+lua to solve the scalability problem.\n```\n\n- Can I dynamically load the configuration?\n```\nAnswer: Yes, but update to the latest version\nstep:\n1) Modify the conf/cfg.json file\n2) Visit http://10.1.xx.60:8080/reload\n3) Note: Every node needs to do the same thing\n```\n\n\n- How to view cluster file information?\n```\nHttp://10.1.xx.60:8080/stat\n\nWhat should I do if there is a file error?\nPlease delete the stat.json file in the data directory. Restart the service, please recalculate the number of files automatically.\n\nOr call\nHttp://10.1.xx.60:8080/repair_stat\n\n```\n- How reliable can it be used in a production environment?\n```\nThis project has been used on a large scale in the production environment, such as fear of not meeting\nYou can stress test its features before use, any\nThe problem can be directly mentioned\n```\n\n- How to run the program in the background?\n```\nPlease use control to run the program later, as follows:\n    First, chmod +x control\n    Second, make sure the control and fileserver are in the same directory\n    Third, ./control start|stop|status #Start and start the sequence, stop, view the status, and so on.\n\n```\n\n\n- Can I have multiple servers on one machine?\n```\nNo, the high availability of the cluster has been considered at the beginning of the design. In order to ensure the true availability of the cluster, it must be different for ip, ip can not use 127.0.0.1\nError \"peers\": [\"http://127.0.0.1:8080\",\"http://127.0.0.1:8081\",\"http://127.0.0.1:8082\"]\nCorrect \"peers\": [\"http://10.0.0.3:8080\",\"http://10.0.0.4:8080\",\"http://10.0.0.5:8080\"]\n```\n- What should I do if the files are not synchronized?\n```\nUnder normal circumstances, the cluster automatically synchronizes the repair files every hour. (The performance is poor, it is recommended to turn off automatic repair in case of massive)\nWhat about the abnormal situation?\nAnswer: Manual synchronization (preferably at low peaks)\nHttp://172.16.70.123:7080/sync?date=20190117\u0026force=1 (Note: To be executed on a server with many files, related to pushing to another server)\nParameter description: date indicates the data of the day of synchronization. force 1. indicates whether to force synchronization of all the day (poor performance), 0. means that only failed files are synchronized.\n\nUnsynchronized situation:\n1) Originally running N sets, now suddenly join one to become N+1\n2) Originally running N sets, one machine has a problem and becomes N-1\n\nWhat if there is a multi-day data inconsistency? Can I synchronize all at once?\nThe answer is yes: (preferably at low peaks)\nHttp://172.16.70.123:7080/repair?force=1\n\n```\n\n- Does the file out of sync affect access?\n```\nAnswer: It will not affect, it will automatically repair the files that are not synchronized when the access is not available.\n```\n\n- How do I check the system status and description?\n```\nHttp://172.16.70.123:7080/status\nNote: (Fs.Peers is not equipped with this unit, if there is a problem with the problem)\nThis machine is Fs.Local\nSts[\"Fs.ErrorSetSize\"] = this.errorset.Cardinality() This will cause memory to increase\n\n```\n\n\n- How to compile (go1.9.2+)?\n```\nGit clone https://github.com/sjqzhang/go-fastdfs.git\nCd go-fastdfs\nMv vendor src\nPwd=`pwd`\nGOPATH=$pwd go build -o fileserver fileserver.go\n```\n\n- How to run a unit test (try to do it under linux)?\n```\n\nGit clone https://github.com/sjqzhang/go-fastdfs.git\nCd go-fastdfs\nMv vendor src\nPwd=`pwd`\nGOPATH=$pwd go test -v fileserver.go fileserver_test.go\n\n```\n\n\n\n- How to test?\n```\nstep:\nFirst, create a files folder\nSecond, copy gen_file.py to the files folder, generate a large number of files through python gen_file.py\nThird, put benchmark.py outside the files directory (that is, the same level as the files directory), press the python benchmark.py (note the ip in the benchmark.py)\nFirst use gen_file.py to generate a large number of files (note that if you want to generate large files, you can multiply the content by a large number)\nE.g:\n# -*- coding: utf-8 -*-\nImport os\nj=0\nFor i in range(0,1000000):\n    If i%1000==0:\n        j=i\n        Os.system('mkdir %s'%(i))\n    With open('%s/%s.txt'%(j,i),'w+') as f:\n        F.write(str(i)*1024)\nThen use benchmark.py for pressure measurement\nIt is also possible to perform pressure measurement simultaneously in multiple machines, and all nodes can be read and written simultaneously.\n```\n\n- Why is the code written in a file?\n```\nFirst, the current code is still very simple, no need to make it too complicated.\nSecond, the individual understands that modularity is not modular when multiple files are separated. You can use the IDE to look at the code structure, which is actually modular.\n```\n\n- Support for breakpoint downloads?\n```\nAnswer: Support\nCurl wget how\nWget -c http://10.1.5.9:8080/group1/default/20190128/16/10/2G\nCulr -C - http://10.1.5.9:8080/group1/default/20190128/16/10/2G\n```\n\n- How is Docker deployed?\n```\nstep:\nmethod one,\n    First, build a mirror\n    Docker build . -t fastdfs\n    Second, run the container (use the environment variable GO_FASTDFS_DIR to point to the storage directory.)\n    Docker run --name fastdfs -v /data/fastdfs_data:/data -e GO_FASTDFS_DIR=/data fastdfs\nMethod 2,\n    First, pull the mirror\n    Docker pull sjqzhang/go-fastdfs\n    Second, run the container\n    Docker run --name fastdfs -v /data/fastdfs_data:/data -e GO_FASTDFS_DIR=/data fastdfs\n\n```\n\n- How to upload large files or breakpoints?\n```\nThe general block uploading must be supported by the client, and the diversity of the language is difficult for the client to maintain, but the function of the block uploading is necessary, and a simple implementation idea is provided for this.\nOption One,\nSplit and merge with linux split cat, see the split and cat help.\nSplit: split -b 1M filename #1M per text\nMerge: cat x* \u003e filename #merge\nOption II,\nWith hjsplit\nHttp://www.hjsplit.org/\nSpecific self-realization\nthird solution,\nIt is recommended to implement the hjsplit split merge function with go, which has cross-platform capabilities. (Unrealized, waiting for you to come....)\nOption 4\nUse the built-in relay function (using the protocol for resumable uploads protocol, [Details](https://tus.io/))\n Note: Option 4, you can only specify one upload server, do not support simultaneous writes, and the uploaded url has changed.\n Original upload url: http://10.1.5.9:8080/\u003cgroup\u003e/upload\n Breakpoint upload url: http://10.1.5.9:8080/\u003cgroup\u003e/big/upload/\n The upload is completed, and then the second pass interface is used to obtain the file information.\n```\n\n- How to pass files in seconds?\n```\nAccess the upload interface by means of http get\nHttp://10.0.5.9:8080/upload?md5=filesum\u0026output=json\nParameter Description:\nMd5=sum(file) The digest algorithm of the file should be consistent with the algorithm of the server (the algorithm supports md5|sha1). If it is a breakpoint, you can use the id of the file, which is the id after urlolad.\nOutput=json|text The format returned\n \n```\n\n- How is the cluster planned and how is it expanded?\n```\nIt is recommended that in the early planning, try to purchase a large-capacity machine as a storage server. If you want two copies, use two to form a cluster. If you want three copies.\nLet the three units form a cluster. (Note that the best configuration for each server remains the same and uses raid5 disk arrays)\n\nIf you increase the availability, just add a new machine to the current cluster peers and fix the cluster.\nRepair method http://172.16.70.123:7080/repair?force=1 (recommended low peak change)\n\nHow to expand?\nFor the sake of simplicity and reliability, you can directly build a new cluster (build is to start the ./fileserver process, set the IP address of the peers, three or five minutes)\nIn the issue, chengyuansen suggested to me to use the increased capacity expansion feature. I feel that the complexity of the code logic and operation and maintenance is increased. I have not added this feature for the time being.\n\n```\n\n\n- Access restriction issues\n```\nFor security reasons, the management API can only be called inside the cluster or with 127.0.0.1.\n```\n\n\n\n\n- If you have any questions, please click [Reply](https://github.com/sjqzhang/go-fastdfs/issues/new)\n## Have questions?\n![QR code](doc/wechat.jpg)\n\n\n#### ![donation](doc/pay.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsjqzhang%2Fgo-fastdfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsjqzhang%2Fgo-fastdfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsjqzhang%2Fgo-fastdfs/lists"}