{"id":17242864,"url":"https://github.com/imcuttle/mouploader","last_synced_at":"2025-04-14T03:33:52.350Z","repository":{"id":45112783,"uuid":"68200673","full_name":"imcuttle/moUploader","owner":"imcuttle","description":"mobile Uploader Plugin","archived":false,"fork":false,"pushed_at":"2024-07-30T16:13:22.000Z","size":18,"stargazers_count":30,"open_issues_count":2,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-23T02:34:03.220Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/imcuttle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-09-14T11:32:54.000Z","updated_at":"2024-07-09T09:49:39.000Z","dependencies_parsed_at":"2024-10-21T15:57:48.620Z","dependency_job_id":null,"html_url":"https://github.com/imcuttle/moUploader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imcuttle%2FmoUploader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imcuttle%2FmoUploader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imcuttle%2FmoUploader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imcuttle%2FmoUploader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/imcuttle","download_url":"https://codeload.github.com/imcuttle/moUploader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248815507,"owners_count":21165937,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T06:14:16.798Z","updated_at":"2025-04-14T03:33:52.313Z","avatar_url":"https://github.com/imcuttle.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 分片上传与断点续传解决方案\n\n上传文件，基本上是每一个网站应用都会具备的一个功能。对于一个网络存储应用，对于上传功能要求更是迫切。  \n如今市面上成熟上传插件，如`WebUploader`，\"体积太大\"，不适合于移动端上传；再加上作为一位程序员的\"操守\"，当然还是更喜欢自己造轮子。\n\n于是花了一天半时间，`MoUploader`应运而生。为什么叫`MoUploader`呢？  \n`Mo`表示`Mobile`(其实更是因为我的绰号moyu)\n\n\u003c!--more--\u003e\n\n## 关于实现原理\n\n- 首先需要明确，上传这东西不仅仅是只需要前端就能完成的很好的，需要前端后端统一数据格式，从而实现断点续传。（所以，该文适合于全栈工程师，至少是想成为）\n- 还有，为什么需要分片，不分片能实现断点续传吗？分片是为了充分利用网络带宽，加快上传速度；不分片也是能够实现断点续传的。详细参考 [HTML5文件上传组件深度剖析](http://fex.baidu.com/blog/2014/04/html5-uploader/).   \n分片上传与断点续传之间没有很直接的关系.\n\n 好了，进入正题\n    - 实现断点续传的前提是需要服务器记录某文件的上传进度，那么根据什么判断是不是同一个文件呢？可以利用文件内容求md5码，如果文件过大，求取md5码也是一个很长的过程，所以对于大文件，只能针对某一段数据进行计算，加上服务器对cookie用户信息的判断，得到相对唯一的key\n    \n    - 在前端页面，需要将文件按照一定大小进行分片，一次请求只发送这一小片数据，所以我们可以同时发起多个请求。但一次同时请求的连接数不宜过多，服务器负载过重\n    \n    对于文件分片操作，H5具有十分强大的File API，直接利用File对象的slice方法即可得到Blob对象.  \n    至于同时传输数据的连接数控制逻辑，就需要花点脑子思考了\n\n    - 前端把数据顺利得传给服务器了，服务器只需要按照数据中给的开始字节位置，与读取到的文件片段数据，写入文件即可\n \n 更多信息就看源码吧！[MoUploader](https://github.com/moyuyc/moUploader)\n \n## 功能实现\n\n- 文件结构\n\n```\nfile-upload/\n├── bower_components/ # bower包\n├── db.js   # 数据操作接口\n├── demo.html\n├── md5.json # 数据\n├── mouploader.js # 源码\n├── README.md \n└── server.js # demo.html服务, 建立在4040端口\n\n1 directories, 8 files.\n```\n(打印文件目录树使用的是自己写的[print-dir](https://github.com/moyuyc/directory-tree))\n\n- 怎么使用\n\n    1. 引入script，amd/cmd/...，\n    2. 使用MoUploader\n    ```js\n    input.onchange = function (e) {\n        var self = this;\n        var moUploader = MoUploader({ \n            files: this.files,\n            uploadUrl: '/upload',\n            request: false,\n            onBeforeUpload: function (index) {\n                if(index\u003e=0) {\n                    self.files[index].progress = appendUploading(self.files[index], index)\n                }\n            },\n            onOverAllProgress: function (index, loaded, total) {\n                console.log(loaded / total)\n                //setProgress(loaded / total, self.files[index].progress)\n            },\n            onLoad: function (index, chunkIndex, chunksNum) {\n                console.log('onLoad', this, arguments)\n            },\n            onAbort: function (index, chunkIndex, chunksNum) {\n                console.log('onAbort', this, arguments)\n            },\n            onError: function (index, chunkIndex, chunksNum) {\n                console.log('onError', this, arguments)\n            },\n            onContinue: function (file, md5, index) {\n                return new Promise(function(reslove, reject) {\n                    var xhr = new XMLHttpRequest()\n                    xhr.open('GET', '/getFile?md5='+md5, true);\n                    xhr.send(null);\n                    xhr.addEventListener('readystatechange', function () {\n                        if(xhr.readyState === 4 \u0026\u0026 xhr.status === 200) {\n                            var json = JSON.parse(xhr.responseText);\n                            log(json)\n                            reslove(json.pos)\n                        }\n                    })\n                })\n            }\n        })\n        \n        // pause or continue upload\n        // if index \u003c 0, will run for all files\n        // moUploader.pause(index);\n        // moUploader.continue(index);    \n    }\n    \n    ```\n    \n    3. 配置选项\n    ```js\n    var default_ops = {\n        // chunk Size: byte\n        chunkSize: (1\u003c\u003c20) * 5,\n        // Number: request Number.\n        // Array: files requests.\n        // Boolean: open or close Slice, if false, chunkSize don't work.\n        request: 3,\n        files: [],\n        uploadUrl: '/',\n        // function: get uploaded pos.\n        // arguments: file, md5, index.\n        // need return a promise object which will return uploaded pos.\n        onContinue: null,\n        // if false, md5 will be setted by filename.\n        md5: true,\n        // md5Size: slice file 0 - md5Size for calculate md5\n        md5Size: (1\u003c\u003c20) * 50,\n        // called when before upload.\n        // arguments: file index or -1 (will begin upload)\n        onBeforeUpload: null,\n        // function: uploading progress listener.\n        // *only listen one request.*\n        // arguments: index, chunkIndex, chunksNum, loaded, total.\n        onProgress: null,\n        // function: overall uploading progress listener.\n        // arguments: index, loaded, total\n        onOverAllProgress: null,\n        // function: called when one request is ended.\n        // arguments: index, chunkIndex, chunksNum\n        onLoad: null,\n        // function: called when one request is aborted.\n        // arguments: index, chunkIndex, chunksNum\n        onAbort: null,\n        // function: called when one request happens error.\n        // arguments: index, chunkIndex, chunksNum\n        onError: null\n    }\n    \n    ```\n    \n    4. 服务器数据处理 (Node.js)\n    \n    数据分段写入文件\n    ```js\n    function writeBuffer(bf, path, pos) {\n        var fd = fs.openSync(path, 'a+');\n        fs.writeSync(fd, bf, 0, bf.length, Number(pos) || 0)\n        console.log(`write buffer, pos: ${pos}, path: ${path}, length: ${bf.length}`)\n    }\n    \n    function store(param, chunks) {\n        param.chunks = param.chunks || 1\n        param.chunk = param.chunk || 0\n        var p = path.join('./upload', param.name)\n        var bf = Buffer.concat(chunks);\n\n        var json = db.get(param.md5);\n        if(json) {\n            json.pos = parseInt(json.pos!=null?json.pos : 0)\n            json.size = parseInt(json.size!=null?json.size : 0)\n        }\n        if(!json || (json.pos+json.size) \u003c= param.pos) {\n            // 新的数据pos比数据库中大，更新数据\n            param.size = bf.length\n            db.set(param.md5, param)\n            db.save();\n            writeBuffer(bf, p, param.pos || 0)\n        }\n    }\n    \n    var multiparty = require('multiparty')\n    var form = new multiparty.Form({\n        autoFields: true,\n        autoFiles: false,\n    });\n    \n    form.on('part', (part) =\u003e {\n        form.on('aborted', () =\u003e {\n            //意外退出或者暂停都会保存数据\n            console.log('aborted');\n            store(param, chunks)\n        })\n\n        var chunks = []\n        part.on('data', (data) =\u003e {\n            if(part.filename) {\n                chunks.push(data)\n            }\n        }).on('end', () =\u003e {\n            console.log('end')\n            store(param, chunks)\n        })\n\n    });\n    form.on('field', (name, value) =\u003e {\n        param[name] = value;\n    });\n    ```\n    \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimcuttle%2Fmouploader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimcuttle%2Fmouploader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimcuttle%2Fmouploader/lists"}