美文网首页java全栈
大文件分段上传,控制接口并发数量

大文件分段上传,控制接口并发数量

作者: 家有饿犬和聋猫 | 来源:发表于2022-03-15 14:43 被阅读0次

说明:使用axios方式上传,文件不能过大,因为过多的连续Ajax请求会使后台崩溃,接口报错;


录屏.gif
第一步:使用input或者antd_upload获取文件
image.png
第二步:调接口获取文件段数,分段列表和分段尺寸;使用slice方法,分段读取文件为blob
   let dataMsg = await createMultipart({
       fileSize: file.size, // 传参数
       filename: file.name
   }).then(
       (rem) => {
           return rem.data;
       },
       (err) => {
           return upFailed(file, onUpload); // 如果接口报错,使用upFailed方法处理
       }
   );
   let urlList = dataMsg?.parts || []; // 分段列表
   let DEFAULT_SIZE = dataMsg?.partSize; // 分段尺寸

   for (let i = 0; i < urlList.length; i++) {
       let url = urlList[i]['url'];
       let fname = encodeURIComponent(file.name);
       let start = i * DEFAULT_SIZE;
       let stepFile;
       if (i === urlList.length - 1) {
          // 使用slice方法,分段读取文件为blob 
           stepFile = file.slice(start, -1); // 如果是最后一段,直接截取剩下的所有内容
       } else {
           stepFile = file.slice(start, start + DEFAULT_SIZE); // 分割文件
       }
       urlList[i]['stepFile'] = stepFile;
       urlList[i]['fname'] = fname;
       urlList[i]['uid'] = file.uid;
   }

urlList已准备好

image.png
数据说明: {
fname: '使用encodeURIComponent 编码过的文件名',
partNumber: '段数序号,合并时候使用',
stepFile: '截取的文件'
uid: 'antd组件生成的文件唯一值',
url: '上传该段文件的路径'
};
第三步:循环urlList,上传每一段文件

准备工作:单个文件上传方法

 const detalItem = ({ url, stepFile, fname, partNumber }) => {
        return new Promise((resolve, reject) => {
            fileAxios({
                url,
                method: 'PUT',
                data: stepFile,
                headers: {
                    'Content-Type': '',
                    'Content-disposition': `filename*=utf-8\'zh_cn\'${fname}`
                }
            })
                .then((res) => {
                    let str = res.headers.etag.split('"').join('');
                    resolve({ eTag: str, partNumber });
                })
                .catch((err) => {
                    reject({ eTag: '', partNumber });
                });
        });
    };


准备工作:并发上传,控制每次上传的接口数量,防止上传接口数量过多,浏览器崩溃。

参数说明:
poolLimit(数字类型):表示限制的并发数;
array(数组类型):表示任务数组;
iteratorFn(函数类型):表示迭代函数,用于实现对每个任务项进行处理,该函数会返回一个 Promise 对象或异步函数;
onUpload: 进度条

async function asyncPool(poolLimit, array, iteratorFn, onUpload) {
    const ret = []; // 存储所有的异步任务
    const executing = []; // 存储正在执行的异步任务
    for (const item of array) {
        // 结束运行
        if (endExecution.end && endExecution.uid === item?.uid) {
            return;
        }

       --------重点开始---------------
        // 调用iteratorFn函数创建异步任务
        const p = Promise.resolve().then(() => iteratorFn(item, array));
        ret.push(p); // 保存新的异步任务

        // 当poolLimit值小于或等于总任务个数时,进行并发控制
        if (poolLimit <= array.length) {
            // 当任务完成后,从正在执行的任务数组中移除已完成的任务
            const e = p.then(() => executing.splice(executing.indexOf(e), 1));
            executing.push(e); // 保存正在执行的异步任务
            if (executing.length >= poolLimit) {
                await Promise.race(executing); // 等待较快的任务执行完成
            }
        }
       --------重点结束---------------

       // 进度条
        onUpload &&
            onUpload({
                loaded: ret.length < array.length ? ret.length : ret.length - 1, // 等到接口合并完成,再返回100%
                total: array.length,
                uid: item['uid'],
                endAction: endAction // 如果用户删除文件,调用此函数,结束文件上传
            }); // 进度条
    }
    return Promise.all(ret);  // 集合多个返回结果
}

整合方法,开始上传

 let etags = [];
    try {
        etags = await asyncPool(5, urlList, detalItem, onUpload);  // 重点
    } catch {
        // 上传失败
        etags = [];
        endExecution.end = true;
        endExecution.uid = file?.uid;
        file['url'] = '';
        file['link'] = '';
        file['attachmentID'] = '';
        return upFailed(file, onUpload);
    }

请求中,保证5条并发数,如果5条中有请求结束了,自动补上

image.png

创建请求,请求全部发出,结束后合并文件

image.png

文件上传完的结果,etags。
eTag是每段文件的唯一值,
partNumber: 文件顺序。后端根据这个数据表来合并文件,避免顺序乱了。


image.png
第四步: 通知后端,合并文件
 let params = {
        attachmentID: dataMsg?.attachmentID,
        uploadID: dataMsg?.uploadID
    };

    if (endExecution.end && endExecution.uid === file?.uid) {
        console.log('删除文件,结束上传,调用结束上传接口,后端清除已经上传的数据');
        cancelMultipart(params);
        file['url'] = '';
        file['link'] = '';
        file['attachmentID'] = '';
        return { file: file, upResult: '' };
    }
      --------重点开始---------------
      result = await completeMultipart({
            ...params,
            etags
        });
        if (etags) {
            onUpload &&
                onUpload({
                    loaded: 100,
                    total: 100,
                    uid: file['uid'],
                    endAction: endAction
                }); // 进度条
        }
        let presignedURL = result?.data?.presignedURL;
        // console.log('result', result, 'presignedURL', presignedURL);
        file['url'] = presignedURL;
        file['link'] = presignedURL;
        file['attachmentID'] = dataMsg?.attachmentID;
        file['status'] = 'done';

        --------重点结束---------------
        return { file: file, upResult: '' };

附加功能:
1 返回进度条onUpload,原理: 当前发出去的请求数,除以总条数
2 结束请求endAction,应用场景,文件正在上传中,删除文件,结束接口调用

全部代码:

React上传组件:

import React, { Component } from 'react';
import { Upload, Progress, Tooltip, Modal } from 'antd';
const { Dragger } = Upload;
export default class List extends Component {
    constructor(props) {
        super(props);
        this.state = {
            fileList: [
                // {
                //     uid: '-1',
                //     name: 'image.png',
                //     status: 'done',
                //     url:
                //         'https://zos.alipayobjects.com/rmsportal/jkjgkEfvpUPVyRjUImniVslZfWPnJuuZ.png'
                // }
            ],
            visible: false,
            upLoading: false
        };
        this.formRef = React.createRef();
        this.littleRef = React.createRef();
        this.departmentRef = React.createRef();
        this.smalledepartmentRef = React.createRef();
    }
 // 进度条
  handleProgress = (progressEvent) => {
        const num = (progressEvent.loaded / progressEvent.total) * 100;
        let percent = num >= 100 ? 100 : num.toFixed(2) * 1;
        const { fileList } = this.state;

        this[`${progressEvent?.uid}_up`] = progressEvent;

        // if (progressEvent.loaded > 5) {
        //     progressEvent.endAction();
        // }

        this.setState({
            fileList: fileList.map((p) => {
                if (p?.uid === progressEvent?.uid) {
                    p['percent'] = percent;
                }
                return p;
            })
        });
    };
    // 删除文件
    onRemove = (file) => {
        if (!file?.status) {
            // 删除正在上传的文件,结束调用
            this[`${file?.uid}_up`] &&
                this[`${file?.uid}_up`]?.endAction &&
                this[`${file?.uid}_up`]?.endAction(file?.uid);
        }
        const { fileList } = this.state;
        let newList = fileList.filter((p) => p.uid !== file.uid);
        let loading = false;
        for (let v of newList) {
            if (!v?.status) {
                loading = true;
                break;
            }
        }
        this.setState({
            fileList: newList,
            upLoading: loading
        });
    };
    beforeUpload = (file, fileLists) => {
        console.log('打印file:', file);
        let repeat = [...this.state.fileList, ...fileLists];
        let obj = {};
        let noRepeat = repeat.reduce((pur, item) => {
            if (!obj[item?.uid]) {
                obj[item?.uid] = true;
                pur.push(item);
            }
            return pur;
        }, []);
        this.setState({ fileList: noRepeat, upLoading: true });
        commonUpload({ file, onUpload: this.handleProgress })
            .then((rem) => {
                const { fileList } = this.state;
                var data = {};
                for (var key in rem.file) {
                    data[key] = rem.file[key];
                }

                let newFilelist = fileList
                    .map((p) => {
                        if (p) {
                            if (p?.uid === data?.['uid']) {
                                p = { ...p, ...data };
                            }
                            return p;
                        }
                    })
                    .filter((p) => p?.status !== 'error');
                if (isNotEmpty(rem.file)) {
                    this.setState({
                        fileList: newFilelist
                    });
                }
            })
            .finally(() => {
                const { fileList } = this.state;
                // 批量上传完成,关闭loading
                let flag = true;
                for (let item of fileList) {
                    if (!item?.status) {
                        flag = false;
                        break;
                    }
                }
                flag && this.setState({ upLoading: false });
                // console.log('this.state.fqwFile', JSON.parse(this.state.fqwFile));
            });
        // 阻止默认上传
        return false;
      };
    render(){
             <Dragger
                           fileList={fileList}
                          className="drag-uploader"
                            onPreview={this.handlePreview} // 点击文件链接或预览图标时的回调
                            onRemove={this.onRemove}
                            multiple={true} // 支持多个文件一起上传
                          // onChange={this.onfileChange}
                          itemRender={(originNode, file, currFileList) => (
                           <UploadListItem
                                            originNode={originNode}
                                            file={file}
                                            currFileList={currFileList}
                                            fileList={fileList}
                                        />
                                    )}
                                    beforeUpload={this.beforeUpload}
                                    showUploadList={{
                                        showPreviewIcon: false,
                                        downloadIcon: true
                                    }}
                          >
                                    {fileList.length >= 15 ? null : UploadButton}
                          </Dragger>
}


进度条uploadListItem.jsx文件

/*
 * @desc   文件上传,自定义上传列表项, 带进度条
 * @author fqw
 */

import React, { Component } from 'react';
import { Progress, Tooltip } from 'antd';
import Cns from 'classnames';
import './index.scss';

const UploadListItem = ({ originNode, file, current, fileList }) => {
    const errorNode = <Tooltip title={file['response']}>{originNode.props.children}</Tooltip>;
    let have = file.percent < 100;
    return (
        <div
            className={Cns('ant-upload-draggable-list-item', have && 'progressIng')}
            style={{ cursor: 'move' }}
            key={file.percent}
        >
            {file.status === 'error' ? errorNode : originNode}
            {have && <Progress style={{ width: '100px' }} percent={file.percent} />}
        </div>
    );
};
export default UploadListItem;

fileAxios.js文件

import { message } from 'antd';
import axios from 'axios';
import { cancelMultipart } from './common';
import {
    getUserPresignedurl,
    submitFileMsg,
    createMultipart,
    completeMultipart
} from 'services/common';

// 结束运行
let endExecution = {
    end: false,
    uid: ''
};
let endAction = (uid) => {
    endExecution.end = true;
    endExecution.uid = uid;
};
let upFailed = (file, onUpload) => {
    endExecution.end = true;
    endExecution.uid = file?.uid;
    file['status'] = 'error';
    file['response'] = '上传失败,请重试';
    message.warning({
        content: `文件 ${file.name} 上传失败,请重试`,
        duration: 5
    });
    onUpload &&
        onUpload({
            loaded: 1, // 结束进度条,不显示
            total: 1,
            uid: file['uid'],
            endAction: endAction
        }); // 进度条
    return { file, upResult: false };
};
// 普通上传
const uploadFile = async (file, onUpload) => {
    // 获取上传接口的路径
    let urlRest = await getUserPresignedurl({ filename: file.name, fileSize: file.size }).then(
        (rem) => {
            if (rem.status === 200) {
                // file['uid'] = rem.data['attachmentID'];
                file = Object.assign(file, rem.data);
                return rem.data;
            }
        }
    );
    // 获取文件类型
    let fileType = file.name.split('.').slice(-1)[0];
    let typesObj = {
        jpg: 'image/jpeg',
        jpe: 'image/jpeg',
        jpeg: 'image/jpeg',
        png: 'image/png',
        gif: 'image/gif',
        bmp: 'application/x-bmp',
        wbmp: 'image/vnd.wap.wbmp',
        ico: 'image/x-icon',
        pdf: 'application/pdf',
        ppt: 'application/x-ppt',
        doc: 'application/msword',
        xls: 'application/vnd.ms-excel'
    };
    let url = urlRest.presignedURL;
    const fileAxios = axios.create();
    let fname = encodeURIComponent(file.name);
    let upBool = false;
    upBool = await fileAxios({
        url,
        method: 'PUT',
        data: file,
        headers: {
            'Content-Type': typesObj[fileType] || '',
            'Content-disposition': `filename*=utf-8\'zh_cn\'${fname}`
        },
        onUploadProgress: (arg) => {
            arg.uid = file.uid;
            onUpload(arg);
        }
    })
        .then((res) => {
            return res.status === 200;
        })
        .catch((err) => {
            return false;
        });
    // 上传失败,结束运行
    if (!upBool) {
        return upFailed(file, onUpload);
    }
    // 获取文件下载或预览链接
    let upResult = await submitFileMsg({
        filename: file.name,
        fileSize: file.size,
        attachmentID: urlRest.attachmentID
    }).then(
        (rem) => {
            file['url'] = rem.data['link'];
            file['status'] = 'done';
            file['attachmentID'] = urlRest.attachmentID;
            file = Object.assign(file, rem.data);
            return true;
        },
        (err) => {
            return false;
        }
    );
    let copy = JSON.parse(JSON.stringify(file));
    copy['name'] = file.name;
    return { file: copy, upResult };
};

// poolLimit(数字类型):表示限制的并发数;
// array(数组类型):表示任务数组;
// iteratorFn(函数类型):表示迭代函数,用于实现对每个任务项进行处理,该函数会返回一个 Promise 对象或异步函数
// onUpload: 进度条

async function asyncPool(poolLimit, array, iteratorFn, onUpload) {
    const ret = []; // 存储所有的异步任务
    const executing = []; // 存储正在执行的异步任务
    for (const item of array) {
        // 结束运行
        if (endExecution.end && endExecution.uid === item?.uid) {
            console.log('结束上传0');
            return;
        }
        // 调用iteratorFn函数创建异步任务
        const p = Promise.resolve().then(() => iteratorFn(item, array));
        ret.push(p); // 保存新的异步任务

        // 当poolLimit值小于或等于总任务个数时,进行并发控制
        if (poolLimit <= array.length) {
            // 当任务完成后,从正在执行的任务数组中移除已完成的任务
            const e = p.then(() => executing.splice(executing.indexOf(e), 1));
            executing.push(e); // 保存正在执行的异步任务
            if (executing.length >= poolLimit) {
                await Promise.race(executing); // 等待较快的任务执行完成
            }
        }

        onUpload &&
            onUpload({
                loaded: ret.length < array.length ? ret.length : ret.length - 1, // 等到接口合并完成,再返回100%
                total: array.length,
                uid: item['uid'],
                endAction: endAction
            }); // 进度条
    }
    return Promise.all(ret);
}

// 分段上传
const multiPartUpload = async (file, onUpload = null) => {
    // 获取段数
    let dataMsg = await createMultipart({
        fileSize: file.size, // 传参数
        filename: file.name
    }).then(
        (rem) => {
            return rem.data;
        },
        (err) => {
            return upFailed(file, onUpload); // 如果接口报错,使用upFailed方法处理
        }
    );
    let urlList = dataMsg?.parts || []; // 分段列表
    let DEFAULT_SIZE = dataMsg?.partSize; // 分段尺寸

    for (let i = 0; i < urlList.length; i++) {
        let url = urlList[i]['url'];
        let fname = encodeURIComponent(file.name);
        let start = i * DEFAULT_SIZE;
        let stepFile;
        if (i === urlList.length - 1) {
            stepFile = file.slice(start, -1); // 如果是最后一段的话,直接截取剩下的所有内容
        } else {
            stepFile = file.slice(start, start + DEFAULT_SIZE); // 分割文件
        }
        urlList[i]['stepFile'] = stepFile;
        urlList[i]['fname'] = fname;
        urlList[i]['uid'] = file.uid;
    }

    const fileAxios = axios.create();

    const detalItem = ({ url, stepFile, fname, partNumber }) => {
        return new Promise((resolve, reject) => {
            fileAxios({
                url,
                method: 'PUT',
                data: stepFile,
                headers: {
                    'Content-Type': '',
                    'Content-disposition': `filename*=utf-8\'zh_cn\'${fname}`
                }
            })
                .then((res) => {
                    let str = res.headers.etag.split('"').join('');
                    resolve({ eTag: str, partNumber });
                })
                .catch((err) => {
                    reject({ eTag: '', partNumber });
                });
        });
    };

    let etags = [];
    try {
        etags = await asyncPool(5, urlList, detalItem, onUpload);
    } catch {
        // 上传失败
        etags = [];
        endExecution.end = true;
        endExecution.uid = file?.uid;
        file['url'] = '';
        file['link'] = '';
        file['attachmentID'] = '';
        return upFailed(file, onUpload);
    }
    let params = {
        attachmentID: dataMsg?.attachmentID,
        uploadID: dataMsg?.uploadID
    };

    if (endExecution.end && endExecution.uid === file?.uid) {
        cancelMultipart(params);
        file['url'] = '';
        file['link'] = '';
        file['attachmentID'] = '';
        return { file: file, upResult: '' };
    }

    let result = null;
    // console.log('etags', etags);
    // 上传完合并文件
    try {
        result = await completeMultipart({
            ...params,
            etags
        });
        if (etags) {
            onUpload &&
                onUpload({
                    loaded: 100,
                    total: 100,
                    uid: file['uid'],
                    endAction: endAction
                }); // 进度条
        }
        let presignedURL = result?.data?.presignedURL;
        // console.log('result', result, 'presignedURL', presignedURL);
        file['url'] = presignedURL;
        file['link'] = presignedURL;
        file['attachmentID'] = dataMsg?.attachmentID;
        file['status'] = 'done';
    } catch {
        file['url'] = '';
        file['link'] = '';
        file['attachmentID'] = '';
        return upFailed(file, onUpload);
    }
    let copy = JSON.parse(JSON.stringify(file));
    copy['name'] = file.name;
    return { file: copy, upResult: '' };
};

export const commonUpload = ({ file, onUpload }) => {
    let fileSize = 100; // 100M
    if (file.size / 1024 / 1024 > fileSize) {
        // 当文件大于100M采用分段上传;
        return multiPartUpload(file, onUpload);
    } else {
        return uploadFile(file, onUpload);
    }
};

相关文章

网友评论

    本文标题:大文件分段上传,控制接口并发数量

    本文链接:https://www.haomeiwen.com/subject/ucoedrtx.html