美文网首页
搭建文件服务器FastDFS

搭建文件服务器FastDFS

作者: 小玉1991 | 来源:发表于2023-04-12 15:14 被阅读0次

资料包:
百度云盘链接: https://pan.baidu.com/s/1dmMJhMCkfBY_rJlpnw9o-w 密码: r5pc

1. 安装gcc、openssl、pcre、zlib等依赖包。

资料包中已经有了相关安装包
可以参考https://blog.csdn.net/weixin_57105109/article/details/126707004

#检查是否安装了gcc
gcc -v
#安装gcc
yum -y install gcc
#安装openssl
tar -zxvf openssl-3.0.5.tar.gz
cd openssl-3.0.5/
./config
make && make install

tar -zxvf pcre2-10.40.tar.gz
cd pcre2-10.40/
./config
make && make install

tar -zxvf zlib-1.2.12.tar.gz
cd zlib-1.2.12/
./config
make && make install

2. 安装libfastcommon

获取libfastcommon安装包:
wget https://github.com/happyfish100/libfastcommon/archive/V1.0.38.tar.gz
解压安装包:tar -zxvf V1.0.38.tar.gz
进入目录:cd libfastcommon-1.0.38
执行编译:./make.sh
安装:./make.sh install

3. 安装FastDFS

获取fdfs安装包
wget https://github.com/happyfish100/fastdfs/archive/V5.11.tar.gz
 
解压安装包:tar -zxvf V5.11.tar.gz
进入目录:cd fastdfs-5.11
执行编译:./make.sh
安装:./make.sh install

配置Trancker

进入/etc/fdfs目录,修改tracker.conf.sample为tracker.conf
 
编辑tracker.conf:vim tracker.conf,修改相关参数
 
base_path=/home/fastdfs/tracker
 
#tracker文件夹里的data文件夹和logs文件夹,会自动生成,否则必须提前创建好 

http.server_port=80 #http端口,保持不变,尽量与nginx相同
 
启动tracker(支持start|stop|restart):
 
/usr/bin/fdfs_trackerd  /etc/fdfs/tracker.conf  start

配置Storage

#进入/etc/fdfs目录,修改storage.conf.sample为storage.conf
cp storage.conf.sample storage.conf
 
编辑storage.conf:vi storage.conf,修改相关参数:
 
base_path=/home/fastdfs/storage
#storage文件夹里的data文件夹和logs文件夹,会自动生成,否则必须提前创建好 。
store_path0=/home/fastdfs/storage  #如果为空,则使用base_path 
#配置该storage监听的tracker的ip和port
tracker_server=10.122.149.211:22122 # 这里不能使用127.0.0.1 。22122端口也尽量不用改
 
#启动storage
/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start

4. 安装nginx和fastdfs-nginx-module模块

安装nginx

下载Nginx安装包
 
wget https://nginx.org/download/nginx-1.18.0.tar.gz
 
下载fastdfs-nginx-module安装包
 
wget https://github.com/happyfish100/fastdfs-nginx-module/archive/V1.20.tar.gz
 
解压nginx:tar -zxvf nginx-1.15.2.tar.gz
 
解压fastdfs-nginx-module:tar -xvf V1.20.tar.gz
 
进入nginx目录:cd nginx-1.15.2
 
安装依赖的库
 
apt-get update yum install libpcre3 libpcre3-dev openssl libssl-dev libperl-dev pcre pcre-devel lib zlib-devel
 
修改fastdfs-nginx-module/src/config文件,修改一下内容
ngx_module_incs="/usr/include/fastdfs /usr/include/fastcommon/"
 
CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/"
 
#配置,并加载fastdfs-nginx-module模块:
#注意这里用自己的路径,不一定是:/root/fastdfs-nginx-module-1.20/src/
./configure --prefix=/usr/local/nginx --add-module=/root/fastdfs-nginx-module-1.20/src/
 
编译安装:
make && make install

nginx常用命令
cd /usr/local/nginx/sbin
./nginx   #启动
./nginx -t #检查配置文件 
./nginx -s reload #重启

配置nginx和fdfs

1. 配置mod-fastdfs.conf,并拷贝到/etc/fdfs文件目录下
cd fastdfs-nginx-module-1.20/src/
cp mod_fastdfs.conf /etc/fdfs
 
2. 进入/etc/fdfs修改mod-fastdfs.conf:
base_path=/home/fastdfs 
tracker_server=10.122.149.211:22122 #tracker的地址
url_have_group_name=true #url是否包含group名称 
store_path0=/home/fastdfs/storage #文件存储的位置
 
3. 配置nginx,80端口server增加location 启动nginx
cd /usr/local/nginx/conf/ 
vi nginx.conf
 
location ~/M00{
    root /home/fastdfs/storage/data;
    ngx_fastdfs_module;
}
 
4. 最后需要拷贝fastdfs解压目录中的http.conf和mime.types:
cd /usr/local/src/fastdfs-5.11/conf 
cp mime.types http.conf /etc/fdfs/

5. FastDFS常用命令测试

#上传文件
1. 进入/etc/fdfs目录,有cp命令拷贝client.conf.sample,为client.conf;
 
2. 修改client.conf相关配置:
base_path=/home/fastdfs/tracker  //tracker服务器文件路径
    tracker_server=10.122.149.211:22122 //tracker服务器IP地址和端口号
    http.tracker_server_port=80 
 
3. 新建一个测试文档1.txt,内容为abc
4. 命令:
/usr/bin/fdfs_upload_file  <config_file> <local_filename>
 
5. 示例:
/usr/bin/fdfs_upload_file  /etc/fdfs/client.conf 1.txt
 
6. 查看结果,进入storage的data目录:
/home/fastdfs/storage/data/00/00/CnqV01trmeyAbAN0AAAABLh3frE677.txt
 
7. 通过wget和浏览器方式访问成功:这里访问的是配置的nginx服务器,注意nginx的端口号
wget http://10.122.149.211/group1/M00/00/00/CnqV01trmeyAbAN0AAAABLh3frE677.txt
 
#下载文件:
1. 命令:
/usr/bin/fdfs_download_file <config_file> <file_id> [local_filename]
2. 示例:
/usr/bin/fdfs_download_file /etc/fdfs/client.conf 
group1/M00/00/00/CnqV01trmeyAbAN0AAAABLh3frE677.txt a.txt
 
#删除文件:
1.命令:
/usr/bin/fdfs_delete_file <config_file> <file_id>
 
2.示例:
/usr/bin/fdfs_delete_file /etc/fdfs/client.conf
group1/M00/00/00/CnqV01trmeyAbAN0AAAABLh3frE677.txt

6.FastDFS使用

添加依赖,或者直接使用jar包

<dependency>
    <groupId>com.github.tobato</groupId>
    <artifactId>fastdfs-client</artifactId>
    <version>1.26.7</version>
</dependency>

FastDFS配置。application. yml文件

fdfs:
  # 文件下载拉取地址。nginx的ip和端口号
  web-server-url: https://img.xxx.com
  so-timeout: 3000
  connect-timeout: 1200
  thumb-image: # 缩略图
    width: 60
    height: 60
  tracker-list: 192.168.2.9:22122
  pool:
    jmx-enabled: false #禁止JMX重复注册,影响springboot JMX监控

FdfsUtil工具类

private static final List<String> IMAGE_TYPE = 
            Arrays.asList("jpg", "jpeg", "png", "bmp", "tif", "gif", "webp");
    private static final List<String> VIDEO_TYPE = Arrays.asList("mp4", "avi");
    private static final String IMG_LINUX_LOCATION = "/usr/local/data/img";

    /**
     * 前端 文件上传
     * @param fastFileStorageClient
     * @param file
     * @param imageServer
     * @param needThumbnail
     * @param needSize
     * @return
     * @throws IOException
     */
    public static UploadFile upload(FastFileStorageClient fastFileStorageClient, 
                                    MultipartFile file, String imageServer, 
                                    Byte needThumbnail, Byte needSize) 
                                    throws IOException {
        UploadFile uploadFile = new UploadFile(
                  file.getOriginalFilename(), file.getSize(), file.getContentType());

        uploadFile(fastFileStorageClient, file.getInputStream(), 
                   uploadFile, imageServer, needThumbnail, needSize);
        return uploadFile;
    }

    /**
     * 后端 文件上传
     * @param fastFileStorageClient
     * @param file
     * @param imageServer
     * @param needThumbnail
     * @param needSize
     * @return
     */
    public static UploadFile upload(FastFileStorageClient fastFileStorageClient, 
                                    File file, String imageServer, 
                                    Byte needThumbnail, Byte needSize) {
        try (InputStream is = new FileInputStream(file)) {
            UploadFile uploadFile = new UploadFile(
                       file.getName(), file.length(), null);

            uploadFile(fastFileStorageClient, is, uploadFile, 
                       imageServer, needThumbnail, needSize);

            return uploadFile;
        } catch (Exception e){
            throw new CommonException("上传文件失败", e);
        }
    }

    /**
     * 上传文件,处理图片
     * @param fastFileStorageClient
     * @param inputStream
     * @param uploadFile
     * @param imageServer
     * @param needThumbnail
     * @param needSize
     * @throws IOException
     */
    private static void uploadFile(FastFileStorageClient fastFileStorageClient, 
                                   InputStream inputStream, UploadFile uploadFile, 
                                   String imageServer, Byte needThumbnail, 
                                   Byte needSize) throws IOException {
        log.info("文件上传 originalFileName={}, fileSize={}", 
                 uploadFile.getOriginalFilename(), uploadFile.getFileSize());

        // 获取文件后缀
        String fileExtName = FilenameUtils.getExtension(
                             uploadFile.getOriginalFilename()).toLowerCase();

        // 限制图片文件上传大小
        if(isSupportType(fileExtName)){
            validateUploadImage(uploadFile, fileExtName);
        }

        // 缓存文件流
        byte[] fileStream = IOUtils.toByteArray(inputStream);

        // 上传原图或文件
        uploadOrigFile(fastFileStorageClient, uploadFile, 
                      imageServer, fileStream, fileExtName);

        // 上传图片缩略图
        if(isSupportType(fileExtName)){
            uploadImageThumbnails(fastFileStorageClient, uploadFile, 
                  imageServer, fileStream, fileExtName, needThumbnail, needSize);
        }

        log.info("上传文件地址={}", uploadFile.getFullFilename());
    }

    /**
     * 限制图片文件上传大小
     * @param uploadFile
     * @param fileExtName
     */
    private static void validateUploadImage(UploadFile uploadFile, 
                                            String fileExtName){
        // gif文件,大小校验,超过4兆,不上传
        if("gif".equals(fileExtName)){
            if(uploadFile.getFileSize() > 4 * 1024 * 1024){
                throw new CommonException("GIF动图大小不能超过4M");
            }
        } else if(VIDEO_TYPE.contains(fileExtName)){
            if(uploadFile.getFileSize() > 40 * 1024 * 1024){
                throw new CommonException("视频大小不能超过40M");
            }
        } else{
            // 普通图片文件不能超过10M
            if(uploadFile.getFileSize() > 10 * 1024 * 1024){
                throw new CommonException("上传单张图片不能超过10M");
            }
        }
    }

    /**
     * 上传原文件
     * @param fastFileStorageClient
     * @param uploadFile
     * @param imageServer
     * @param fileStream
     * @param fileExtName
     */
    private static void uploadOrigFile(FastFileStorageClient fastFileStorageClient, 
                                       UploadFile uploadFile, String imageServer,
                                       byte[] fileStream, String fileExtName){
        try (ByteArrayInputStream is = new ByteArrayInputStream(fileStream)) {
            StorePath originalStorePath = fastFileStorageClient.uploadFile(
                      is, uploadFile.getFileSize(), fileExtName, null);
            uploadFile.setFullFilename(imageServer.concat("/").concat(
                       originalStorePath.getFullPath()));
        } catch (IOException e) {
            log.error("upload Image error", e.getCause());
            throw new FdfsUploadImageException("upload Image error", e.getCause());
        }
    }

    /**
     * 上传图片缩略图
     * @param fastFileStorageClient
     * @param uploadFile
     * @param imageServer
     * @param fileStream
     * @param fileExtName
     * @param needThumbnail
     * @param needSize
     * @throws IOException
     */
    private static void uploadImageThumbnails(
                                        FastFileStorageClient fastFileStorageClient, 
                                        UploadFile uploadFile, String imageServer,
                                        byte[] fileStream, String fileExtName, 
                                        Byte needThumbnail, Byte needSize) 
                                        throws IOException {
        // 默认图片缩略图取原图
        uploadFile.setThumbnail(uploadFile.getFullFilename());
        // 缩略图也需要计算图片尺寸大小
        if(Constant.YES.equals(needSize) || Constant.YES.equals(needThumbnail)){
            // 获取图片尺寸
            BufferedImage bufferedImage = ImageIO.read(new ByteArrayInputStream(
                                          fileStream));

            if(bufferedImage != null){
                uploadFile.setWidth(bufferedImage.getWidth());
                uploadFile.setHeight(bufferedImage.getHeight());

                // 压缩处理
                if(Constant.YES.equals(needThumbnail)){
                    InputStream is = null;
                    try {
                        ByteArrayOutputStream out = new ByteArrayOutputStream();
                        float outputQuality = getOutputQuality(
                                               uploadFile.getFileSize());
                        if(!"jpg".equals(fileExtName)){
                            // 先转成jpg
                            String newPicPath = IMG_LINUX_LOCATION + File.separator 
                                             + UUID.randomUUID().toString() + ".jpg";
                            Thumbnails.of(new ByteArrayInputStream(fileStream))
                                                  .scale(1f).toFile(newPicPath);
                            // 原比例不变,大小压缩
                            Thumbnails.of(newPicPath).scale(1f)
                                  .outputQuality(outputQuality).toOutputStream(out);
                            // 删除jpg图片
                            new File(newPicPath).delete();
                        } else {
                            Thumbnails.of(new ByteArrayInputStream(fileStream))
                                        .scale(1f).outputQuality(outputQuality)
                                        .toOutputStream(out);
                        }
                        is = new ByteArrayInputStream(out.toByteArray());

                        // 上传缩略图
                        StorePath thumbnailStorePath = fastFileStorageClient
                          .uploadFile(is, is.available(), fileExtName, null);
                        uploadFile.setThumbnail(imageServer.concat("/")
                            .concat(thumbnailStorePath.getFullPath()));

                        log.info("上传缩略图地址={}", uploadFile.getThumbnail());
                    } catch (IOException e) {
                        log.error("upload ThumbImage error", e.getCause());
                        throw new FdfsUploadImageException(
                                  "upload ThumbImage error",e.getCause());
                    } finally {
                        IOUtils.closeQuietly(is);
                    }
                }
            }
        }
    }

    /**
     * 设置压缩比例
     * @param fileSize
     * @return
     */
    private static float getOutputQuality(long fileSize){
        if (fileSize < 64 * 1024) {
            return 1f;
        } else if (fileSize < 256 * 1024) {
            return 0.4f;
        } else if (fileSize < 2 * 1024 * 1024) {
            return 0.2f;
        }

        return 0.1f;
    }

    /**
     * 判断文件格式是否支持
     * @param fileExtName
     * @return
     */
    private static boolean isSupportType(String fileExtName) {
        return IMAGE_TYPE.contains(fileExtName) || VIDEO_TYPE.contains(fileExtName);
    }

    /**
     * 删除文件
     * @param fileUrl 文件访问地址
     * @param fastFileStorageClient
     */
    public static void deleteFile(FastFileStorageClient fastFileStorageClient, 
                                                               String fileUrl) {
        if (StringUtils.isEmpty(fileUrl)) {
            return;
        }
        try {
            StorePath storePath = StorePath.parseFromUrl(fileUrl);
            fastFileStorageClient.deleteFile(
                                storePath.getGroup(), storePath.getPath());
        } catch (FdfsUnsupportStorePathException e) {
            log.warn("异常信息:", e);
        }
    }

}

UploadFile

@Data
public class UploadFile{
  private String originalFilename;
  private long fileSize;
  private String contentType;
  private String fullFilename;
  public UploadFile(){}
 public UploadFile(String originalFilename,long fileSize,String contentType){
    this.originalFilename=originalFilename;
    this.fileSize=fileSize;
    this.contentType=contentType;
  }
}

FdfsController

@Slf4j
@RestController
@RequestMapping("/fdfs")
@Api(tags = "文件上传")
public class FdfsController {

    @Value("${fdfs.web-server-url}")
    private String imageServer;

    @Autowired
    private FastFileStorageClient fastFileStorageClient;

    @PostMapping(value = "/upload", consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
                                    produces = MediaType.APPLICATION_JSON_VALUE)
    @ApiOperation(value = "文件上传", httpMethod = "POST")
    public String upload(@RequestParam("file") MultipartFile file,
     @RequestParam(value = "needSize", defaultValue = "0") Byte needSize,
     @RequestParam(value = "needThumbnail", defaultValue = "0") Byte needThumbnail) {
        try {
            long start = System.currentTimeMillis();

            // 获取文件后缀
            String fileExtName = FilenameUtils.getExtension(
                                  file.getOriginalFilename()).toUpperCase();

            UploadFile uploadFile = FdfsUtil.upload(
                 fastFileStorageClient, file, imageServer, needThumbnail, needSize);

            long end = System.currentTimeMillis();

            log.info("文件上传耗时:{} 耗秒", (end - start));

             return JSON.toJSONString(uploadFile);
        } catch (Exception e) {
            log.error("上传文件异常:", e);
        }

        log.warn(String.format("文件 %s 上传失败", file.getOriginalFilename()));

        return "上传文件异常";
    }

}

==========服务端上传文件================
  @Value("${fdfs.web-server-url}")
  private String imageServer;

  @Autowired
  private FastFileStorageClient fastFileStorageClient;
  File file = new File("xxxx");
  UploadFile uploadFile = FdfsUtil.upload(
                    fastFileStorageClient, file, imageServer, null, null);
  String fileUrl = uploadFile.getFullFilename();
  file.deleteOnExit();
  return fileUrl;

相关文章

网友评论

      本文标题:搭建文件服务器FastDFS

      本文链接:https://www.haomeiwen.com/subject/mkcoddtx.html