使用Amazon S3接口云存储

作者: 会点代码的大叔 | 来源:发表于2017-10-19 16:04 被阅读1444次

AWS S3云存储使用CloudFront优化资源访问速度
使用Amazon S3接口云存储
AWS应用场景[2021-03-21]
七牛云WordPress插件-节省带宽，加速网站
minio简介
minio分布式存储环境搭建
Minio的安装&springBoot整合Minio进行上传下载
Python上传文件到MinIO服务
aws服务从入门到精通|s3文件的操作
对象存储 S3 C SDK 文档

本文主要介绍了如果使用Java操作Amazon S3接口

Amazon S3介绍

Amazon S3，也叫AWS S3全称是：Amazon Simple Storage Service

官网介绍

Amazon S3 是专为从任意位置存储和检索任意数量的数据而构建的对象存储，这些数据包括来自网站和移动应用程序、公司应用程序的数据以及来自 IoT 传感器或设备的数据。

简单一句话，AWS S3是一个可以存放文件的地方，并且很可靠，我们可以通过程序对文件进行上传、下载、删除、遍历等操作。

我们可以把AWS S3看成是FTP：

文件存放

FTP:/xxx/yyy/zzz.txt
AWS:/xxx/yyy/zzz.txt，在AWS S3中，这个叫做key

权限控制

FTP:用户名、密码
AWS:accessKey和secretKey，这个一般是基础设施的伙伴搭建好环境之后告知的，可以看做用户名密码

根目录

FTP:使用用户名登录进去之后的目录
AWS:bucket

目录结构

FTP:/xxx/yyy
AWS:/xxx/yyy

访问地址

FTP:ftp://IP：端口
AWS:http://域名：端口，也叫ENDPOINT，实际上就是一个连接,如："http://test.ojstorage.ecm.group.tkuat.com"

AWS SDK for Java

官方的SDK for Java：https://aws.amazon.com/cn/sdk-for-java/

Demo：单位连github连了几天，又上不去了...贴几个单元测试用例

POM.XML需要引入：

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-s3</artifactId>
</dependency>

常量定义

package com.taikang.aws.aws_demo;

public class AppTest{
    
    //测试环境
    final String ENDPOINT = "http://IP:port";
    final String accessKey = "xxxxx";
    final String secretKey = "yyyyy";
    final String bucket_name = "bucket";
}

遍历文件

package com.taikang.aws.aws_demo;

import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.ListObjectsV2Request;
import com.amazonaws.services.s3.model.ListObjectsV2Result;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectSummary;

/**
 * 遍历文件
 * 
 * 两周方法遍历bucketName下面的所有文件
 * @author liuwei117
 *
 */
public class AWSListTest extends AppTest{
    private static Logger logger = LoggerFactory.getLogger(AWSListTest.class);
    
    @Test
    public void listObject(){
        AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
        
        ClientConfiguration clientConfig = new ClientConfiguration();
        
        try {
            clientConfig.setProtocol(Protocol.HTTP);
            
            AmazonS3Client conn = new AmazonS3Client(credentials,clientConfig);
            conn.setEndpoint(ENDPOINT);
            
            ObjectListing objects = conn.listObjects(bucket_name);
            do {
                 for (S3ObjectSummary objectSummary : objects.getObjectSummaries()){
                     logger.info("\t"+objectSummary.getKey() + "\t" + objectSummary.getSize() 
                             + "\t" +objectSummary.getOwner() + "\t" +objectSummary.getLastModified().toString());
                     
                 }
            } while (objects.isTruncated());
            
        } catch (Exception e) {
            e.printStackTrace();
        } 
    }
    
    @Test
    public void listObjectWithClientBuilder(){
        AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
        
        ClientConfiguration clientConfig = new ClientConfiguration();
        
        try {
            clientConfig.setProtocol(Protocol.HTTP);
            
            AmazonS3ClientBuilder builder=  AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials));
            EndpointConfiguration endpointConfiguration = new EndpointConfiguration(ENDPOINT, Regions.AP_SOUTHEAST_1.getName());
            builder.setEndpointConfiguration(endpointConfiguration);
            
                    AmazonS3 conn = builder.build();
            
            ListObjectsV2Request req = new ListObjectsV2Request().withBucketName(bucket_name);
            
            ListObjectsV2Result result;
            
do {
result = conn.listObjectsV2(req);

for (S3ObjectSummary objectSummary : result.getObjectSummaries()) {
    logger.info("\t"+objectSummary.getKey() + "\t" + objectSummary.getSize() 
                             + "\t" +objectSummary.getOwner() + "\t" +objectSummary.getLastModified().toString());
}
} while(result.isTruncated());
            
        } catch (Exception e) {
            e.printStackTrace();
        } 
    }
    
    /**
     * 根据bucket和key进行文件的查找
     */
    @Test
    public void getObjectByKey(){
        AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
        
        ClientConfiguration clientConfig = new ClientConfiguration();
        
        try {
            clientConfig.setProtocol(Protocol.HTTP);
            
            AmazonS3ClientBuilder builder=  AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials));
            EndpointConfiguration endpointConfiguration = new EndpointConfiguration(ENDPOINT, Regions.AP_SOUTHEAST_1.getName());
            builder.setEndpointConfiguration(endpointConfiguration);
            
    AmazonS3 conn = builder.build();
            
    S3Object object = conn.getObject(new GetObjectRequest(bucket_name, "e0500720170818105659.wav"));
    
    logger.info(object.getKey());
    /*if(object != null ){
        ObjectMetadata metadata = object.getObjectMetadata();
        
        logger.info(metadata.getUserMetaDataOf("managecom"));
        logger.info(metadata.getUserMetaDataOf("salecom"));
        logger.info(metadata.getUserMetaDataOf("contno"));
        logger.info(metadata.getUserMetaDataOf("recordid"));
    }*/
            
        } catch (Exception e) {
            e.printStackTrace();
        } 
    }
}

上传文件

package com.taikang.aws.aws_demo;


import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.net.URL;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;

/**
 * 上传文件
 * 
 * @author liuwei117
 *
 */
public class AWSUploadTest extends AppTest{
    private static Logger logger = LoggerFactory.getLogger(AWSUploadTest.class);
    
    //本地的一个文件
    private File file = new File("E:" + File.separator + "EliteClient2017060710.log");  
    
    private String managecom = "1" ;
    private String salecom = "1TT" ;
    private String contno = "12345678" ;
    private String recordid = "11111111" ;
    
    //这里做一下目录分级：上传的文件放在1/1TT/下面
    private String key = managecom + "/" + salecom + "/" + file.getName();
    
    @Test
    public void UploadObject(){
        AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
        
        ClientConfiguration clientConfig = new ClientConfiguration();
        
        try {
            clientConfig.setProtocol(Protocol.HTTP);
            
            AmazonS3Client conn = new AmazonS3Client(credentials,clientConfig);
            conn.setEndpoint(ENDPOINT);
            
            ObjectMetadata metadata = new ObjectMetadata();
            metadata.setContentLength(file.length());

            metadata.addUserMetadata("managecom", managecom);  //可以设置一些属性，如公司码 = 1
            metadata.addUserMetadata("salecom", salecom);
            metadata.addUserMetadata("contno", contno);
            metadata.addUserMetadata("recordid", recordid);
            
            FileInputStream fi = new FileInputStream(file);  
            
            //这里用managecom/salecom/filename作为文件的key,相当bucketName是一个根目录，创建了一级目录managecom和二级目录salecom，再放进去了一个文件
            conn.putObject(bucket_name , key , fi , metadata); 
            //conn.putObject(new PutObjectRequest(bucket_name, key, file));
            
            //将上传了的文件生成一个url连接，可以直接通过url连接访问文件
            GeneratePresignedUrlRequest urlRequest = new GeneratePresignedUrlRequest(bucket_name, key);
            URL url = conn.generatePresignedUrl(urlRequest);
            
            logger.info("[" + key + "]'s url is " + url);
        } catch (Exception e) {
            e.printStackTrace();
        } 
    }
    
    @Test
    public void listObjectWithClientBuilder(){
        AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
        
        ClientConfiguration clientConfig = new ClientConfiguration();
        
        try {
            clientConfig.setProtocol(Protocol.HTTP);
            
            AmazonS3ClientBuilder builder=  AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials));
            EndpointConfiguration endpointConfiguration = new EndpointConfiguration(ENDPOINT, Regions.AP_SOUTHEAST_1.getName());
            builder.setEndpointConfiguration(endpointConfiguration);
            
                    AmazonS3 conn = builder.build();
    
                    ObjectMetadata metadata = new ObjectMetadata();
            metadata.setContentLength(file.length());

            metadata.addUserMetadata("managecom", managecom);  //可以设置一些属性，如公司码 = 1
            metadata.addUserMetadata("salecom", salecom);
            metadata.addUserMetadata("contno", contno);
            metadata.addUserMetadata("recordid", recordid);
            
            FileInputStream fi = new FileInputStream(file);  
            
    conn.putObject(new PutObjectRequest(bucket_name, key, fi , metadata));
            //conn.putObject(new PutObjectRequest(bucket_name, key, file));
            
        } catch (Exception e) {
            e.printStackTrace();
        } 
    }
}

删除文件

package com.taikang.aws.aws_demo;

import java.io.File;

import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;

/**
 * 遍历文件
 * @author liuwei117
 *
 */
public class AWSDeleteTest extends AppTest{
    private static Logger logger = LoggerFactory.getLogger(AWSDeleteTest.class);
    
    //本地的一个文件
    private File file = new File("E:" + File.separator + "EliteClient2017060710.log");  
    
    private String managecom = "1" ;
    private String salecom = "1TT" ;
    private String contno = "12345678" ;
    private String recordid = "11111111" ;
    
    //这里做一下目录分级：上传的文件放在1/1TT/下面
    //private String key = managecom + "/" + salecom + "/" + file.getName();
    private String key = "1/1TT/11111111";
    @Test
    public void deleteObject(){
        AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
        
        ClientConfiguration clientConfig = new ClientConfiguration();
        
        try {
            clientConfig.setProtocol(Protocol.HTTP);
            
            AmazonS3Client conn = new AmazonS3Client(credentials,clientConfig);
            conn.setEndpoint(ENDPOINT);
            
            conn.deleteObject(bucket_name , key);
        } catch (Exception e) {
            e.printStackTrace();
        } 
    }
    
    @Test
    public void deleteObjectWithClientBuilder(){
        AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
        
        ClientConfiguration clientConfig = new ClientConfiguration();
        
        try {
            clientConfig.setProtocol(Protocol.HTTP);
            
            AmazonS3ClientBuilder builder=  AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials));
            EndpointConfiguration endpointConfiguration = new EndpointConfiguration(ENDPOINT, Regions.AP_SOUTHEAST_1.getName());
            builder.setEndpointConfiguration(endpointConfiguration);
            
    AmazonS3 conn = builder.build();
    
    conn.deleteObject(bucket_name , key);
        } catch (Exception e) {
            e.printStackTrace();
        } 
    }
}

遇到的坑

必须用域名访问，如果AWS S3的地址没有绑定域名，也需要在host文件中进行配置。
继续踩... ...

AWS S3云存储使用CloudFront优化资源访问速度
S3云存储 Amazon Simple Storage Service (Amazon S3) 是一种对象存储服务...
使用Amazon S3接口云存储
本文主要介绍了如果使用Java操作Amazon S3接口 Amazon S3介绍 Amazon S3，也叫AWS ...
AWS应用场景[2021-03-21]
场景公司使用Amazon S3在特定的虚拟私有云(VPC)中存储只能Amazon EC2访问的文档。该公司担心，...
七牛云WordPress插件-节省带宽，加速网站
七牛云存储是由七牛提供的在线存储服务，和传统的云存储服务（如 Amazon S3）不同，七牛云存储还增加了上传下载...
minio简介
简介 MinIO是在Apache License v2.0下发布的对象存储服务器。它与Amazon S3云存储服务...
minio分布式存储环境搭建
1.概述 Minio是Apache License v2.0下发布的对象存储服务器。它与Amazon S3云存储服...
Minio的安装&springBoot整合Minio进行上传下载
Minio简介 Minio是Apache License v2.0下发布的对象存储服务器。它与Amazon S3云...
Python上传文件到MinIO服务
MinIO Python SDK 是简单存储服务（又名S3）客户端，用于对任何与Amazon S3 兼具的对存储服...
aws服务从入门到精通|s3文件的操作
s3 简介(什么是s3) Amazon Simple Storage Service 是互联网存储解决方案。该服务...
对象存储 S3 C SDK 文档
公有云对象存储 S3 C SDK 文档一 SDK说明本文主要介绍公有云对象存储 C语言 s3-API接口，包...

使用Amazon S3接口云存储

Amazon S3介绍

AWS SDK for Java

常量定义

遍历文件

上传文件

删除文件

相关文章

AWS S3云存储使用CloudFront优化资源访问速度

使用Amazon S3接口云存储

AWS应用场景[2021-03-21]

七牛云WordPress插件-节省带宽，加速网站

minio简介

minio分布式存储环境搭建

Minio的安装&springBoot整合Minio进行上传下载

Python上传文件到MinIO服务

aws服务从入门到精通|s3文件的操作

对象存储 S3 C SDK 文档

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读