一. 业务背景
在业务系统使用过程中, 发现:
- B端系统: 采购系统(PMS), 仓库管理系统(WMS), 订单管理系统(OMS)等部分图片不展示, 不能根据图片进行信息的核对, 影响转化率.
- C端部分图片加载缓慢, 在慢网络环境下无法加载图片, 影响转化率,GMV
二. 跟踪分析
问题:
- B端系统使用了第三方图床, 部分图片链接已经失效, 部分图床进行了防盗链
- C端使用原图展示, 原图过大, 加载缓慢, 部分不能显示
- 公司两个地方存储了图片(C端, 爬虫), 造成资源浪费, 但是中间环节仍然使用原始链接
和各业务方沟通后, 当前图片使用情况如下:
image.png
三. 解决方案
方案:
- 收敛图片处理和存储, 图片保存在AWS S3
- 对CDN和S3增加缓存设置, 提升图片加载速度
- 支持不同业务对图片尺寸需求;
四. 实现细节
图片处理流程收敛到商品运营中心上货环节
备注: 商品图片处理包含尺寸:原图.webp, 原图.jpg, 350x350.webp, 350x350.jpg, 200x200.webp, 200x200.jpg, 50x50.webp, 50x50.jpg
4.1 图片处理方案
收敛过程需要考虑网络问题: 商品运营中心系统在aliyun杭州, 但是图片处理在AWS美西;
有两种处理方案:
方案一: 图片处理和上传在AWS美西, 商品运营中心上货后调用图片处理
问题: 图片处理只能异步进行, 跨云异步处理图片, 图片处理完跨云回商品运营中心务接口, 进行url更新并发送商品更新消息.
- 调用链路太长, 两次跨云, 跨云的网络风险高(当时还没有专线)
- 图片处理必须跨云下载图片一次
方案二: 图片处理和上传在aliyun杭州, 跨云上传一次图片到AWS S3
- 业务调用图片处理, 图片处理后回调都在内网进行, 降低跨云通讯的复杂性
- 图片处理跨云上传一次图片到S3
对比后选择方案二,在aliyun杭州跨云上传一次图片到AWS S3
4.2 商品管理后台图片处理流程
image.png
优化点:
- 添加失败重试机制, 网络因素较多, 须有重试, 且不应该有无效图片的商品
- 引入AWS S3 lambda, 原图只上传一次到AWS; 图片格式转换, 等比缩放通过lambda进行, 避免了额外的网络开销
- 图片下载添加代理机制 (国内下载不到某些国外的图床图片)
4.3 图片处理核心代码逻辑
java图片格式转换和等比缩放使用jdk自带图片处理接口ImageIO实现
备注:
图片处理流程为: 下载原图数据流 -- 原图格式转换为webp格式 -- 上传到AWS s3 -- s3 lambda生成不同尺寸的webp/jpg格式图片
ImageUtils
- 进行单个图片格式转换, 等比缩放
import com.clubfactory.center.common.util.BaseUtil;
import com.clubfactory.image.core.constants.ImageConstants;
import org.apache.commons.lang3.StringUtils;
import javax.imageio.IIOImage;
import javax.imageio.ImageIO;
import javax.imageio.ImageWriteParam;
import javax.imageio.ImageWriter;
import javax.imageio.stream.MemoryCacheImageOutputStream;
import java.awt.*;
import java.awt.image.BufferedImage;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.Iterator;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class ImageUtils {
/**
* 图片尺寸允许的最大文件大小
*/
private static final Pattern CLOUD_FRONT_CALICDN = Pattern.compile("^https*://d3kpm7yklociqe.cloudfront.net/calicdn(\\d+)/(.*)$");
private static final Pattern CLOUD_FRONT_ALICDN = Pattern.compile("^https*://d3kpm7yklociqe.cloudfront.net/alicdn(\\d+)/(.*)$");
private static final Pattern CLOUD_FRONT_ALIIMG = Pattern.compile("^https*://d3kpm7yklociqe.cloudfront.net/aliimg(\\d+)/(.*)$");
private static final Pattern CLUB_CALICDN = Pattern.compile("^https*://cdn.fromfactory.club/calicdn(\\d+)/(.*)$");
private static final Pattern CLUB_ALICDN = Pattern.compile("^https*://cdn.fromfactory.club/alicdn(\\d+)/(.*)");
private static final Pattern CLUB_ALIIMG = Pattern.compile("^^https*://cdn.fromfactory.club/aliimg(\\d+)/(.*)$");
/**
* <b>function:</b> 通过目标对象的大小和标准（指定）大小计算出图片缩小的比例
*
* @param targetWidth 目标的宽度
* @param targetHeight 目标的高度
* @param srcWidth 标准（指定）宽度
* @param srcHeight 标准（指定）高度
* @return 最小的合适比例
* @author hoojo
* @createDate 2012-2-6 下午04:41:48
*/
private static double getScaling(double srcWidth, double srcHeight, double targetWidth, double targetHeight) {
double widthScaling = 0d;
double heightScaling = 0d;
if (srcWidth > targetWidth) {
widthScaling = targetWidth / (srcWidth * 1.00d);
} else {
widthScaling = 1d;
}
if (srcHeight > targetHeight) {
heightScaling = targetHeight / (srcHeight * 1.00d);
} else {
heightScaling = 1d;
}
return Math.min(widthScaling, heightScaling);
}
/**
* <b>function:</b> 将Image的宽度、高度缩放到指定width、height，
*
* @return 图片保存路径、名称
* @throws IOException
* @author hoojo
* @createDate 2012-2-6 下午04:54:35
*/
public static byte[] resizeAndConvert(BufferedImage srcImage, int targetWidth, int targetHeight, Double minQuality, String format, int maxSize, String hashValue) throws IOException {
double ratio = getScaling(srcImage.getWidth(), srcImage.getHeight(), targetWidth, targetHeight);
int newWeigh = (int) (srcImage.getWidth() * ratio), newHeight = (int) (srcImage.getHeight() * ratio);
BufferedImage tag = new BufferedImage(newWeigh, newHeight, BufferedImage.TYPE_INT_RGB);
Image tmp = srcImage.getScaledInstance(newWeigh, newWeigh, Image.SCALE_SMOOTH);
tag.getGraphics().drawImage(tmp, 0, 0, newWeigh, newHeight, null);
Iterator<ImageWriter> iterator = ImageIO.getImageWritersByFormatName(format);
ImageWriter imageWriter = iterator.next();
ImageWriteParam imageWriteParam = imageWriter.getDefaultWriteParam();
imageWriteParam.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
if (format.equals(ImageConstants.WEBP)) {
imageWriteParam.setCompressionType("Lossless");
}
ByteArrayOutputStream os;
MemoryCacheImageOutputStream memoryOs;
float nextQuality = 0.95f;
do {
os = new ByteArrayOutputStream();
imageWriteParam.setCompressionQuality(nextQuality);
imageWriter.setOutput(memoryOs = new MemoryCacheImageOutputStream(os));
IIOImage iio_image = new IIOImage(tag, null, null);
imageWriter.write(null, iio_image, imageWriteParam);
memoryOs.flush();
nextQuality -= 0.05;
} while (nextQuality >= minQuality && os.size() > maxSize);
imageWriter.dispose();
return os.toByteArray();
}
public static byte[] convert(BufferedImage srcImage, String format, String hashValue) throws IOException {
BufferedImage tag = new BufferedImage(srcImage.getWidth(), srcImage.getHeight(), BufferedImage.TYPE_INT_RGB);
Image tmp = srcImage.getScaledInstance(srcImage.getWidth(), srcImage.getHeight(), Image.SCALE_SMOOTH);
tag.getGraphics().drawImage(tmp, 0, 0, srcImage.getWidth(), srcImage.getHeight(), null);
Iterator<ImageWriter> iterator = ImageIO.getImageWritersByFormatName(format);
ImageWriter imageWriter = iterator.next();
ImageWriteParam imageWriteParam = imageWriter.getDefaultWriteParam();
imageWriteParam.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
if (format.equals(ImageConstants.WEBP)) {
imageWriteParam.setCompressionType("Lossless");
}
imageWriteParam.setCompressionQuality(1);
IIOImage iio_image = new IIOImage(tag, null, null);
ByteArrayOutputStream os = new ByteArrayOutputStream();
MemoryCacheImageOutputStream memoryOs = new MemoryCacheImageOutputStream(os);
imageWriter.setOutput(memoryOs);
imageWriter.write(null, iio_image, imageWriteParam);
memoryOs.flush();
return os.toByteArray();
}
public static final String getName(String hashValue, Integer size, String fileFormat) {
if (BaseUtil.isPositive(size)) {
return String.format("%s_%dx%d.%s", hashValue, size, size, fileFormat);
} else {
return String.format("%s.%s", hashValue, fileFormat);
}
}
public static final String getKey(String hash, Integer size, String fileFormat) {
return String.format("%s/%s/%s", hash.substring(0, 2), hash.substring(hash.length() - 2), getName(hash, size, fileFormat));
}
public static String getValidateUrl(String url) {
if (BaseUtil.isEmpty(url)) {
return url;
}
url = url.replace("\n", "").replace("\r", "");
url = url.trim();
url = url.replaceAll("^https:/*", "https://").replaceAll("^http:/*", "http://");
if (!url.startsWith("http")) {
url = "http:" + url;
}
//处理尺寸
if (url.contains("contestimg.wish")) {
if (url.contains("200x200")) {
url = StringUtils.replace(url, "_200x200", "");
} else if (url.contains("50x50")) {
url = StringUtils.replace(url, "_50x50", "");
} else if (url.contains("350x350")) {
url = StringUtils.replace(url, "_350x350", "");
}
} else if (url.contains("alicdn.com")) {
if (url.contains("200x200")) {
url = StringUtils.replace(url, "_200x200.jpg", "");
} else if (url.contains("50x50")) {
url = StringUtils.replace(url, "_50x50.jpg", "");
} else if (url.contains("350x350")) {
url = StringUtils.replace(url, "_350x350.jpg", "");
}
}
// 处理其他
Matcher result;
if (url.contains("d3kpm7yklociqe.cloudfront.net")) {
if (url.contains("calicdn")) {
result = CLOUD_FRONT_CALICDN.matcher(url);
if (result.matches()) {
return String.format("http://cbu%s.alicdn.com/%s", result.group(1), result.group(2));
} else {
return url;
}
} else if (url.contains("alicdn")) {
result = CLOUD_FRONT_ALICDN.matcher(url);
if (result.matches()) {
return String.format("http://g%s.a.alicdn.com/%s", result.group(1), result.group(2));
} else {
return url;
}
} else if (url.contains("aliimg")) {
result = CLOUD_FRONT_ALIIMG.matcher(url);
if (result.matches()) {
return String.format("http://i%s.i.aliimg.com/%s", result.group(1), result.group(2));
} else {
return url;
}
} else if (url.contains("ebay")) {
url = StringUtils.replace(url, "d3kpm7yklociqe.cloudfront.net/ebay", "i.ebayimg.com");
} else if (url.contains("wish")) {
url = StringUtils.replace(url, "d3kpm7yklociqe.cloudfront.net/wish", "contestimg.wish.com");
} else {
return url;
}
} else if (url.contains("cdn.fromfactory.club")) {
if (url.contains("calicdn")) {
result = CLUB_CALICDN.matcher(url);
if (result.matches()) {
return String.format("http://cbu%s.alicdn.com/%s", result.group(1), result.group(2));
} else {
return url;
}
} else if (url.contains("alicdn")) {
result = CLUB_ALICDN.matcher(url);
if (result.matches()) {
return String.format("http://g%s.a.alicdn.com/%s", result.group(1), result.group(2));
} else {
return url;
}
} else if (url.contains("aliimg")) {
result = CLUB_ALIIMG.matcher(url);
if (result.matches()) {
return String.format("http://i%s.i.aliimg.com/%s", result.group(1), result.group(2));
} else {
return url;
}
} else if (url.contains("ebay")) {
url = StringUtils.replace(url, "cdn.fromfactory.club/ebay", "i.ebayimg.com");
} else if (url.contains("wish")) {
url = StringUtils.replace(url, "cdn.fromfactory.club/wish", "contestimg.wish.com");
} else {
return url;
}
} else if (url.contains("www.dropbox.com")) {
url = StringUtils.replace(url, "www.dropbox.com", "dl.dropboxusercontent.com");
} else if (url.contains("w=1000&h=1000&m=0")) {
url = url.split("&w")[0];
return url;
} else {
return url;
}
return url;
}
}
ImageBiz
- 图片下载-格式转换-上传AWS S3
- 校验图片链接是否有效
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.clubfactory.center.common.thread.CommonThreadFactory;
import com.clubfactory.center.common.util.*;
import com.clubfactory.image.core.dao.S3ImageMapMapper;
import com.clubfactory.image.core.dataobject.S3ImageMapDO;
import com.clubfactory.image.core.proxy.AwsS3Proxy;
import com.clubfactory.image.core.proxy.SshProxyDownloadProxy;
import com.clubfactory.image.core.util.ImageUtils;
import com.clubfactory.image.core.util.MyURIEncoder;
import com.google.common.base.Stopwatch;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
import com.jcraft.jsch.JSchException;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.asynchttpclient.AsyncHttpClient;
import org.asynchttpclient.Response;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import org.springframework.stereotype.Component;
import javax.imageio.ImageIO;
import java.awt.image.BufferedImage;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.Collectors;
import static com.clubfactory.image.core.constants.ImageConstants.*;
/**
* description:
*
* @author wyh
* @date 2018/11/27 14:38
*/
@Slf4j
@Component
public class ImageBiz {
@Autowired
private S3ImageMapMapper s3ImageMapMapper;
@Autowired
private AwsS3Proxy awsS3Proxy;
@Value("${spring.profiles.active}")
private String env;
private AtomicLong count = new AtomicLong(0);
private ThreadPoolTaskExecutor dealImageTaskExecutor;
private ThreadPoolTaskExecutor checkImagetaskExecutor;
/**
* 连接超时时间 10s
*/
public static final Integer CONNECT_TIMEOUT = 10;
/**
* 整个http过程最大的允许时间 300s 5min
*/
public static final Integer MAX_TIME = 300;
{
//处理图片线程池
dealImageTaskExecutor = new ThreadPoolTaskExecutor();
dealImageTaskExecutor.setCorePoolSize(30);
dealImageTaskExecutor.setMaxPoolSize(30);
dealImageTaskExecutor.setQueueCapacity(30);
dealImageTaskExecutor.setKeepAliveSeconds(120);
dealImageTaskExecutor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
dealImageTaskExecutor.setThreadFactory(new CommonThreadFactory());
dealImageTaskExecutor.initialize();
dealImageTaskExecutor.setAllowCoreThreadTimeOut(true);
dealImageTaskExecutor.setDaemon(true);
dealImageTaskExecutor.setWaitForTasksToCompleteOnShutdown(true);
dealImageTaskExecutor.setAwaitTerminationSeconds(300);
//校验图片是否有效线程池
checkImagetaskExecutor = new ThreadPoolTaskExecutor();
checkImagetaskExecutor.setCorePoolSize(100);
checkImagetaskExecutor.setMaxPoolSize(100);
checkImagetaskExecutor.setQueueCapacity(100);
checkImagetaskExecutor.setKeepAliveSeconds(120);
checkImagetaskExecutor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
checkImagetaskExecutor.setThreadFactory(new CommonThreadFactory());
checkImagetaskExecutor.initialize();
checkImagetaskExecutor.setAllowCoreThreadTimeOut(true);
checkImagetaskExecutor.setDaemon(true);
checkImagetaskExecutor.setWaitForTasksToCompleteOnShutdown(true);
checkImagetaskExecutor.setAwaitTerminationSeconds(300);
}
public Map<String, String> updateImage(List<String> urlList) {
return updateImage(urlList, true);
}
/**
* description: 图片格式转换为webp,并上传到aws s3
*
* @param
* @return
* @author wyh
* @date 2018/11/29 13:41
*/
public Map<String, String> updateImage(List<String> urlList, boolean checkCache) {
if (BaseUtil.isEmpty(urlList)) {
return Maps.newHashMapWithExpectedSize(0);
}
//过滤掉不需要往下处理的url
urlList = urlList.stream().filter(url -> BaseUtil.isNotEmpty(url) && !url.startsWith("https://s3.amazonaws.com/fromfactory.club.image"))
.collect(Collectors.toList());
HashSet<String> existsUrlSet = Sets.newHashSet();
if (checkCache) {
List<S3ImageMapDO> param = urlList.stream().map(url -> {
S3ImageMapDO entity = new S3ImageMapDO();
String hash = MD5Util.hash(url);
entity.setOriginUrl(url);
entity.setS3Key(hash);
entity.setPrefix(hash.substring(hash.length() - 2));
return entity;
}).collect(Collectors.toList());
//查询已经上传过的图片
if (BaseUtil.isEmpty(param)) {
return Maps.newHashMapWithExpectedSize(0);
}
existsUrlSet = Sets.newHashSet(s3ImageMapMapper.getByOriginUrl(param));
}
ConcurrentHashMap<String, String> result = new ConcurrentHashMap<>(urlList.size());
HashSet<String> finalExistsUrlSet = existsUrlSet;
urlList.stream().distinct().filter(url -> finalExistsUrlSet.contains(url)).forEach(url -> {
result.put(url, awsS3Proxy.getS3Url(MD5Util.hash(url), 0, JPG));
});
List<CompletableFuture> futures = Lists.newArrayList();
urlList.stream().distinct().filter(url -> !finalExistsUrlSet.contains(url)).forEach(url -> {
futures.add(CompletableFuture.runAsync(() -> doUpload(result, url), dealImageTaskExecutor));
});
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
return result;
}
private void doUpload(ConcurrentHashMap<String, String> result, String url) {
Stopwatch stopwatch = Stopwatch.createStarted();
String hash = MD5Util.hash(url);
try {
try {
//下载图片到本地
doUploadInner(url, hash);
} catch (FileNotFoundException e) {
//todo 文件不存在删除图片
return;
} catch (Exception e) {
if (url.startsWith("https:")) {
doUploadInner(StringUtils.replace(url, "https:", "http:"), hash);
} else if (url.startsWith("http:")) {
doUploadInner(StringUtils.replace(url, "http:", "https:"), hash);
}
}
} catch (Exception e) {
log.error("图片上传aws s3失败 url={}, {}", url, ExceptionUtil.printStack(e));
//qywxBiz.asynSendMsg(Lists.newArrayList(391), String.format("%s环境,图片上传aws_s3失败 url=%s", env, url));//发微信消息
return;
}
//记录处理
S3ImageMapDO entity = new S3ImageMapDO()
.setOriginUrl(url)
.setS3Key(hash)
.setStatus(1);
s3ImageMapMapper.upsertWithPrefix(entity, hash.substring(hash.length() - 2));
String s3Url = awsS3Proxy.getS3Url(hash, 0, JPG);
long curCount = count.incrementAndGet();
log.warn(String.format("图片上传到aws_s3成功, 处理总数:%d, 耗时:%d毫秒, url:%s \ns3Url:%s", curCount, stopwatch.elapsed(TimeUnit.MILLISECONDS), url, s3Url));
result.put(url, s3Url);
}
private void doUploadInner(String url, String hash) throws Exception {
byte[] imageByte = downloadPicture(url);
BufferedImage srcImage = ImageIO.read(new ByteArrayInputStream(imageByte));
converAndUpload(hash, srcImage);
}
private void converAndUpload(String hash, BufferedImage srcImage) throws Exception {
byte[] imageByte = ImageUtils.convert(srcImage, WEBP, hash);
ObjectMetadata meta = new ObjectMetadata();
meta.setContentLength(imageByte.length);
meta.setContentType(WEBP_MIME);
//缓存10年
meta.setCacheControl("max-age=315360000");
awsS3Proxy.uploadToS3(new ByteArrayInputStream(imageByte), ImageUtils.getKey(hash, 0, WEBP), meta);
}
//链接url下载图片
public static byte[] downloadPicture(String url) throws IOException, JSchException {
byte[] bytes = new byte[0];
try {
HttpURLConnection connection = (HttpURLConnection) new URL(MyURIEncoder.encode(url)).openConnection();
// 建立连接所用的时间
connection.setConnectTimeout(CONNECT_TIMEOUT * 1000);
// 建立连接后从服务器读取到可用资源所8用的时间
connection.setReadTimeout(MAX_TIME * 1000);
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36");
connection.setRequestProperty("Accept", "*/*");
InputStream in = connection.getInputStream();
bytes = getByte(in);
in.close();
connection.disconnect();
} catch (FileNotFoundException e) {
//todo 文件不存在删除图片
} catch (IOException e) {
//网络不可达, 通过ssh代理, 再去下载一次
bytes = SshProxyDownloadProxy.downloadImageByProxy(url);
}
//测试使用
//bytes = SshProxyDownloadUtil.downloadImageByProxy(url);
return bytes;
}
public static byte[] getByte(InputStream inputStream) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
try (DataInputStream in = new DataInputStream(inputStream)) {
byte[] buffer = new byte[4096];
int count = 0;
while ((count = in.read(buffer)) != -1) {
out.write(buffer, 0, count);
}
}
return out.toByteArray();
}
private final String proxyIp = "172.31.12.140";
private final Integer proxyPort = 9090;
public AsyncHttpClient httpClient = HttpUtil.getAsyncHttpClient(3_000, 2_000);
public AsyncHttpClient proxyHttpClient = HttpUtil.getAsyncHttpClient(3_000, 2_000, proxyIp, proxyPort);
private final int CODE_200 = 200;
private final int CODE_403 = 403;
private final String CONTENT_TYPE = "Content-Type";
public Set<String> checkAvailableImageByProxy(List<String> urlList) {
//国内网络进行检查
Set<String> result = Sets.newConcurrentHashSet();
List<CompletableFuture> futures = urlList.stream().distinct().map(url -> CompletableFuture.runAsync(() -> {
try {
if (isAvailableImage(HttpUtil.getHttpHead(url, httpClient))) {
result.add(url);
}
} catch (Exception e) {
log.info("url:{}, {}", url, e);
}
}, checkImagetaskExecutor)).collect(Collectors.toList());
FutureUtil.joinVoid(futures);
//检查不成功的国外网络进行检查
urlList.removeAll(result);
futures = urlList.stream().distinct().map(url -> CompletableFuture.runAsync(() -> {
try {
if (isAvailableImage(HttpUtil.getHttpHead(url, proxyHttpClient))) {
result.add(url);
}
} catch (Exception e) {
log.info("url:{}, {}", url, e);
}
}, checkImagetaskExecutor)).collect(Collectors.toList());
FutureUtil.joinVoid(futures);
return result;
}
private boolean isAvailableImage(Response response) {
switch (response.getStatusCode()) {
case CODE_403:
return true;
case CODE_200:
String contentType = response.getHeader(CONTENT_TYPE);
if (BaseUtil.isNotEmpty(contentType) && isImageHead(contentType)) {
return true;
}
default:
return false;
}
}
private boolean isImageHead(String contentType) {
switch (contentType) {
case "image/jpg":
case "image/jpeg":
case "image/png":
case "image/tiff":
case "image/pcx":
case "image/bmp":
return true;
}
return false;
}
}
4.4 遇到的问题
4.4.1 jdk原生不支持webp格式
webp属于较新的图片格式, jdk原生不支持; 需要引入依赖jar包"webp-imageio-core"才能支持webp格式
备注: jdk以插件方式支持添加新图片格式处理
引入jar包"webp-imageio-core"后, mac上调试报错
- 发现linux上运行正常; 初步定系统相关的依赖有问题
- 进一步分析发现webp-imageio-core根据不同操作系统, 引入不同动态链接库, 判断为mac对应的态链接库"libwebp-imageio.dylib"有问题
- 找到"libwebp-imageio.dylib"对应的git项目"webp-imageio", 在mac上编译生成动态链接库
- 使用编译生成的mac动态链接库本机调试运行, 功能正常
到此完美支持webp图片格式转换&等比缩放
4.4.2 图片处理多格式支持
使用twelvemonkeys.imageio替换jdk原生图片处理
最终支持图片格式为: jpeg tiff bmp pnm psd hdr iff pcx pict sgi tga icns thumbsdb
背景: 发现图片处理不支持tiff格式图片, 解决方案为添加jar包依赖twelvemonkeys.imageio中的imageio-tiff
进一步了解twelvemonkeys.imageio发现,其对jpeg等常用格式支持的比jdk原生要好(性能等因素), 而且其支持的格式范围非常广, 因此使用twelvemonkeys.imagei替换jdk原生图片处理
4.4.3 图片处理url转码支持
重写UrlEncode, dontNeedEncoding添加不转换字符
背景: 发现部分图片url, 浏览器访问正常, 代码http访问没有网络资源
分析发现, 浏览器url和代码中的url已经不一致, 浏览器自动对url进行了转码, 重写UrlEncode实现和浏览器一致的编码转换
MyURIEncoder
- dontNeedEncodin原生包含: - _ . *
- dontNeedEncodin添加: ; / ? : @ & = + $ , # ! ' ( ) % [ ]
import sun.security.action.GetPropertyAction;
import java.io.CharArrayWriter;
import java.io.UnsupportedEncodingException;
import java.nio.charset.Charset;
import java.nio.charset.IllegalCharsetNameException;
import java.nio.charset.UnsupportedCharsetException;
import java.security.AccessController;
import java.util.BitSet;
/**
* Utility class for HTML form encoding. This class contains static methods
* for converting a String to the <CODE>application/x-www-form-urlencoded</CODE> MIME
* format. For more information about HTML form encoding, consult the HTML
* <A HREF="http://www.w3.org/TR/html4/">specification</A>.
* <p>
* <p>
* When encoding a String, the following rules apply:
* <p>
* <ul>
* <li>The alphanumeric characters "{@code a}" through
* "{@code z}", "{@code A}" through
* "{@code Z}" and "{@code 0}"
* through "{@code 9}" remain the same.
* <li>The special characters "{@code .}",
* "{@code -}", "{@code *}", and
* "{@code _}" remain the same.
* <li>The space character " " is
* converted into a plus sign "{@code +}".
* <li>All other characters are unsafe and are first converted into
* one or more bytes using some encoding scheme. Then each byte is
* represented by the 3-character string
* "<i>{@code %xy}</i>", where <i>xy</i> is the
* two-digit hexadecimal representation of the byte.
* The recommended encoding scheme to use is UTF-8. However,
* for compatibility reasons, if an encoding is not specified,
* then the default encoding of the platform is used.
* </ul>
* <p>
* <p>
* For example using UTF-8 as the encoding scheme the string "The
* string ü@foo-bar" would get converted to
* "The+string+%C3%BC%40foo-bar" because in UTF-8 the character
* ü is encoded as two bytes C3 (hex) and BC (hex), and the
* character @ is encoded as one byte 40 (hex).
*
* @author Herb Jellinek
* @since JDK1.0
*/
public class MyURIEncoder {
static BitSet dontNeedEncoding;
static final int caseDiff = ('a' - 'A');
static String dfltEncName = null;
static {
/* The list of characters that are not encoded has been
* determined as follows:
*
* RFC 2396 states:
* -----
* Data characters that are allowed in a URI but do not have a
* reserved purpose are called unreserved. These include upper
* and lower case letters, decimal digits, and a limited set of
* punctuation marks and symbols.
*
* unreserved = alphanum | mark
*
* mark = "-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")"
*
* Unreserved characters can be escaped without changing the
* semantics of the URI, but this should not be done unless the
* URI is being used in a context that does not allow the
* unescaped character to appear.
* -----
*
* It appears that both Netscape and Internet Explorer escape
* all special characters from this list with the exception
* of "-", "_", ".", "*". While it is not clear why they are
* escaping the other characters, perhaps it is safest to
* assume that there might be contexts in which the others
* are unsafe if not escaped. Therefore, we will use the same
* list. It is also noteworthy that this is consistent with
* O'Reilly's "HTML: The Definitive Guide" (page 164).
*
* As a last note, Intenet Explorer does not encode the "@"
* character which is clearly not unreserved according to the
* RFC. We are being consistent with the RFC in this matter,
* as is Netscape.
*
*/
dontNeedEncoding = new BitSet(256);
int i;
for (i = 'a'; i <= 'z'; i++) {
dontNeedEncoding.set(i);
}
for (i = 'A'; i <= 'Z'; i++) {
dontNeedEncoding.set(i);
}
for (i = '0'; i <= '9'; i++) {
dontNeedEncoding.set(i);
}
//dontNeedEncoding.set(' '); /* encoding a space to a + is done * in the encode() method */
dontNeedEncoding.set('-');
dontNeedEncoding.set('_');
dontNeedEncoding.set('.');
dontNeedEncoding.set('*');
//对以下在 URI 中具有特殊含义的 ASCII 标点符号 ;/?:@&=+$,# 不需要转义
dontNeedEncoding.set(';');
dontNeedEncoding.set('/');
dontNeedEncoding.set('?');
dontNeedEncoding.set(':');
dontNeedEncoding.set('@');
dontNeedEncoding.set('&');
dontNeedEncoding.set('=');
dontNeedEncoding.set('+');
dontNeedEncoding.set('$');
dontNeedEncoding.set(',');
dontNeedEncoding.set('#');
dontNeedEncoding.set('!');
dontNeedEncoding.set('\'');
dontNeedEncoding.set('(');
dontNeedEncoding.set(')');
dontNeedEncoding.set('%');
dontNeedEncoding.set('[');
dontNeedEncoding.set(']');
dfltEncName = AccessController.doPrivileged(
new GetPropertyAction("file.encoding")
);
}
/**
* You can't call the constructor.
*/
private MyURIEncoder() {
}
/**
* Translates a string into {@code x-www-form-urlencoded}
* format. This method uses the platform's default encoding
* as the encoding scheme to obtain the bytes for unsafe characters.
*
* @param s {@code String} to be translated.
* @return the translated {@code String}.
* @deprecated The resulting string may vary depending on the platform's
* default encoding. Instead, use the encode(String,String)
* method to specify the encoding.
*/
@Deprecated
public static String encode(String s) {
String str = null;
try {
str = encode(s, dfltEncName);
} catch (UnsupportedEncodingException e) {
// The system should always have the platform default
}
return str;
}
/**
* Translates a string into {@code application/x-www-form-urlencoded}
* format using a specific encoding scheme. This method uses the
* supplied encoding scheme to obtain the bytes for unsafe
* characters.
* <p>
* <em><strong>Note:</strong> The <a href=
* "http://www.w3.org/TR/html40/appendix/notes.html#non-ascii-chars">
* World Wide Web Consortium Recommendation</a> states that
* UTF-8 should be used. Not doing so may introduce
* incompatibilities.</em>
*
* @param s {@code String} to be translated.
* @param enc The name of a supported
* <a href="../lang/package-summary.html#charenc">character
* encoding</a>.
* @return the translated {@code String}.
* @throws UnsupportedEncodingException If the named encoding is not supported
* @since 1.4
*/
public static String encode(String s, String enc)
throws UnsupportedEncodingException {
boolean needToChange = false;
StringBuffer out = new StringBuffer(s.length());
Charset charset;
CharArrayWriter charArrayWriter = new CharArrayWriter();
if (enc == null)
throw new NullPointerException("charsetName");
try {
charset = Charset.forName(enc);
} catch (IllegalCharsetNameException e) {
throw new UnsupportedEncodingException(enc);
} catch (UnsupportedCharsetException e) {
throw new UnsupportedEncodingException(enc);
}
for (int i = 0; i < s.length(); ) {
int c = (int) s.charAt(i);
if (dontNeedEncoding.get(c)) {
out.append((char) c);
i++;
} else {
// convert to external encoding before hex conversion
do {
charArrayWriter.write(c);
/*
* If this character represents the start of a Unicode
* surrogate pair, then pass in two characters. It's not
* clear what should be done if a bytes reserved in the
* surrogate pairs range occurs outside of a legal
* surrogate pair. For now, just treat it as if it were
* any other character.
*/
if (c >= 0xD800 && c <= 0xDBFF) {
/*
System.out.println(Integer.toHexString(c)
+ " is high surrogate");
*/
if ((i + 1) < s.length()) {
int d = (int) s.charAt(i + 1);
/*
System.out.println("\tExamining "
+ Integer.toHexString(d));
*/
if (d >= 0xDC00 && d <= 0xDFFF) {
/*
System.out.println("\t"
+ Integer.toHexString(d)
+ " is low surrogate");
*/
charArrayWriter.write(d);
i++;
}
}
}
i++;
} while (i < s.length() && !dontNeedEncoding.get((c = (int) s.charAt(i))));
charArrayWriter.flush();
String str = new String(charArrayWriter.toCharArray());
byte[] ba = str.getBytes(charset);
for (int j = 0; j < ba.length; j++) {
out.append('%');
char ch = Character.forDigit((ba[j] >> 4) & 0xF, 16);
// converting to use uppercase letter as part of
// the hex value if ch is a letter.
if (Character.isLetter(ch)) {
ch -= caseDiff;
}
out.append(ch);
ch = Character.forDigit(ba[j] & 0xF, 16);
if (Character.isLetter(ch)) {
ch -= caseDiff;
}
out.append(ch);
}
charArrayWriter.reset();
needToChange = true;
}
}
return (needToChange ? out.toString() : s);
}
}
4.4.4 图片处理多尺寸支持, 全链路定义图片尺寸, 图片使用规范化
背景: 发现C端部分图片展示缓慢, 分析发现C端,商品管理后台等多个系统使用原图展示, 部分原图过大(大于10M), 大量商品原图大于1M
问题反馈后多方参与, 发现当前系统支持图片格式为:原图.webp, 原图.jpg, 350x350.webp, 350x350.jpg, 200x200.webp, 200x200.jpg, 50x50.webp, 50x50.jpg
缺失用于商品详情展示的尺寸, 产品制定规范, 添加新尺寸800x800 650x650, 所有详情, 大图使用800x800, 各系统中使用800x800展示详情
图片处理中心, 等比缩放添加800x800 650x650尺寸, 存量历史图片新增800x800 650x650尺寸
五. 前后数据对比:
- 处理前: 详情图片大小200k~2M 加载时间(446k ): 2.39s
- 处理后: 详情图片平均大小60k 加载时间(60.5k): 351ms
C端加载速度得到大幅提升, 同时也节省了CDN流量费用
六. 后续优化:
优化点: 在能保证图片质量的条件下, 进一步提高压缩比, 提升加载速度
网友评论