美文网首页
URL 转码遇到的坑

URL 转码遇到的坑

作者: wyh1791 | 来源:发表于2019-10-11 20:29 被阅读0次

    1. 背景

    做图片中心时, 需要根据图片URL去下载图片, 发现部分URL浏览器里面能访问,但是代码无法下载

    原因: 需要对URL进行转码

    2. 使用UrlEncode

    一开始直接使用UrlEncode对url转码, 如下

    • 原始链接
    https://s3.amazonaws.com/fromfactory.club.image/9f/0f/9fbd598c4681773f6371c27dfb64180f.webp
    
    • UrlEncode后
    https%3a%2f%2fs3.amazonaws.com%2ffromfactory.club.image%2f9f%2f0f%2f9fbd598c4681773f6371c27dfb64180f.webp
    

    2.1 问题

    斜杠,冒号等字符也被UrlEncode进行了转码, 显然这样的转码后是不能下载的

    3. 原因分析

    编码的意义在于,假如URL的参数中的中文或特殊字符在发送到服务端时,服务端无法解析它的真正意义,会导致服务端不能理解客户端的请求, 此时, 需要对它进行编码

    3.1 哪些字符需要编码

    RFC3986文档规定,URL中只允许包含英文字母(a-zA-Z)、数字(0-9)、-_.~4个特殊字符以及所有保留字符。

    • 保留字符:Url可以划分成若干个组件,协议、主机、路径等。有一些字符(:/?#[]@)是用作分隔不同组件的。
      当组件中的普通数据包含这些特殊字符时,需要对其进行编码。

    • RFC3986中指定了以下字符为==保留字符:! * ’ ( ) ; : @ & = + $ , / ? # [ ]==

    对于图片下载中, 如下都不转码

    • 英文字母 a-zA-Z
    • 数字 0-9
    • 4个特殊字符 _.~、
    • 保留字符 ! * ’ ( ) ; : @ & = + $ , / ? # [ ]==

    3.2 UrlEncode不转码字符

    UrlEncode中维护了dontNeedEncoding, 但是dontNeedEncoding只包含了

    • 英文字母(a-zA-Z)
    • 数字(0-9)
    • 空格 - _ . *
      ab8f20ac3c5a4b0f9beb02bd9be627fb_image.png

    如下字符也需要不转码: ! ~ ’ ( ) ; : @ & = + $ , / ? # [ ]==

    4. 解决方案

    重写UrlEncode添加不转码字符

    7be93685d8544476aa88d21c35ec3dd1_image.png

    4.1 代码

    package com.clubfactory.center.product.util;
    
    
    import sun.security.action.GetPropertyAction;
    
    import java.io.CharArrayWriter;
    import java.io.UnsupportedEncodingException;
    import java.nio.charset.Charset;
    import java.nio.charset.IllegalCharsetNameException;
    import java.nio.charset.UnsupportedCharsetException;
    import java.security.AccessController;
    import java.util.BitSet;
    
    /**
     * Utility class for HTML form encoding. This class contains static methods
     * for converting a String to the <CODE>application/x-www-form-urlencoded</CODE> MIME
     * format. For more information about HTML form encoding, consult the HTML
     * <A HREF="http://www.w3.org/TR/html4/">specification</A>.
     *
     * <p>
     * When encoding a String, the following rules apply:
     *
     * <ul>
     * <li>The alphanumeric characters &quot;{@code a}&quot; through
     *     &quot;{@code z}&quot;, &quot;{@code A}&quot; through
     *     &quot;{@code Z}&quot; and &quot;{@code 0}&quot;
     *     through &quot;{@code 9}&quot; remain the same.
     * <li>The special characters &quot;{@code .}&quot;,
     *     &quot;{@code -}&quot;, &quot;{@code *}&quot;, and
     *     &quot;{@code _}&quot; remain the same.
     * <li>The space character &quot; &nbsp; &quot; is
     *     converted into a plus sign &quot;{@code +}&quot;.
     * <li>All other characters are unsafe and are first converted into
     *     one or more bytes using some encoding scheme. Then each byte is
     *     represented by the 3-character string
     *     &quot;<i>{@code %xy}</i>&quot;, where <i>xy</i> is the
     *     two-digit hexadecimal representation of the byte.
     *     The recommended encoding scheme to use is UTF-8. However,
     *     for compatibility reasons, if an encoding is not specified,
     *     then the default encoding of the platform is used.
     * </ul>
     *
     * <p>
     * For example using UTF-8 as the encoding scheme the string &quot;The
     * string &#252;@foo-bar&quot; would get converted to
     * &quot;The+string+%C3%BC%40foo-bar&quot; because in UTF-8 the character
     * &#252; is encoded as two bytes C3 (hex) and BC (hex), and the
     * character @ is encoded as one byte 40 (hex).
     *
     * @author  Herb Jellinek
     * @since   JDK1.0
     */
    public class MyURIEncoder {
        static BitSet dontNeedEncoding;
        static final int caseDiff = ('a' - 'A');
        static String dfltEncName = null;
    
        static {
    
            /* The list of characters that are not encoded has been
             * determined as follows:
             *
             * RFC 2396 states:
             * -----
             * Data characters that are allowed in a URI but do not have a
             * reserved purpose are called unreserved.  These include upper
             * and lower case letters, decimal digits, and a limited set of
             * punctuation marks and symbols.
             *
             * unreserved  = alphanum | mark
             *
             * mark        = "-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")"
             *
             * Unreserved characters can be escaped without changing the
             * semantics of the URI, but this should not be done unless the
             * URI is being used in a context that does not allow the
             * unescaped character to appear.
             * -----
             *
             * It appears that both Netscape and Internet Explorer escape
             * all special characters from this list with the exception
             * of "-", "_", ".", "*". While it is not clear why they are
             * escaping the other characters, perhaps it is safest to
             * assume that there might be contexts in which the others
             * are unsafe if not escaped. Therefore, we will use the same
             * list. It is also noteworthy that this is consistent with
             * O'Reilly's "HTML: The Definitive Guide" (page 164).
             *
             * As a last note, Intenet Explorer does not encode the "@"
             * character which is clearly not unreserved according to the
             * RFC. We are being consistent with the RFC in this matter,
             * as is Netscape.
             *
             */
    
            dontNeedEncoding = new BitSet(256);
            int i;
            for (i = 'a'; i <= 'z'; i++) {
                dontNeedEncoding.set(i);
            }
            for (i = 'A'; i <= 'Z'; i++) {
                dontNeedEncoding.set(i);
            }
            for (i = '0'; i <= '9'; i++) {
                dontNeedEncoding.set(i);
            }
            //dontNeedEncoding.set(' '); /* encoding a space to a + is done * in the encode() method */
            dontNeedEncoding.set('-');
            dontNeedEncoding.set('_');
            dontNeedEncoding.set('.');
            dontNeedEncoding.set('*');
    
            //对以下在 URI 中具有特殊含义的 ASCII 标点符号    ;/?:@&=+$,#  不需要转义
            dontNeedEncoding.set(';');
            dontNeedEncoding.set('/');
            dontNeedEncoding.set('?');
            dontNeedEncoding.set(':');
            dontNeedEncoding.set('@');
            dontNeedEncoding.set('&');
            dontNeedEncoding.set('=');
            dontNeedEncoding.set('+');
            dontNeedEncoding.set('$');
            dontNeedEncoding.set(',');
            dontNeedEncoding.set('#');
            dontNeedEncoding.set('!');
            dontNeedEncoding.set('\'');
            dontNeedEncoding.set('(');
            dontNeedEncoding.set(')');
            dontNeedEncoding.set('%');
            dontNeedEncoding.set('[');
            dontNeedEncoding.set(']');
    
            dfltEncName = AccessController.doPrivileged(
                    new GetPropertyAction("file.encoding")
            );
        }
    
        /**
         * You can't call the constructor.
         */
        private MyURIEncoder() { }
    
        /**
         * Translates a string into {@code x-www-form-urlencoded}
         * format. This method uses the platform's default encoding
         * as the encoding scheme to obtain the bytes for unsafe characters.
         *
         * @param   s   {@code String} to be translated.
         * @deprecated The resulting string may vary depending on the platform's
         *             default encoding. Instead, use the encode(String,String)
         *             method to specify the encoding.
         * @return  the translated {@code String}.
         */
        @Deprecated
        public static String encode(String s) {
    
            String str = null;
    
            try {
                str = encode(s, dfltEncName);
            } catch (UnsupportedEncodingException e) {
                // The system should always have the platform default
            }
    
            return str;
        }
    
        /**
         * Translates a string into {@code application/x-www-form-urlencoded}
         * format using a specific encoding scheme. This method uses the
         * supplied encoding scheme to obtain the bytes for unsafe
         * characters.
         * <p>
         * <em><strong>Note:</strong> The <a href=
         * "http://www.w3.org/TR/html40/appendix/notes.html#non-ascii-chars">
         * World Wide Web Consortium Recommendation</a> states that
         * UTF-8 should be used. Not doing so may introduce
         * incompatibilities.</em>
         *
         * @param   s   {@code String} to be translated.
         * @param   enc   The name of a supported
         *    <a href="../lang/package-summary.html#charenc">character
         *    encoding</a>.
         * @return  the translated {@code String}.
         * @exception  UnsupportedEncodingException
         *             If the named encoding is not supported
         * @since 1.4
         */
        public static String encode(String s, String enc)
                throws UnsupportedEncodingException {
    
            boolean needToChange = false;
            StringBuffer out = new StringBuffer(s.length());
            Charset charset;
            CharArrayWriter charArrayWriter = new CharArrayWriter();
    
            if (enc == null)
                throw new NullPointerException("charsetName");
    
            try {
                charset = Charset.forName(enc);
            } catch (IllegalCharsetNameException e) {
                throw new UnsupportedEncodingException(enc);
            } catch (UnsupportedCharsetException e) {
                throw new UnsupportedEncodingException(enc);
            }
    
            for (int i = 0; i < s.length();) {
                int c = (int) s.charAt(i);
                if (dontNeedEncoding.get(c)) {
                    out.append((char)c);
                    i++;
                } else {
                    // convert to external encoding before hex conversion
                    do {
                        charArrayWriter.write(c);
                        /*
                         * If this character represents the start of a Unicode
                         * surrogate pair, then pass in two characters. It's not
                         * clear what should be done if a bytes reserved in the
                         * surrogate pairs range occurs outside of a legal
                         * surrogate pair. For now, just treat it as if it were
                         * any other character.
                         */
                        if (c >= 0xD800 && c <= 0xDBFF) {
                            /*
                              System.out.println(Integer.toHexString(c)
                              + " is high surrogate");
                            */
                            if ( (i+1) < s.length()) {
                                int d = (int) s.charAt(i+1);
                                /*
                                  System.out.println("\tExamining "
                                  + Integer.toHexString(d));
                                */
                                if (d >= 0xDC00 && d <= 0xDFFF) {
                                    /*
                                      System.out.println("\t"
                                      + Integer.toHexString(d)
                                      + " is low surrogate");
                                    */
                                    charArrayWriter.write(d);
                                    i++;
                                }
                            }
                        }
                        i++;
                    } while (i < s.length() && !dontNeedEncoding.get((c = (int) s.charAt(i))));
    
                    charArrayWriter.flush();
                    String str = new String(charArrayWriter.toCharArray());
                    byte[] ba = str.getBytes(charset);
                    for (int j = 0; j < ba.length; j++) {
                        out.append('%');
                        char ch = Character.forDigit((ba[j] >> 4) & 0xF, 16);
                        // converting to use uppercase letter as part of
                        // the hex value if ch is a letter.
                        if (Character.isLetter(ch)) {
                            ch -= caseDiff;
                        }
                        out.append(ch);
                        ch = Character.forDigit(ba[j] & 0xF, 16);
                        if (Character.isLetter(ch)) {
                            ch -= caseDiff;
                        }
                        out.append(ch);
                    }
                    charArrayWriter.reset();
                    needToChange = true;
                }
            }
    
            return (needToChange? out.toString() : s);
        }
    }
    
    

    相关文章

      网友评论

          本文标题:URL 转码遇到的坑

          本文链接:https://www.haomeiwen.com/subject/emixmctx.html