美文网首页Framwork程序猿的进阶屋Nginx
详解proxy_pass、upstream与resolver

详解proxy_pass、upstream与resolver

作者: boldcautious | 来源:发表于2018-06-04 10:48 被阅读1549次

    应用场景

    这里列举几个应用场景,下文会针对这几个场景并结合代码进行分析。

    1. proxy_pass + upstream
        upstream foo.example.com {
            server 127.0.0.1:8001;
        }
    
        server {
            listen       80;
            server_name  localhost;
    
            location /foo {
                proxy_pass http://foo.example.com;
            }
        }
    

    访问http://localhost/foo,proxy模块会将请求转发到127.0.0.1的8001端口上。

    1. 只有proxy_pass,没有upstream与resolver
        server {
            listen       80;
            server_name  localhost;
    
            location /foo {
                proxy_pass http://foo.example.com;
            }
        }
    

    实际上是隐式创建了upstream,upstream名字就是foo.example.com。upstream模块利用本机设置的DNS服务器(或/etc/hosts),将foo.example.com解析成IP,访问http://localhost/foo,proxy模块会将请求转发到解析后的IP上。

    如果本机未设置DNS服务器,或者DNS服务器无法解析域名,则nginx启动时会报类似如下错误:

    nginx: [emerg] host not found in upstream "foo.example.com" in /path/nginx/conf/nginx.conf:110

    1. proxy_pass + resolver(变量设置域名)
        server {
            listen       80;
            server_name  localhost;
    
            resolver 114.114.114.114;
            location /foo {
                set $foo foo.example.com;
                proxy_pass http://$foo;
            }
        }
    

    访问http://localhost/foo,nginx会动态利用resolver设置的DNS服务器(本机设置的DNS服务器或/etc/hosts无效),将域名解析成IP,proxy模块会将请求转发到解析后的IP上。

    1. proxy_pass + upstream(显式) + resolver(变量设置域名)
        upstream foo.example.com {
            server 127.0.0.1:8001;
        }
    
        server {
            listen       80;
            server_name  localhost;
    
            resolver 114.114.114.114;
            location /foo {
                set $foo foo.example.com;
                proxy_pass http://$foo;
            }
        }
    

    访问http://localhost/foo时,upstream模块会优先查找是否有定义upstream后端服务器,如果有定义则直接利用,不再走DNS解析。所以proxy模块会将请求转发到127.0.0.1的8001端口上。

    1. proxy_pass + upstream(隐式) + resolver(变量设置域名)
        server {
            listen       80;
            server_name  localhost;
    
            resolver 114.114.114.114;
            location /foo {
                set $foo foo.example.com;
                proxy_pass http://$foo;
            }
    
            location /foo2 {
                proxy_pass http://foo.example.com;
            }
        }
    

    location /foo2实际上是隐式定义了upstream foo.example.com,并由本地DNS服务器进行了域名解析,访问http://localhost/foo时,upstream模块会优先查找upstream,即隐式定义的foo.example.com,proxy模块会将请求转发到解析后的IP上。

    1. proxy_pass + resolver(不用变量设置域名)
        server {
            listen       80;
            server_name  localhost;
    
            resolver 114.114.114.114;
            location /foo {
                proxy_pass http://foo.example.com;
            }
        }
    

    不使用变量设置域名,则resolver的设置不起作用,此时相当于场景2,只有proxy_pass的场景。

    1. proxy_pass + upstream + resolver(不用变量设置域名)
        upstream foo.example.com {
            server 127.0.0.1:8001;
        }
    
        server {
            listen       80;
            server_name  localhost;
    
            resolver 114.114.114.114;
            location /foo {
                proxy_pass http://foo.example.com;
            }
        }
    

    不使用变量设置域名,则resolver的设置不起作用,此时相当于场景1 proxy_pass + upstream。

    1. proxy_pass 直接指定IP加端口号
        server {
            listen       80;
            server_name  localhost;
    
            location /foo {
                proxy_pass http://127.0.0.1:8001/;
            }
        }
    

    实际上是隐式创建了upstream,proxy_pass会将请求转发到127.0.0.1的8001端口上。

    主要代码

    解析proxy_pass指令的代码:

    static char *
    ngx_http_proxy_pass(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
    {
        ngx_http_proxy_loc_conf_t *plcf = conf;
    
        size_t                      add;
        u_short                     port;
        ngx_str_t                  *value, *url;
        ngx_url_t                   u;
        ngx_uint_t                  n;
        ngx_http_core_loc_conf_t   *clcf;
        ngx_http_script_compile_t   sc;
    
        if (plcf->upstream.upstream || plcf->proxy_lengths) {
            return "is duplicate";
        }
    
        clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module);
    
        clcf->handler = ngx_http_proxy_handler;
    
        if (clcf->name.data[clcf->name.len - 1] == '/') {
            clcf->auto_redirect = 1;
        }
    
        value = cf->args->elts;
    
        url = &value[1];
    
        /* 查找指令中$符号的位置,判断是否使用了变量 */
        n = ngx_http_script_variables_count(url);
    
        if (n) {
            /* 使用变量设置域名 */
            ngx_memzero(&sc, sizeof(ngx_http_script_compile_t));
    
            sc.cf = cf;
            sc.source = url;
            sc.lengths = &plcf->proxy_lengths;
            sc.values = &plcf->proxy_values;
            sc.variables = n;
            sc.complete_lengths = 1;
            sc.complete_values = 1;
    
            if (ngx_http_script_compile(&sc) != NGX_OK) {
                return NGX_CONF_ERROR;
            }
    
    #if (NGX_HTTP_SSL)
            plcf->ssl = 1;
    #endif
    
            return NGX_CONF_OK;
        }
    
        if (ngx_strncasecmp(url->data, (u_char *) "http://", 7) == 0) {
            add = 7;
            port = 80;
    
        } else if (ngx_strncasecmp(url->data, (u_char *) "https://", 8) == 0) {
    
    #if (NGX_HTTP_SSL)
            plcf->ssl = 1;
    
            add = 8;
            port = 443;
    #else
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "https protocol requires SSL support");
            return NGX_CONF_ERROR;
    #endif
    
        } else {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "invalid URL prefix");
            return NGX_CONF_ERROR;
        }
    
        ngx_memzero(&u, sizeof(ngx_url_t));
    
        u.url.len = url->len - add;
        u.url.data = url->data + add;
        u.default_port = port;
        u.uri_part = 1;
        u.no_resolve = 1;
    
        plcf->upstream.upstream = ngx_http_upstream_add(cf, &u, 0);
        if (plcf->upstream.upstream == NULL) {
            return NGX_CONF_ERROR;
        }
    
        plcf->vars.schema.len = add;
        plcf->vars.schema.data = url->data;
        plcf->vars.key_start = plcf->vars.schema;
    
        ngx_http_proxy_set_vars(&u, &plcf->vars);
    
        plcf->location = clcf->name;
    
        if (clcf->named
    #if (NGX_PCRE)
            || clcf->regex
    #endif
            || clcf->noname)
        {
            if (plcf->vars.uri.len) {
                ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                   "\"proxy_pass\" cannot have URI part in "
                                   "location given by regular expression, "
                                   "or inside named location, "
                                   "or inside \"if\" statement, "
                                   "or inside \"limit_except\" block");
                return NGX_CONF_ERROR;
            }
    
            plcf->location.len = 0;
        }
    
        plcf->url = *url;
    
        return NGX_CONF_OK;
    }
    

    upstream定义的后端服务器的处理逻辑,包括显式定义的和隐式定义的。隐式定义,即proxy_pass指定的后端服务器的地址没有显式用upstream定义,nginx内部会定义一个。

    ngx_int_t
    ngx_http_upstream_init_round_robin(ngx_conf_t *cf,
        ngx_http_upstream_srv_conf_t *us)
    {
        ngx_url_t                      u;
        ngx_uint_t                     i, j, n, w;
        ngx_http_upstream_server_t    *server;
        ngx_http_upstream_rr_peer_t   *peer, **peerp;
        ngx_http_upstream_rr_peers_t  *peers, *backup;
    
        us->peer.init = ngx_http_upstream_init_round_robin_peer;
    
        /* 
         * 使用upstream指令显式定义upstream 
         * 或者proxy_pass直接指定IP的场景
         */
        if (us->servers) {
            server = us->servers->elts;
    
            n = 0;
            w = 0;
    
            for (i = 0; i < us->servers->nelts; i++) {
                if (server[i].backup) {
                    continue;
                }
    
                n += server[i].naddrs;
                w += server[i].naddrs * server[i].weight;
            }
    
            if (n == 0) {
                ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
                              "no servers in upstream \"%V\" in %s:%ui",
                              &us->host, us->file_name, us->line);
                return NGX_ERROR;
            }
    
            peers = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_rr_peers_t));
            if (peers == NULL) {
                return NGX_ERROR;
            }
    
            peer = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_rr_peer_t) * n);
            if (peer == NULL) {
                return NGX_ERROR;
            }
    
            peers->single = (n == 1);
            peers->number = n;
            peers->weighted = (w != n);
            peers->total_weight = w;
            peers->name = &us->host;
    
            n = 0;
            peerp = &peers->peer;
    
            for (i = 0; i < us->servers->nelts; i++) {
                /* 设置sockaddr、weight、max_fails、fail_timeout等属性 */
            }
    
            us->peer.data = peers;
    
            /* 处理backup servers相关逻辑 */
    
            return NGX_OK;
        }
    
        /* 未使用upstream指令,使用proxy_pass隐式定义upstream */
        /* an upstream implicitly defined by proxy_pass, etc. */
    
        if (us->port == 0) {
            ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
                          "no port in upstream \"%V\" in %s:%ui",
                          &us->host, us->file_name, us->line);
            return NGX_ERROR;
        }
    
        ngx_memzero(&u, sizeof(ngx_url_t));
    
        u.host = us->host;
        u.port = us->port;
    
        if (ngx_inet_resolve_host(cf->pool, &u) != NGX_OK) {
            if (u.err) {
                ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
                              "%s in upstream \"%V\" in %s:%ui",
                              u.err, &us->host, us->file_name, us->line);
            }
    
            return NGX_ERROR;
        }
    
        n = u.naddrs;
    
        peers = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_rr_peers_t));
        if (peers == NULL) {
            return NGX_ERROR;
        }
    
        peer = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_rr_peer_t) * n);
        if (peer == NULL) {
            return NGX_ERROR;
        }
    
        peers->single = (n == 1);
        peers->number = n;
        peers->weighted = 0;
        peers->total_weight = n;
        peers->name = &us->host;
    
        peerp = &peers->peer;
    
        for (i = 0; i < u.naddrs; i++) {
            /* 设置sockaddr、weight、max_fails、fail_timeout等属性 */
        }
    
        us->peer.data = peers;
    
        /* implicitly defined upstream has no backup servers */
    
        return NGX_OK;
    }
    

    upstream模块初始化请求时的逻辑:

    static void
    ngx_http_upstream_init_request(ngx_http_request_t *r)
    {
        ngx_str_t                      *host;
        ngx_uint_t                      i;
        ngx_resolver_ctx_t             *ctx, temp;
        ngx_http_cleanup_t             *cln;
        ngx_http_upstream_t            *u;
        ngx_http_core_loc_conf_t       *clcf;
        ngx_http_upstream_srv_conf_t   *uscf, **uscfp;
        ngx_http_upstream_main_conf_t  *umcf;
    
        if (r->aio) {
            return;
        }
    
        u = r->upstream;
    
        /* NGX_HTTP_CACHE 等其他处理 */
    
        cln->handler = ngx_http_upstream_cleanup;
        cln->data = r;
        u->cleanup = &cln->handler;
    
        if (u->resolved == NULL) {
            /* 如果没有使用resolver设置DNS,直接取upstream的设置 */
            uscf = u->conf->upstream;
    
        } else {
    
    #if (NGX_HTTP_SSL)
            u->ssl_name = u->resolved->host;
    #endif
    
            host = &u->resolved->host;
    
            if (u->resolved->sockaddr) {
    
                if (u->resolved->port == 0
                    && u->resolved->sockaddr->sa_family != AF_UNIX)
                {
                    ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                                  "no port in upstream \"%V\"", host);
                    ngx_http_upstream_finalize_request(r, u,
                                                   NGX_HTTP_INTERNAL_SERVER_ERROR);
                    return;
                }
    
                if (ngx_http_upstream_create_round_robin_peer(r, u->resolved)
                    != NGX_OK)
                {
                    ngx_http_upstream_finalize_request(r, u,
                                                   NGX_HTTP_INTERNAL_SERVER_ERROR);
                    return;
                }
    
                ngx_http_upstream_connect(r, u);
    
                return;
            }
    
            umcf = ngx_http_get_module_main_conf(r, ngx_http_upstream_module);
    
            uscfp = umcf->upstreams.elts;
    
            /* 在显式/隐式定义的upstream中查找 */
            for (i = 0; i < umcf->upstreams.nelts; i++) {
    
                uscf = uscfp[i];
    
                if (uscf->host.len == host->len
                    && ((uscf->port == 0 && u->resolved->no_port)
                         || uscf->port == u->resolved->port)
                    && ngx_strncasecmp(uscf->host.data, host->data, host->len) == 0)
                {
                    goto found;
                }
            }
    
            if (u->resolved->port == 0) {
                ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                              "no port in upstream \"%V\"", host);
                ngx_http_upstream_finalize_request(r, u,
                                                   NGX_HTTP_INTERNAL_SERVER_ERROR);
                return;
            }
    
            temp.name = *host;
    
            ctx = ngx_resolve_start(clcf->resolver, &temp);
            if (ctx == NULL) {
                ngx_http_upstream_finalize_request(r, u,
                                                   NGX_HTTP_INTERNAL_SERVER_ERROR);
                return;
            }
    
            if (ctx == NGX_NO_RESOLVER) {
                ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                              "no resolver defined to resolve %V", host);
    
                ngx_http_upstream_finalize_request(r, u, NGX_HTTP_BAD_GATEWAY);
                return;
            }
    
            ctx->name = *host;
            ctx->handler = ngx_http_upstream_resolve_handler;
            ctx->data = r;
            ctx->timeout = clcf->resolver_timeout;
    
            u->resolved->ctx = ctx;
    
            if (ngx_resolve_name(ctx) != NGX_OK) {
                u->resolved->ctx = NULL;
                ngx_http_upstream_finalize_request(r, u,
                                                   NGX_HTTP_INTERNAL_SERVER_ERROR);
                return;
            }
    
            return;
        }
    
    found:
    
        if (uscf == NULL) {
            ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0,
                          "no upstream configuration");
            ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
            return;
        }
    
    #if (NGX_HTTP_SSL)
        u->ssl_name = uscf->host;
    #endif
    
        if (uscf->peer.init(r, uscf) != NGX_OK) {
            ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
            return;
        }
    
        u->peer.start_time = ngx_current_msec;
    
        if (u->conf->next_upstream_tries
            && u->peer.tries > u->conf->next_upstream_tries)
        {
            u->peer.tries = u->conf->next_upstream_tries;
        }
    
        ngx_http_upstream_connect(r, u);
    }
    

    详细分析

    场景1

    解析proxy_pass的函数ngx_http_proxy_pass中,没有找到$符号(即,变量设置域名),走ngx_http_proxy_pass后半部分的处理逻辑。ngx_http_upstream_init_round_robin初始化upstream时,走显式定义upstream的逻辑。proxy_pass转发请求初始化时,ngx_http_upstream_init_request中直接使用upstream中的后端server建立连接。

    场景2

    ngx_http_upstream_init_round_robin初始化upstream时,走隐式定义upstream的逻辑,会调用ngx_inet_resolve_host对proxy_pass中的域名进行解析,设置upstream。proxy_pass转发请求初始化时,ngx_http_upstream_init_request中直接使用upstream中的设置,也就是利用本地设置的DNS服务器解析出的IP,建立连接。

    场景3

    解析proxy_pass指令时,找到了$符号,设置ngx_http_script_compile_t,并利用ngx_http_script_compile进行编译,不走后半部分逻辑。配置文件没有显式/隐式定义upstream,所以不会调用ngx_http_upstream_init_round_robin方法。proxy_pass转发请求初始化时,ngx_http_upstream_init_request中发现没有显式也没有隐式定义的upstream,随后调用ngx_resolve_start,对域名进行解析,之后将请求转发过去。

    场景4

    解析proxy_pass指令时,找到了$符号,设置ngx_http_script_compile_t,并利用ngx_http_script_compile进行编译,不走后半部分逻辑。显式调用了upstream,所以调用ngx_http_upstream_init_round_robin方法中的显式upstream的处理逻辑。proxy_pass转发请求初始化时,ngx_http_upstream_init_request中优先查找upstream,如果找到了,直接将请求转发到upstream中的后端server上。如果upstream中没有找到,则对域名进行解析,然后将请求转发到解析后的IP上。

    场景5

    基本与场景4相同,不同之处在于调用ngx_http_upstream_init_round_robin方法时,走隐式upstream部分的处理逻辑。

    场景6

    与场景2相同。

    场景7

    与场景1相同。

    场景8

    实际上是隐式创建了upstream,但是因为proxy_pass中指定了IP和端口号,所以ngx_http_upstream_init_round_robin初始化upstream时,us->servers不为空,所以走该函数的上半部分逻辑。与场景1有些类似。

    相关文章

      网友评论

        本文标题:详解proxy_pass、upstream与resolver

        本文链接:https://www.haomeiwen.com/subject/etwksftx.html