curl的奇特问题

最近在使用curl的时候发现一个奇怪的问题。使用curl访问sftp服务器时总是报错。我执行的命令如下:

curl -u sftptest:passwd sftp://127.0.0.1:22/home/sftptest/

然后返回了一大篇html, 主要的信息如下:

...
<title>ERROR: The requested URL could not be retrieved</title>
...
<p><b>Unsupported Request Method and Protocol</b></p>

首先需要确认的是sftp是否在curl中开启了。执行curl -V发现确实支持curl. 接下来打开verbose模式，发现了一些有趣的信息:

*   Trying 172.17.10.80...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to (nil) (172.17.18.84) port 8080 (#0)
* Server auth using Basic with user 'sftptest'
> GET sftp://sftptest:password@127.0.0.1/home/sftptest HTTP/1.1
> Host: 127.0.0.1:22
> Authorization: Basic c2Z0cHRlc3Q6MXEydzNlNHI=
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.52.1 OpenSSL/1.1.0e libssh2/1.7.0_DEV
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 400 Bad Request
< Server: nps/2.3.1
< Mime-Version: 1.0
< Date: Wed, 25 Oct 2017 10:03:23 GMT
< Content-Type: text/html
< Content-Length: 4205
< X-Squid-Error: ERR_INVALID_URL 0
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from netentsec-nps-172.17.18.84
< Connection: close
< 
{ [data not shown]
* Curl_http_done: called premature == 0

100  4205  100  4205    0     0  29450      0 --:--:-- --:--:-- --:--:-- 29822
* Closing connection 0

有意思的是，curl 连接上我的代理服务器后，使用http协议去访问sftp://127.0.0.1. 所以我们会得到Bad request (400)的响应。

如果我把127.0.0..1从代理中排除(e.g. export no_proxy=127.0.0.1)则请求就可以成功了。

挖掘真相

为什么在连接代理后，curl会使用http协议去访问sftp服务器呢？ 我在curl的repository中创建了issue. 从官方的回答中得到了答案。因为我正在使用老版本的curl(7.52), 这个版本不支持自动转换隧道协议(tunneling). 在7.55.0版本后才支持这种自动转换。摘自issue的回复

Since 7.55.0, curl will enable tunneling automatically when you try to use SFTP over an HTTP proxy.

所以如果使用最新的curl, 这个问题应该就能解决了。接下来尝试下载和编译了最新的curl(7.56), 但是不幸的是仍然不能成功，并且返回了另一个错误:

* STATE: INIT => CONNECT handle 0x1a42ec8; line 1425 (connection #-5000)
* Added connection 0. The cache now contains 1 members
*   Trying 172.17.18.84...
* TCP_NODELAY set
* STATE: CONNECT => WAITCONNECT handle 0x1a42ec8; line 1477 (connection #0)
* Connected to 172.17.18.84 (172.17.18.84) port 8080 (#0)
* STATE: WAITCONNECT => WAITPROXYCONNECT handle 0x1a42ec8; line 1594 (connection #0)
* Marked for [keep alive]: HTTP default
* allocate connect buffer!
* Establish HTTP proxy tunnel to 127.0.0.1:22
* Server auth using Basic with user 'sftptest'
> CONNECT 127.0.0.1:22 HTTP/1.1
> Host: 127.0.0.1:22
> User-Agent: curl/7.57.0-DEV
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 503 Service Unavailable
< Server: nps/2.3.1
< Mime-Version: 1.0
< Date: Thu, 26 Oct 2017 07:09:25 GMT
< Content-Type: text/html
< Content-Length: 4069
< X-Squid-Error: ERR_CONNECT_FAIL 111
< Vary: Accept-Language
< Content-Language: en

注意到新的错误是503 Service Unavailable. 为什么会这样呢？我们可以一步一步分析原因。

首先，可以轻易发现连接代理服务器是成功的:

*   Trying 172.17.18.84...
* TCP_NODELAY set
* STATE: CONNECT => WAITCONNECT handle 0x1a42ec8; line 1477 (connection #0)
* Connected to 172.17.18.84 (172.17.18.84) port 8080 (#0)
* STATE: WAITCONNECT => WAITPROXYCONNECT handle 0x1a42ec8; line 1594 (connection #0)
* Marked for [keep alive]: HTTP default
* allocate connect buffer!
* Establish HTTP proxy tunnel to 127.0.0.1:22

Tcp连接的状态变化为: CONNECT => WAITCONNECT => WAITPROXYCONNECT => Establish HTTP proxy tunnel. 这个状态变化显然是正确的，代表连接代理服务器成功。

接下来尝试连接sftp服务器:

* Server auth using Basic with user 'sftptest'
> CONNECT 127.0.0.1:22 HTTP/1.1
> Host: 127.0.0.1:22
> User-Agent: curl/7.57.0-DEV
> Proxy-Connection: Keep-Alive

但是得到的响应却是:

< HTTP/1.1 503 Service Unavailable

也就是说，从代理服务器发起sftp连接到sftp服务器失败 - 并且是sftp服务器服务不可用。这下真相逐渐浮出水面: 因为代理服务器尝试去连接127.0.0.1:22, 但是这个地址对于代理服务器而言，是连接自己的22端口，而不是去连接sftp服务器！如果把这里的127.0.0.1替换为sftp服务器的内网地址，则可以成功:

* STATE: INIT => CONNECT handle 0x2238ec8; line 1425 (connection #-5000)
* Rebuilt URL to: sftp://sftptest@172.26.9.94/
* Added connection 0. The cache now contains 1 members
*   Trying 172.17.18.84...
* TCP_NODELAY set
* STATE: CONNECT => WAITCONNECT handle 0x2238ec8; line 1477 (connection #0)
* Connected to 172.17.18.84 (172.17.18.84) port 8080 (#0)
* STATE: WAITCONNECT => WAITPROXYCONNECT handle 0x2238ec8; line 1594 (connection #0)
* Marked for [keep alive]: HTTP default
* allocate connect buffer!
* Establish HTTP proxy tunnel to 172.26.9.94:22
* Server auth using Basic with user 'sftptest'
> CONNECT 172.26.9.94:22 HTTP/1.1
> Host: 172.26.9.94:22
> User-Agent: curl/7.57.0-DEV
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* STATE: WAITPROXYCONNECT => SENDPROTOCONNECT handle 0x2238ec8; line 1573 (connection #0)
* CONNECT phase completed!
* SFTP 0x223f170 state change from SSH_STOP to SSH_INIT
* SFTP 0x223f170 state change from SSH_INIT to SSH_S_STARTUP
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x2238ec8; line 1608 (connection #0)
* SFTP 0x223f170 state change from SSH_S_STARTUP to SSH_HOSTKEY
* SSH MD5 fingerprint: 4cf03683e054a3398c91d76a16715e6b
* SSH host check: 2, key: <none>
* SFTP 0x223f170 state change from SSH_HOSTKEY to SSH_SESSION_FREE
* Marked for [closure]: SSH session free
* SFTP 0x223f170 state change from SSH_SESSION_FREE to SSH_STOP
* multi_done
* SSH DISCONNECT starts now
* SSH DISCONNECT is done
* Closing connection 0
* The cache now contains 0 members

结论

要解决curl使用sftp访问127.0.0.1或localhost的问题，可以采用:

• 方法1 （推荐）: 把127.0.0.1或localhost加入no_proxy. 这样无论使用哪个版本的curl, 都能正常工作。
• 方法2: 目标地址使用内网ip地址，而不要使用127.0.0.1, 并且保证curl的版本在7.55以上。 但是这样做的话，curl会先连接代理服务器，再发起sftp请求，性能会有损耗。