利用Pycurl库监控WEB访问质量（二）

作者: iDefeng | 来源:发表于2016-12-13 14:06 被阅读178次

上一篇文章使用urllib2来监控WEB访问质量（http://www.jianshu.com/p/c74ddabd5212），理论上来说已经可以满足要求，不过其很多性能指标都是我们自己计算出来的，比如响应时间。而Pycurl却能够得到整个HTTP连接过程的所需数据，比如HTTP Code,DNS解析时间，完整响应时间等。
Pycurl地址：https://pypi.python.org/pypi/pycurl/ 或者：http://pycurl.io/docs/latest/
PyCurl的常用方法（参考：http://www.jianshu.com/p/53e9082633e8）：

c = pycurl.Curl() #创建一个curl对象
c.setopt(pycurl.CONNECTTIMEOUT, 5) #连接的等待时间，设置为0则不等待
c.setopt(pycurl.TIMEOUT, 5) #请求超时时间
c.setopt(pycurl.NOPROGRESS, 0) #是否屏蔽下载进度条，非0则屏蔽
c.setopt(pycurl.MAXREDIRS, 5) #指定HTTP重定向的最大数
c.setopt(pycurl.FORBID_REUSE, 1) #完成交互后强制断开连接，不重用
c.setopt(pycurl.FRESH_CONNECT,1) #强制获取新的连接，即替代缓存中的连接
c.setopt(pycurl.DNS_CACHE_TIMEOUT,60) #设置保存DNS信息的时间，默认为120秒
c.setopt(pycurl.URL,"[http://www.baidu.com](http://www.baidu.com)") #指定请求的URL
c.setopt(pycurl.USERAGENT,"Mozilla/5.2 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50324)") #配置请求HTTP头的User-Agent
c.setopt(pycurl.HEADERFUNCTION, getheader) #将返回的HTTP HEADER定向到回调函数getheaderc.setopt(pycurl.WRITEFUNCTION, getbody) #将返回的内容定向到回调函数getbody
c.setopt(pycurl.WRITEHEADER, fileobj) #将返回的HTTP HEADER定向到fileobj文件对象
c.setopt(pycurl.WRITEDATA, fileobj) #将返回的HTML内容定向到fileobj文件对象
c = pycurl.Curl() #创建一个curl对象c.getinfo(pycurl.HTTP_CODE) #返回的HTTP状态码
c.getinfo(pycurl.TOTAL_TIME) #传输结束所消耗的总时间
c.getinfo(pycurl.NAMELOOKUP_TIME) #DNS解析所消耗的时间
c.getinfo(pycurl.CONNECT_TIME) #建立连接所消耗的时间
c.getinfo(pycurl.PRETRANSFER_TIME) #从建立连接到准备传输所消耗的时间
c.getinfo(pycurl.STARTTRANSFER_TIME) #从建立连接到传输开始消耗的时间
c.getinfo(pycurl.REDIRECT_TIME) #重定向所消耗的时间
c.getinfo(pycurl.SIZE_UPLOAD) #上传数据包大小
c.getinfo(pycurl.SIZE_DOWNLOAD) #下载数据包大小
c.getinfo(pycurl.SPEED_DOWNLOAD) #平均下载速度
c.getinfo(pycurl.SPEED_UPLOAD) #平均上传速度
c.getinfo(pycurl.HEADER_SIZE) #HTTP头部大小

有了上面的信息，我们就可以利用pycurl来监控我们的WEB站点了，目标了上一篇一致，得到HTTP CODE（HTTP_CODE）和响应时间（TOTAL_TIME）。另外，我们的发邮件模块中，进行了优化，让发出来的邮件更漂亮。代码如下：

# coding: utf-8
import pycurl
try:
    from io import BytesIO
except ImportError:
    from StringIO import StringIO as BytesIO
from email.header import Header
from email.mime.text import MIMEText
from email.utils import parseaddr, formataddr
import smtplib


def _format_addr(s):
    name, addr = parseaddr(s)
    return formataddr(( \
        Header(name, 'utf-8').encode(), \
        addr.encode("utf-8") if isinstance(addr, unicode) else addr))


from_addr = "XXX"
to_addr = "XXX"  # 发送给多个用户，使用逗号分隔
password = "XXX"
smtp_server = "smtp.XXX.com"


def format_msg(s):
    # type: (object) -> object
    msg = MIMEText(s, _subtype='html', _charset='utf-8')
    msg['From'] = _format_addr(from_addr)
    msg['To'] = _format_addr(to_addr)
    msg['Subject'] = Header(u'来自XXX的监控小黑屋', 'utf-8').encode()
    return msg.as_string()


def send_email(s):
    server = smtplib.SMTP(smtp_server, 25)
    server.set_debuglevel(0)
    server.login(from_addr, password)
    server.sendmail(from_addr, ["zhangdf@etlchina.net"], s)
    server.quit()

flag = False    # 决定是否发报警邮件的标志
list_of_sites = []
result_of_access = '''
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>XXX监控报告邮件</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0 " />
    <meta name="format-detection" content="telephone=no" />
    <!--[if !mso]><!-->
    <link href="https://fonts.googleapis.com/css?family=Open+Sans:300,400,600,700,800" rel="stylesheet">
    <!--<![endif]-->
    <style type="text/css">
      body {
      -webkit-text-size-adjust: 100% !important;
      -ms-text-size-adjust: 100% !important;
      -webkit-font-smoothing: antialiased !important;
      }
      img {
      border: 0 !important;
      outline: none !important;
      }
      p {
      Margin: 0px !important;
      Padding: 0px !important;
      }
      table {
      border-collapse: collapse;
      mso-table-lspace: 0px;
      mso-table-rspace: 0px;
      }
      td, a, span {
      border-collapse: collapse;
      mso-line-height-rule: exactly;
      }
      .ExternalClass * {
      line-height: 100%;
      }
      span.MsoHyperlink {
      mso-style-priority:99;
      color:inherit;}
      span.MsoHyperlinkFollowed {
      mso-style-priority:99;
      color:inherit;}
      @media only screen and (min-width:481px) and (max-width:599px) {
      table[class=em_main_table] {
      width: 100% !important;
      }
      table[class=em_wrapper] {
      width: 100% !important;
      }
      td[class=em_hide], br[class=em_hide] {
      display: none !important;
      }
      img[class=em_full_img] {
      width: 100% !important;
      height: auto !important;
      }
      td[class=em_align_cent] {
      text-align: center !important;
      }
      td[class=em_pad_top]{
      padding-top:20px !important;
      }
      td[class=em_aside]{
      padding-left:10px !important;
      padding-right:10px !important;
      }
      td[class=em_height]{
      height: 20px !important;
      }
      td[class=em_space]{
      width:10px !important;
      }
      td[class=em_width55] {
      width:80px !important;
      text-align:center !important;
      }
      td[class=em_width75] {
      width:100px !important;
      }
      td[class=em_font]{
      font-size:14px !important;
      }
      td[class=em_font2] {
      text-align:center !important;
      }
      td[class=em_align_cent1] {
      text-align: center !important;
      padding-bottom: 10px !important;
      }
      }
      @media only screen and (max-width:480px) {
      table[class=em_main_table] {
      width: 100% !important;
      }
      table[class=em_wrapper] {
      width: 100% !important;
      }
      td[class=em_hide], br[class=em_hide], span[class=em_hide] {
      display: none !important;
      }
      img[class=em_full_img] {
      width: 100% !important;
      height: auto !important;
      }
      td[class=em_align_cent] {
      text-align: center !important;
      }
      td[class=em_pad_top]{
      padding-top:20px !important;
      }
      td[class=em_height]{
      height: 20px !important;
      }
      td[class=em_aside]{
      padding-left:10px !important;
      padding-right:10px !important;
      }
      td[class=em_font]{
      font-size:14px !important;
      line-height:28px !important;
      }
      td[class=em_font1]{
      font-size:14px !important;
      line-height:18px !important;
      }
      td[class=em_font2]{
      font-size:14px !important;
      line-height:18px !important;
      text-align:center !important;
      }
      td[class=em_space]{
      width:10px !important;
      }
      span[class=em_br]{
      display:block !important;
      }
      td[class=em_width55] {
      width:55px !important;
      font-size:15px !important;
      line-height:19px !important;
      text-align:center !important;
      }
      td[class=em_width75] {
      width:75px !important;
      font-size:15px !important;
      line-height:19px !important;
      }
      td[class=em_align_cent1] {
      text-align: center !important;
      padding-bottom: 10px !important;
      }
      }
    </style>
  </head>
  <body style="margin:0px; padding:0px;" bgcolor="#ffffff">
    <table width="100%" border="0" cellspacing="0" cellpadding="0" bgcolor="#ffffff">
      <!-- BODY  -->
      <tr>
        <td align="center" valign="top"  bgcolor="#ffffff">
          <table width="600" cellpadding="0" cellspacing="0" border="0" align="center" class="em_main_table" style="table-layout:fixed;">
            <!-- === LOGO SECTION === -->
            <tr>
              <td height="40" class="em_height"> </td>
            </tr>
            <tr>
              <td align="center"><a href="#" target="_blank" style="text-decoration:none;">![LoGo Here](http:https://img.haomeiwen.com/i3900471/db3d2ef41db98dc2.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)</a></td>
            </tr>
            <tr>
              <td height="30" class="em_height"> </td>
            </tr>
            <!-- === //LOGO SECTION === -->
            <!-- === NEVIGATION SECTION === -->
            <tr>
              <td height="1" bgcolor="#fed69c" style="font-size:0px; line-height:0px;">![](https://www.sendwithus.com/assets/img/emailmonks/images/spacer.gif)</td>
            </tr>
            <tr>
              <td height="1" bgcolor="#fed69c" style="font-size:0px; line-height:0px;">![](https://www.sendwithus.com/assets/img/emailmonks/images/spacer.gif)</td>
            </tr>
            <!-- === //NEVIGATION SECTION === -->
            <!-- THANK YOU SECTION -->
            <tr>
              <td valign="top" class="em_aside">
                <table width="100%" border="0" cellspacing="0" cellpadding="0">
                  <tr>
                    <td height="27" class="em_height"> </td>
                  </tr>
                  <tr>
                    <td valign="top" align="center" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px;line-height:22px; font-weight:bold; color:#30373b; text-transform:uppercase;">XX网站监控信息记录</td>
                  </tr>
                  <tr>
                    <td height="23" class="em_height"> </td>
                  </tr>
                  <tr>
                    <td valign="top">
                      <table width="100%" border="0" cellspacing="0" cellpadding="0">
                        <tr>
                          <td valign="top">
                            <table width="290" border="0" cellspacing="0" cellpadding="0" align="left" class="em_wrapper" bgcolor="#f6f7f8">
                              <tr>
                                <td align="center" valign="middle" class="em_font1" height="42" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; font-weight:bold; color:#30373b; text-transform:uppercase;">
                                  <span style="color:#feae39;">机房信息 :</span> 亦庄-XX数据中心
                                </td>
                              </tr>
                            </table>
                            <table width="290" border="0" cellspacing="0" cellpadding="0" align="right" class="em_wrapper">
                              <tr>
                                <td valign="top" class="em_pad_top">
                                  <table width="100%" border="0" cellspacing="0" cellpadding="0"  bgcolor="#f6f7f8" >
                                    <tr>
                                      <td align="center" valign="middle" class="em_font1" height="42" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; font-weight:bold; color:#30373b; text-transform:uppercase;">
                                        <span style="color:#feae39;">机房电话 :</span> 138XXXXXXXX
                                      </td>
                                    </tr>
                                  </table>
                                </td>
                              </tr>
                            </table>
                          </td>
                        </tr>
                      </table>
                    </td>
                  </tr>
                  <tr>
                    <td height="20" style="font-size:1px; line-height:1px;"> </td>
                  </tr>
                  <tr>
                    <td valign="top">
                      <table width="100%" border="0" cellspacing="0" cellpadding="0" bgcolor="#f6f7f8">
                        <tr>
                          <td width="25" class="em_space"> </td>
                          <td valign="top">
                            <table width="100%" border="0" cellspacing="0" cellpadding="0">
                              <tr>
                                <td height="12" style="font-size:1px; line-height:1px;"> </td>
                              </tr>
                              <tr>
                                <td align="left" valign="top"  class="em_font1" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:20px; font-weight:bold; color:#feae39; text-transform:uppercase;">XXX :  <span style=" color:#30373b;">136XXXXXXXX/186XXXXXXXX  |  XXX@XXX.net</span></td>
                              </tr>
                              <tr>
                                <td height="12" style="font-size:1px; line-height:1px;"> </td>
                              </tr>
                            </table>
                          </td>
                          <td width="25" class="em_space"> </td>
                        </tr>
                      </table>
                    </td>
                  </tr>
                  <tr>
                    <td height="25" class="em_height"> </td>
                  </tr>
                  <tr>
                    <td valign="top">
                      <table width="100%" border="0" cellspacing="0" cellpadding="0">
                        <tr>
                          <td valign="top">
                            <table width="100%" border="0" cellspacing="0" cellpadding="0" align="center">
                              <tr>
                                <td width="30" class="em_hide"> </td>
                                <td width="255" valign="top" align="left" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#30373b; font-weight:bold;">网站</td>
                                <td width="5"></td>
                                <td width="35" class="em_width55" valign="top" align="left" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#30373b; font-weight:bold;">Code</td>
                                <td width="138" class="em_hide"></td>
                                <td width="5"></td>
                                <td align="right" class="em_width75" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#30373b; font-weight:bold;">响应时间</td>
                                <td width="30" class="em_hide"> </td>
                              </tr>
                            </table>
                          </td>
                        </tr>
                        <tr>
                          <td height="14" style="font-size:1px; line-height:1px;"> </td>
                        </tr>
                        <tr>
                          <td valign="top">
                            <table width="100%" border="0" cellspacing="0" cellpadding="0" align="center">

'''
with open("sites.txt") as f:
    list_of_sites = f.readlines()
# print list_of_sites

num_sites = len(list_of_sites)

buffer = BytesIO()
c = pycurl.Curl()
c.setopt(c.FRESH_CONNECT, 1)

for site in list_of_sites:
    if len(site) == 0 or site.startswith("#"):
        pass
    else:
        c.setopt(c.URL, site.replace('\n', ''))
        c.setopt(c.WRITEFUNCTION, buffer.write)
        try:
            c.perform()
            http_code = c.getinfo(c.HTTP_CODE)
            total_time = c.getinfo(c.TOTAL_TIME)
            if http_code == 200:
                if total_time <= 5.0:
                    msg = """
                    <tr>
                        <td width="30" class="em_hide"> </td>
                        <td width="255" valign="top" align="left" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#30373b;"><a href="%s">%s</a></td>
                        <td width="5"></td>
                        <td width="35" class="em_width55" valign="top" align="center" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#30373b;">%s</td>
                         <td width="138" class="em_hide"></td>
                        <td width="5"></td>
                        <td align="right" valign="top" class="em_width75" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#30373b;">%s</td>
                        <td width="30" class="em_hide"> </td>
                    </tr>
                    """ % (site, site, http_code, total_time)
                    result_of_access += msg
                else:
                    msg = """
                    <tr>
                        <td width="30" class="em_hide"> </td>
                        <td width="255" valign="top" align="left" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#30373b;"><a href="%s">%s</a></td>
                        <td width="5"></td>
                        <td width="35" class="em_width55" valign="top" align="center" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#30373b;">%s</td>
                         <td width="138" class="em_hide"></td>
                        <td width="5"></td>
                        <td align="right" valign="top" class="em_width75" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#30373b;">%s</td>
                        <td width="30" class="em_hide"> </td>
                  </tr>
                    """ % (site, site, http_code, total_time)
                    result_of_access += msg
                    flag = True
            else:
                msg = """
                    <tr>
                        <td width="30" class="em_hide"> </td>
                        <td width="255" valign="top" align="left" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#30373b;"><a href="%s">%s</a></td>
                        <td width="5"></td>
                        <td width="35" class="em_width55" valign="top" align="center" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#00ff00;">%s</td>
                        <td width="138" class="em_hide"></td>
                        <td width="5"></td>
                        <td align="right" valign="top" class="em_width75" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#30373b;">%s</td>
                        <td width="30" class="em_hide"> </td>
                    </tr>

                    """ % (site, site, http_code, total_time)
                result_of_access += msg
                flag = True
        except pycurl.error, error:
            errno, errstr = error
            print errno, errstr
            msg = """
                            <tr>
                                <td width="30" class="em_hide"> </td>
                                <td width="255" valign="top" align="left" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#30373b;"><a href="%s">%s</a></td>
                                <td width="5"></td>
                                <td width="35" class="em_width55" valign="top" align="center" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#FFOOOO;">%s</td>
                                <td width="138" class="em_hide"></td>
                                <td width="5"></td>
                                <td align="right" valign="top" class="em_width75" style="font-family:'Open Sans', Arial, sans-serif; font-size:17px; line-height:24px; color:#FF0000;">%s</td>
                                <td width="30" class="em_hide"> </td>
                            </tr>

                            """ % (site, site, "Error", "Unknown")
            result_of_access += msg
            flag = True

result_of_access += """
                        </table>
                        </td>
                        </tr>
                        <tr>
                          <td height="14" style="font-size:1px; line-height:1px;"> </td>
                        </tr>
                        <tr>
                          <td height="14" style="font-size:1px; line-height:1px;"> </td>
                        </tr>
                        <tr>
                          <td height="8" class="em_hide" style="font-size:1px; line-height:1px;"> </td>
                        </tr>
                        <tr>
                          <td height="1" bgcolor="#eaebeb" style="font-size:0px;line-height:0px;">![](https://www.sendwithus.com/assets/img/emailmonks/images/spacer.gif)</td>
                        </tr>
                        <tr>
                          <td height="14" style="font-size:1px; line-height:1px;"> </td>
                        </tr>
                        <tr>
                          <td height="11" class="em_hide" style="font-size:1px; line-height:1px;"> </td>
                        </tr>
                        <tr>
                          <td valign="top">
                            <table width="100%" border="0" cellspacing="0" cellpadding="0" align="center">
                              <tr>
                                <td align="right" style="font-family:'Open Sans', Arial, sans-serif; font-size:20px; line-height:24px; color:#30373b; font-weight:bold;">总网站数:    15   个</td>
                                <td width="30" class="em_hide"> </td>
                              </tr>
                            </table>
                          </td>
                        </tr>
                      </table>
                    </td>
                  </tr>
                  <tr>
                    <td height="20" class="em_height"> </td>
                  </tr>
                </table>
              </td>
            </tr>
            <!-- //THANK YOU SECTION -->
          </table>
        </td>
      </tr>
      <!-- //BODY  -->
      <!-- === FOOTER SECTION === -->
      <tr>
        <td align="center" valign="top"  bgcolor="#30373b" class="em_aside">
          <table width="600" cellpadding="0" cellspacing="0" border="0" align="center" class="em_main_table" style="table-layout:fixed;">
            <tr>
              <td height="22" class="em_height"> </td>
            </tr>
            <tr>
              <td align="center" style="font-family:'Open Sans', Arial, sans-serif; font-size:12px; line-height:18px; color:#848789; text-transform:uppercase;">
               <span style="text-decoration:underline;"><a href="#" target="_blank" style="text-decoration:underline; color:#848789;">PRIVACY STATEMENT</a></span>   |   <span style="text-decoration:underline;"><a href="#" target="_blank" style="text-decoration:underline; color:#848789;">TERMS OF SERVICE</a></span><span class="em_hide">   |   </span><span class="em_br"></span><span style="text-decoration:underline;"><a href="#" target="_blank" style="text-decoration:underline; color:#848789;">RETURNS</a></span>
              </td>
            </tr>
            <tr>
              <td height="10" style="font-size:1px; line-height:1px;"> </td>
            </tr>
            <tr>
              <td align="center" style="font-family:'Open Sans', Arial, sans-serif; font-size:12px; line-height:18px; color:#848789;text-transform:uppercase;">
                ©2‌016XXXXXXXX. All Rights Reserved.
              </td>
            </tr>
            <tr>
              <td height="10" style="font-size:1px; line-height:1px;"> </td>
            </tr>
            <tr>
              <td align="center" style="font-family:'Open Sans', Arial, sans-serif; font-size:12px; line-height:18px; color:#848789;text-transform:uppercase;">
                If you do not wish to receive any further emails from us, please
              </td>
            </tr>
            <tr>
              <td height="25" class="em_height"> </td>
            </tr>
          </table>
        </td>
      </tr>
      <!-- === //FOOTER SECTION === -->
    </table>
    <div style="display:none; white-space:nowrap; font:20px courier; color:#ffffff; background-color:#ffffff;">                                         </div>
  </body>
</html>
"""
c.close()

msg = format_msg(result_of_access)
if flag:
    send_email(msg)

效果如下：

Paste_Image.png

注意点：

上述代码中Pycurl在windows下执行，响应很慢，即响应时间特别长，事实上并不会这么长，这个已经确认非代码问题，在linux下很快。属于pycurl的问题了。
在linux下有时会碰到无法解析域名的情况，目前还没找到原因。(这个原因已经找到，我的域名列表文件site.txt是在windows下建立好，上传到Linux上的，实际windows和Linux的默认字符编码并不一样，所以一定要注意这个问题)

利用Pycurl库监控WEB访问质量（二）

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

python

Python 运维