美文网首页学习
可能是最全的User-Agent列表

可能是最全的User-Agent列表

作者: SeanCheney | 来源:发表于2019-08-19 17:31 被阅读0次

    User-Aagent String这个网站收集了爬虫、浏览器、主机、邮件客户端等客户端的请求头的UA字段。其中浏览器的UA头有将近10000个。但是访问这个网站很慢。

    http://useragentstring.com/pages/useragentstring.php

    把浏览器的所有UA头下载下来,字符串长度小于80的丢弃,存成csv文件,得到6244条:

    # -*- coding: utf-8 -*-
    
    import requests
    import pandas as pd
    from lxml import etree
    
    url = 'http://useragentstring.com/pages/useragentstring.php?typ=Browser'
    
    header = {
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
            'User-Agent': 'Mozilla/5.0 (compatible; ABrowse 0.4; Syllable)'
        }
    
    response = requests.get(url,
                            headers=header,
                            timeout=60
                            )
    
    tree = etree.HTML(response.text)
    browsers = tree.xpath('//ul/li/a/text()')
    browsers = [browser for browser in browsers if len(browser)>80]
    
    print(browsers)
    print(len(browsers))
    
    df = pd.DataFrame({'id':range(1,len(browsers)+1), 'ua':browsers})
    print(df)
    
    df.to_csv('user_agents.csv', index=False)
    
    

    这个网站还提供了一个验证ua的接口,比较实用:

    http://www.useragentstring.com/?uas=Opera/9.70%20(Linux%20i686%20;%20U;%20en-us)%20Presto/2.2.0&getJSON=all

    下面是部分的,完整的太长,超出了文章长度限制:

    UA_LIST = ['Mozilla/5.0 (compatible; U; ABrowse 0.6; Syllable) AppleWebKit/420+ (KHTML, like Gecko)', 'Mozilla/5.0 (compatible; U; ABrowse 0.6;  Syllable) AppleWebKit/420+ (KHTML, like Gecko)', 'Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Acoo Browser 1.98.744; .NET CLR 3.5.30729)', 'Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Acoo Browser 1.98.744; .NET CLR   3.5.30729)', 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0;   Acoo Browser; GTB5; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;   SV1) ; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)', 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; SV1; Acoo Browser; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; Avant Browser)', 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1;   .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)', 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; GTB5; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Maxthon; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)', 'Mozilla/4.0 (compatible; Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Acoo Browser 1.98.744; .NET CLR 3.5.30729); Windows NT 5.1; Trident/4.0)', 'Mozilla/4.0 (compatible; Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; Acoo Browser; .NET CLR 1.1.4322; .NET CLR 2.0.50727); Windows NT 5.1; Trident/4.0; Maxthon; .NET CLR 2.0.50727; .NET CLR 1.1.4322; InfoPath.2)', 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Acoo Browser; GTB6; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)']
    

    相关文章

      网友评论

        本文标题:可能是最全的User-Agent列表

        本文链接:https://www.haomeiwen.com/subject/pekesctx.html