美文网首页大数据 爬虫Python AI SqlPython小哥哥
Python爬虫使用selenium爬取qq群的成员(全自动实现

Python爬虫使用selenium爬取qq群的成员(全自动实现

作者: 14e61d025165 | 来源:发表于2019-04-16 15:17 被阅读0次

    1. 效果图,其中涉及一些真名我就打码了,还有qq号我也打码了,见谅

    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962650 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image> <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962656 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    如果你在学习Python的过程当中有遇见任何问题,可以加入我的python交流学习qq群:683380553,多多交流问题,互帮互助,群里有不错的学习教程和开发工具。学习python有任何问题(学习方法,学习效率,如何就业),可以随时来咨询我,如果你准备学习大数据,也欢迎加入大数据学习交流qq群683380553,每天与大家分享学习资源哦。

    2. 解析,通过访问qq群官网并登陆

    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962660 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    3. 分析登陆的元素,下图一目了然,怎么获取这个登陆元素应该都知道了

    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962666 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    4. 代码奉上

    <pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">url = 'https://qun.qq.com/'# 构建谷歌驱动器browser = webdriver.Chrome()# 请求urlbrowser.get(url)# 模拟登陆,首先找到登陆的id,并点击browser.find_element_by_css_selector('#headerInfo p a').click()
    </pre>

    5. 点击之后出现这么一个框框(这个框框可把我折磨的阿)原因是这样的,寻常的获取这个框框是不能获取到的

    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962669 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    6. 先看看这个框所在的位置,这个框框竟然在另一个html代码里面,也就是说在浏览器看的时候,出现了两个html标签,老实说,我是第一次看到这种情况的,奈何我的html也不好,连入门都算不上,没办法,我就去百度了,果然黄天不负有心人,说是因为iframe这个标签可以再放html代码,所以就是这种情况了

    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962672 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    1. 既然知道了是怎么一回事之后,那就可以继续操作了,首先我们先找到iframe这个标签,然后获取它的src属性,这个链接就是这个框框登陆的链接了,如果不获取这个iframe标签的src属性,那么我们使用selenium是获取不到这个框框的元素的。
    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962677 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    <pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"># 点击之后会弹出一个登陆框,这时候我们用显示等待来等待这个登陆框加载出来WebDriverWait(browser, 1000).until( EC.presence_of_all_elements_located( (By.CSS_SELECTOR, '#loginWin iframe') ))print('登陆框已加载')# 登陆框加载之后,我们发现整个登陆框其实就是另一个网网页# 如果在原网页操作这个登陆框的话,是不能操作的# 所以我们只需要提取iframe标签的src属性,然后再去访问这个url即可实现# 自动登陆# 找到iframe标签并获取srciframe_url = browser.find_element_by_css_selector('#loginWin iframe').get_attribute('src')# 再访问这个urlbrowser.get(iframe_url)# 找到快捷登陆的头像并点击# 首先用显示等待这个头像已经加载完成WebDriverWait(browser, 1000).until( EC.presence_of_all_elements_located( (By.ID, 'qlogin_list') ))browser.find_element_by_css_selector('#qlogin_list a').click()print('登陆成功')
    </pre>

    8.登陆成功之后我们需要的是群管理,是ul标签的第四个li标签,通过xpath获取

    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962682 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    <pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"># 登陆成功之后,我们就找到群管理的标签并点击,首先等待这个元素加载完成WebDriverWait(browser, 1000).until( EC.presence_of_all_elements_located( (By.XPATH, './/ul[@id="headerNav"]/li[4]') ))browser.find_element_by_xpath('.//ul[@id="headerNav"]/li[4]').click()
    </pre>

    9. 点击群管理之后,进入群管理界面,我们需要的是成员管理

    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962684 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    <pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"># 点击之后,我们找到成员管理标签并点击WebDriverWait(browser, 1000).until( EC.presence_of_all_elements_located( (By.CLASS_NAME, 'color-tit') ))browser.find_element_by_class_name('color-tit').click()
    </pre>

    10. 点击成员管理之后会重新新建一个窗口,这个时候就会出现句柄,我们需要将当然窗口的句柄切换到新打开的这个界面,不然的话,是获取不到新打开界面的信息的,注释已经写了

    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962691 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    <pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"># 打印全部窗口句柄# print(browser.window_handles)# 打印当前窗口句柄# print(browser.current_window_handle)# 注意这里点击成员管理之后会自动跳转到一个新窗口打开这个页面# 所以我们需要将窗口句柄切换到这个新窗口
    browser.switch_to.window(browser.window_handles[1])

    解释一下browser.switch_to.window是获取当前一共有几个窗口# 这里是2个# browser.switch_to.window这个是指定当前游标切换到哪个窗口# 其实也可以这么写# all_window = browser.switch_to.window返回的是一个列表# browser.switch_to.window(all_window[1])# 效果是一样的

    </pre>

    11. 我们需要的是我加入的群信息

    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962696 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    <pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"># 切换句柄之后,我们显示等待窗口出来 WebDriverWait(browser, 1000).until( EC.presence_of_all_elements_located( (By.CLASS_NAME, 'my-all-group') ) )
    # 筛选出我加入的群标签 lis = browser.find_elements_by_xpath('.//div[@class="my-all-group"]/ul[2]/li')
    </pre>

    12. 遍历列表,取出信息

    <pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"># 遍历num= 0while True: if num == len(lis): break try: # 按顺序选择群并获取信息 # 先点击该群获取成员信息 lis[num].click() # 显示等待信息加载完成 WebDriverWait(browser, 1000).until( EC.presence_of_all_elements_located( (By.CLASS_NAME, 'list') ) ) # 获取该群当前有多少人,后面翻页需要 groupMemberNum = eval(browser.find_element_by_id('groupMemberNum').text) # 每一次翻页都会刷新21条信息,所以写个循环 # 这里加1是因为假如一个群有36人,那么count=1,如果循环的话就不会翻页了 # 也就是只能抓到一页的数据,大家可以自己想想其中的流程就知道了 count = groupMemberNum // 21 + 1 # 这里我只爬取每个群的一部分,如果想爬取全部成员信息 # 请注释下面的if语句 if count > 2: count = 1 # 每次循环都进行翻页 # while count: # count -= 1 # # browser.execute_script('document.documentElement.scrollTop=100000') # time.sleep(2) time.sleep(2) # 开始获取成员信息 trs = browser.find_elements_by_class_name('mb') if trs: # 遍历 for tr in trs: tds = tr.find_elements_by_tag_name('td')[2:] if len(tds) == 8: # qq网名 qq_name = tds[0].text # 群名称 group_name = tds[1].text # qq号 qq_number = tds[2].text # 性别 gender = tds[3].text # qq年龄 qq_year = tds[4].text # 入群时间 join_time = tds[5].text # 等级(积分) level = None # 最后发言时间 end_time = tds[6].text
    # 声明一个字典存储数据 data_dict = {} data_dict['qq_name'] = qq_name data_dict['group_name'] = group_name data_dict['qq_number'] = qq_number data_dict['gender'] = gender data_dict['qq_year'] = qq_year data_dict['join_time'] = join_time data_dict['level'] = level data_dict['end_time'] = end_time
    print(data_dict) elif len(tds) == 9: # qq网名 qq_name = tds[0].text # 群名称 group_name = tds[1].text # qq号 qq_number = tds[2].text # 性别 gender = tds[3].text # qq年龄 qq_year = tds[4].text # 入群时间 join_time = tds[5].text # 等级(积分) level = tds[6].text # 最后发言时间 end_time = tds[7].text
    # 声明一个字典存储数据 data_dict = {} data_dict['qq_name'] = qq_name data_dict['group_name'] = group_name data_dict['qq_number'] = qq_number data_dict['gender'] = gender data_dict['qq_year'] = qq_year data_dict['join_time'] = join_time data_dict['level'] = level data_dict['end_time'] = end_time data_list.append(data_dict)
    print(data_dict)
    browser.find_element_by_id('changeGroup').click() time.sleep(3) WebDriverWait(browser, 1000).until( EC.presence_of_all_elements_located( (By.CLASS_NAME, 'ui-dialog') ) ) lis = browser.find_elements_by_xpath('.//div[@class="my-all-group"]/ul[2]/li') num += 1 except Exception as e: lis = browser.find_elements_by_xpath('.//div[@class="my-all-group"]/ul[2]/li') num += 1 continue
    </pre>

    13. 一些需要说明的思路,num是判断是否爬完这些群,如果爬完了,那就退出循环

    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962705 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image> <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962709 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    14. 因为lis是每个群,所以lis[num].click()是点击当前的群,进入查看群成员信息

    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962713 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    15. 最后需要注意的就是当我们爬取当前群的成员信息之后,怎么切换到下一个群

    <tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1555398962717 ql-align-center" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; text-align: left; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;"> image

    <input class="pgc-img-caption-ipt" placeholder="图片描述(最多50字)" value="" style="box-sizing: border-box; outline: 0px; color: rgb(102, 102, 102); position: absolute; left: 187.5px; transform: translateX(-50%); padding: 6px 7px; max-width: 100%; width: 375px; text-align: center; cursor: text; font-size: 12px; line-height: 1.5; background-color: rgb(255, 255, 255); background-image: none; border: 0px solid rgb(217, 217, 217); border-radius: 4px; transition: all 0.2s cubic-bezier(0.645, 0.045, 0.355, 1) 0s;"></tt-image>

    16. 完整代码附上

    <pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"># 导入需要的包# 爬取qq群的成员信息from selenium import webdriverfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.common.by import Byimport timeimport jsonimport csv

    开始登陆def login_spider():

    url = 'https://qun.qq.com/'    # 构建谷歌驱动器    browser = webdriver.Chrome()    # 请求url    browser.get(url)    # 模拟登陆,首先找到登陆的id,并点击    browser.find_element_by_css_selector('#headerInfo p a').click()    # 点击之后会弹出一个登陆框,这时候我们用显示等待来等待这个登陆框加载出来    WebDriverWait(browser, 1000).until(        EC.presence_of_all_elements_located(            (By.CSS_SELECTOR, '#loginWin iframe')        )    )    print('登陆框已加载')    # 登陆框加载之后,我们发现整个登陆框其实就是另一个网网页    # 如果在原网页操作这个登陆框的话,是不能操作的    # 所以我们只需要提取iframe标签的src属性,然后再去访问这个url即可实现    # 自动登陆    # 找到iframe标签并获取是如此熟悉    iframe_url = browser.find_element_by_css_selector('#loginWin iframe').get_attribute('src')    # 再访问这个url    browser.get(iframe_url)    # 找到快捷登陆的头像并点击    # 首先用显示等待这个头像已经加载完成    WebDriverWait(browser, 1000).until(        EC.presence_of_all_elements_located(            (By.ID, 'qlogin_list')        )    )    browser.find_element_by_css_selector('#qlogin_list a').click()    print('登陆成功')
    return browser
    

    切换句柄操作def switch_spider(browser): # 登陆成功之后,我们就找到群管理的标签并点击,首先等待这个元素加载完成 WebDriverWait(browser, 1000).until( EC.presence_of_all_elements_located( (By.XPATH, './/ul[@id="headerNav"]/li[4]') ) ) browser.find_element_by_xpath('.//ul[@id="headerNav"]/li[4]').click() # 点击之后,我们找到成员管理标签并点击 WebDriverWait(browser, 1000).until( EC.presence_of_all_elements_located( (By.CLASS_NAME, 'color-tit') ) ) browser.find_element_by_class_name('color-tit').click() # 打印全部窗口句柄 # print(browser.window_handles) # 打印当前窗口句柄 # print(browser.current_window_handle) # 注意这里点击成员管理之后会自动跳转到一个新窗口打开这个页面 # 所以我们需要将窗口句柄切换到这个新窗口 browser.switch_to.window(browser.window_handles[1]) # 解释一下browser.switch_to.window是获取当前一共有几个窗口 # 这里是2个 # browser.switch_to.window这个是指定当前游标切换到哪个窗口 # 其实也可以这么写 # all_window = browser.switch_to.window返回的是一个列表 # browser.switch_to.window(all_window[1]) # 效果是一样的

    return browser
    

    开始采集数据def start_spider(browser): # 声明一个列表存储字典 data_list = [] # 切换句柄之后,我们显示等待窗口出来 WebDriverWait(browser, 1000).until( EC.presence_of_all_elements_located( (By.CLASS_NAME, 'my-all-group') ) )

    # 筛选出我加入的群标签    lis = browser.find_elements_by_xpath('.//div[@class="my-all-group"]/ul[2]/li')    # 遍历    num = 0    while True:        try:            # 按顺序选择群并获取信息            # 先点击该群获取成员信息            lis[num].click()            # 显示等待信息加载完成            WebDriverWait(browser, 1000).until(                EC.presence_of_all_elements_located(                    (By.CLASS_NAME, 'list')                )            )            # 获取该群当前有多少人,后面翻页需要            groupMemberNum = eval(browser.find_element_by_id('groupMemberNum').text)            # 每一次翻页都会刷新21条信息,所以写个循环            # 这里加1是因为假如一个群有36人,那么count=1,如果循环的话就不会翻页了            # 也就是只能抓到一页的数据,大家可以自己想想其中的流程就知道了            count = groupMemberNum // 21 + 1            # 这里我只爬取每个群的一部分,如果想爬取全部成员信息            # 请注释下面的if语句            if count > 5:                count = 5            # 每次循环都进行翻页            while count:                count -= 1
                browser.execute_script('document.documentElement.scrollTop=100000')                time.sleep(2)            time.sleep(3)            # 开始获取成员信息            trs = browser.find_elements_by_class_name('mb')            if trs:                # 遍历                for tr in trs:                    tds = tr.find_elements_by_tag_name('td')[2:]                    if len(tds) == 8:                        # qq网名                        qq_name = tds[0].text                        # 群名称                        group_name = tds[1].text                        # qq号                        qq_number = tds[2].text                        # 性别                        gender = tds[3].text                        # qq年龄                        qq_year = tds[4].text                        # 入群时间                        join_time = tds[5].text                        # 等级(积分)                        level = None                        # 最后发言时间                        end_time = tds[6].text
                        # 声明一个字典存储数据                        data_dict = {}                        data_dict['qq_name'] = qq_name                        data_dict['group_name'] = group_name                        data_dict['qq_number'] = qq_number                        data_dict['gender'] = gender                        data_dict['qq_year'] = qq_year                        data_dict['join_time'] = join_time                        data_dict['level'] = level                        data_dict['end_time'] = end_time
                        print(data_dict)                    elif len(tds) == 9:                        # qq网名                        qq_name = tds[0].text                        # 群名称                        group_name = tds[1].text                        # qq号                        qq_number = tds[2].text                        # 性别                        gender = tds[3].text                        # qq年龄                        qq_year = tds[4].text                        # 入群时间                        join_time = tds[5].text                        # 等级(积分)                        level = tds[6].text                        # 最后发言时间                        end_time = tds[7].text
                        # 声明一个字典存储数据                        data_dict = {}                        data_dict['qq_name'] = qq_name                        data_dict['group_name'] = group_name                        data_dict['qq_number'] = qq_number                        data_dict['gender'] = gender                        data_dict['qq_year'] = qq_year                        data_dict['join_time'] = join_time                        data_dict['level'] = level                        data_dict['end_time'] = end_time                        data_list.append(data_dict)
                        print(data_dict)
            browser.find_element_by_id('changeGroup').click()            time.sleep(3)            WebDriverWait(browser, 1000).until(                EC.presence_of_all_elements_located(                    (By.CLASS_NAME, 'ui-dialog')                )            )            lis = browser.find_elements_by_xpath('.//div[@class="my-all-group"]/ul[2]/li')            num += 1        except Exception as e:            continue
    return data_list
    

    def main():
    browser = login_spider() browser = switch_spider(browser) data_list = start_spider(browser)
    # 将数据写入json文件 with open('data_json.json', 'a+', encoding='utf-8') as f: json.dump(data_list, f) print('json文件写入完成') # 这里的编码格式不要写错了,不然会出现乱码,因为群里面的大神名字贼骚 with open('data_csv.csv', 'w', encoding='utf-8-sig', newline='') as f: # 表头 title = data_list[0].keys() # 声明writer writer = csv.DictWriter(f, title) # 写入表头 writer.writeheader() # 批量写入数据 writer.writerows(data_list) print('csv文件写入完成')
    if name == 'main':
    main()
    </pre>

    相关文章

      网友评论

        本文标题:Python爬虫使用selenium爬取qq群的成员(全自动实现

        本文链接:https://www.haomeiwen.com/subject/kdpbwqtx.html