想找点文献提高自己，还被反爬了，Python搞起，反爬之反反爬！

作者: 14e61d025165 | 来源:发表于2019-04-23 14:21 被阅读0次

想找点文献提高自己，还被反爬了，Python搞起，反爬之反反爬！
反爬及反反爬
Python爬虫——学习字体反爬获取某招聘信息
反反爬虫之js加密参数获取
保姆级反爬教学，JS逆向实现字体反爬
java爬虫(jsoup)如何设置HTTP代理ip爬数据
Python爬虫杂记 - 字体文件反爬（二）
反爬虫到底是怎么一回事？
Python爬虫反反爬：CSS反爬加密彻底破解！
Python爬虫入门教程30岁了，想找点文献提高自己，还被反爬了

学术搜索

学习理论的知识少不了去检索文献，好多文献为你的实操提供了合理的支撑，我所在的大学内网默认是有知网账户的，非常NICE

Google学术搜索是一个文献检索服务，目前主要是提供维普资讯、万方数据等几个学术文献资源库的检索服务。通过Google学术搜索只能够查找到这些学术资料的“报告、摘要及引用内容... 来源百度百科

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1556000447094" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

可以添加QQ群1004391443，有飞机大战、颜值打分器、打砖块小游戏、红包提醒神器、小姐姐表白神器等具体的实训项目，有清晰源码，有相应的文件

我们的目标

获取现在访问的链接地址，当你使用谷歌浏览器的开发者工具抓取的时候，得到的是一个js加密函数

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1556000447096" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

注意看上图 2 的位置，接下来，我们采用上篇博客的方式，去尝试获取visit函数的具体内容

我们要在所有的请求链接中去检索一个visit方法，注意步骤

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1556000447098" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

双击方法名，进入

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1556000447101" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

找到核心方法

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">function visit(url) {
var newTab = window.open('about:blank');
if(Gword!='') url = strdecode(url);
// var newTab = window.open(url);
newTab.location.href = url;
//newTab.location.reload(true);
}
</pre>

发现url在跳转前调用了一个 strdecode 函数，你只需要关注这个函数的实现就可以了

再次查看visit的调用函数，找到参数的生成方式为

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">onclick="visit('' + autourl[b] + '')"
</pre>

对 autourl[b] 我们是可以直接用爬虫在HTML页面获取到的

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1556000447105" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">function auto(b) {
t = (tim - ts[b]) / 100;
tt = t.toString().split('.');
if(tt.length==1) t = t.toString() + '.00';
else if(tt[1].length < 2) t = t.toString() + '0';
if (t > 4) document.getElementById("txt" + b).innerHTML = '<font color=red>连接超时！</font>';
else document.getElementById("txt" + b).innerHTML = 'takes ' + t + 's. <a href="javascript:;" class="ok" onclick="visit('' + autourl[b] + '')"> 现在访问 </a>'
}
function visit(url) {
var newTab = window.open('about:blank');
if(Gword!='') url = strdecode(url);
// var newTab = window.open(url);
newTab.location.href = url;
//newTab.location.reload(true);
}
</pre>

参数分析

if(Gword!='') url = strdecode(url); 如果 Gword 为空，调用的是 strdecode 方法，查阅之后，发现相关代码也在下面

Gword 在上面的一张图片中我们也已经获取到了，可以向上看

strdecode函数分析

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Gword
//code
function strdecode(string) {
string = base64decode(string);
key = Gword+'ok ';
len = key.length;
code = '';
for (i = 0; i < string.length; i++) {
var k = i % len;
code += String.fromCharCode(string.charCodeAt(i) ^ key.charCodeAt(k))
}
return base64decode(code)
}
var base64DecodeChars = new Array(-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 62, -1, -1, -1, 63, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, -1, -1, -1, -1, -1, -1, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, -1, -1, -1, -1, -1, -1, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, -1, -1, -1, -1, -1);
function base64decode(str) {
var c1, c2, c3, c4;
var i, len, out;
len = str.length;
i = 0;
out = "";
while (i < len) {
do {
c1 = base64DecodeChars[str.charCodeAt(i++) & 0xff]
} while (i < len && c1 == -1);
if (c1 == -1) break;
do {
c2 = base64DecodeChars[str.charCodeAt(i++) & 0xff]
} while (i < len && c2 == -1);
if (c2 == -1) break;
out += String.fromCharCode((c1 << 2) | ((c2 & 0x30) >> 4));
do {
c3 = str.charCodeAt(i++) & 0xff;
if (c3 == 61) return out;
c3 = base64DecodeChars[c3]
} while (i < len && c3 == -1);
if (c3 == -1) break;
out += String.fromCharCode(((c2 & 0XF) << 4) | ((c3 & 0x3C) >> 2));
do {
c4 = str.charCodeAt(i++) & 0xff;
if (c4 == 61) return out;
c4 = base64DecodeChars[c4]
} while (i < len && c4 == -1);
if (c4 == -1) break;
out += String.fromCharCode(((c3 & 0x03) << 6) | c4)
}
return out
}
</pre>

这个地方有2个解决方案了

1是用Python重写编写相关逻辑
2通过Python调用JS直接实现

我们采用方案2 将 base64decode 复制到一个文件中，然后通过 execjs 进行调用

Python 执行JS库 execjs

execjs可以在python中运行javascript代码

官网：https://pypi.org/project/PyExecJS/

安装：pip install PyExecJS

可以切换清华源

安装成功之后在pycharm中引入一下，不出错误，表示运行成功

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1556000447120" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">

image

我们对JS进行编译

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import execjs
with open('scmor.js', 'r', encoding='utf-8') as f:
js = f.read()
ctx = execjs.compile(js) # 对JS进行编译
</pre>

核心的方法

<pre spellcheck="false" style="box-sizing: border-box; margin: 5px 0px; padding: 5px 10px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; font-size: 16px; line-height: inherit; font-family: inherit; vertical-align: baseline; cursor: text; counter-reset: list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; background-color: rgb(240, 240, 240); border-radius: 3px; white-space: pre-wrap; color: rgb(34, 34, 34); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">def decode(string):
string = ctx.call('base64decode', string) # base64解码string参数,string参数上面获取到的autourls里面的值
key = " link@scmor.comok " # Gword的值+ 'ok ' key 在HTML页面中可以获取到
Len = len(key) # Gword长度
code = ''
for i in range(0, len(string)):
k = i % Len
n = ord(str(string[i])) ^ ord(str(key[k]))
code += chr(n)
return ctx.call('base64decode', code)
</pre>

运行结果展示

<tt-image data-tteditor-tag="tteditorTag" contenteditable="false" class="syl1556000447125" data-render-status="finished" data-syl-blot="image" style="box-sizing: border-box; cursor: text; color: rgb(34, 34, 34); font-family: "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", "Helvetica Neue", Arial, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: block;">