一、简介
Bearded-avenger(cifv3)是一个定期爬取公开威胁数据的平台（the fastest way to consume threat intelligence）。通过定制yaml规则对定期公布威胁数据的站点进行数据爬取，并归纳整理出统一的格式方便安全人员使用。
数据库有两种选择：Sqlite和Elasticsearch，我选择了强大的Elasticsearch
github：https://github.com/csirtgadgets/bearded-avenger
二、准备一个搭建cifv3的服务器环境
建议使用：（本人本次搭建在ESX上）
操作系统：ubuntu14.04
内核数：8个
内存：16G
三、选择一个最新的稳定版本下载到本地
本次使用：https://github.com/csirtgadgets/bearded-avenger/releases/tag/3.0.0a16
cd /root/
wget https://github.com/csirtgadgets/bearded-avenger/archive/3.0.0a16.tar.gz
四、前期网络环境及下载源的设置
1. 更改为阿里云源
mv /etc/apt/sources.list /etc/apt/sources.list.bak
vim /etc/apt/sources.list
deb http://mirrors.aliyun.com/ubuntu/ trusty main multiverse restricted universe
deb http://mirrors.aliyun.com/ubuntu/ trusty-backports main multiverse restricted universe
deb http://mirrors.aliyun.com/ubuntu/ trusty-proposed main multiverse restricted universe
deb http://mirrors.aliyun.com/ubuntu/ trusty-security main multiverse restricted universe
deb http://mirrors.aliyun.com/ubuntu/ trusty-updates main multiverse restricted universe
deb-src http://mirrors.aliyun.com/ubuntu/ trusty main multiverse restricted universe
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-backports main multiverse restricted universe
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-proposed main multiverse restricted universe
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-security main multiverse restricted universe
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-updates main multiverse restricted universe
2. ubuntu配置pip国内镜像源
cd ~
mkdir .pip
cd .pip
vim pip.conf
[global]
trusted-host = mirrors.ustc.edu.cn
index-url = https://mirrors.ustc.edu.cn/pypi/web/simple
3. 更新系统源和软件
apt-get update
apt-get upgrade
五、一些可能出错的提前避免
由于使用的是一键安装脚本，所以每次出错都要重新执行一遍，而且其中可能命令重复又会出现新的问题，在此给出了所有可能出错的解决方法，可以运行解决办法中的命令，让大家讲问题扼杀在萌芽期。
1. ImportError: No module named packaging.version
错误截图：
pip.jpg
解决办法：
apt-get purge -y python-pip
wget https://bootstrap.pypa.io/get-pip.py
python ./get-pip.py
apt-get install python-pip
2. geoipupdate失败（国内更新geoip库可能较慢，超过超时时间）
错误截图：
geoip.jpg
解决办法：
add-apt-repository ppa:maxmind/ppa
aptitude update
aptitude install geoipupdate
geoipupdate -v
3. ansible_env错误
错误截图：
ansible.jpg
解决办法：
vim bearded-avenger/deployment/ubuntu14/roles/ubuntu14/tasks/user.yml
ansible_env.SUDO_USER 改成 ansible_env.USER
ansible2.png
4. 执行测试脚本时出错（存在国内无法访问的域名）
错误截图：
test.png
部分出错代码：
"=================================== FAILURES ===================================",
"______________________________ test_gatherer_asn _______________________________",
"",
" def test_gatherer_asn():",
" a = Asn(fast=False)",
" ",
" def _resolve(i):",
" return data",
" ",
" a._resolve_ns = _resolve",
" x = a.process(Indicator(indicator='216.90.108.0'))",
" ",
"> assert x.asn == '23028'",
"E assert None == '23028'",
"E + where None = {\\n \"indicator\": \"216.90.108.0\",\\n \"itype\": \"ipv4\"\\n}.asn",
"",
"test/test_gatherer_asn.py:28: AssertionError",
"----------------------------- Captured stderr call -----------------------------",
"2017-02-17 07:08:11,757 - INFO - cif.utils[22][MainThread] - \u001b[32m0.108.90.216.origin.asn.cymru.com - The DNS operation timed out after 5.0050868988 seconds -- this may be normal\u001b[0m",
"2017-02-17 07:08:11,757 - INFO - cif.utils[22][MainThread] - \u001b[32m\u001b[32m0.108.90.216.origin.asn.cymru.com - The DNS operation timed out after 5.0050868988 seconds -- this may be normal\u001b[0m\u001b[0m",
"2017-02-17 07:08:11,757 - INFO - cif.utils[22][MainThread] - \u001b[32m\u001b[32m\u001b[32m0.108.90.216.origin.asn.cymru.com - The DNS operation timed out after 5.0050868988 seconds -- this may be normal\u001b[0m\u001b[0m\u001b[0m",
"================ 1 failed, 17 passed, 9 skipped in 7.83 seconds ================"
解决办法：
暂时删掉过不去的测试脚本
cd bearded-avenger/test
mv test_gatherer_asn.py test_gatherer_asn.py.bak
5. docker pull elasticsearch镜像时可能超时
在本地安装docker并pull elasticsearch:2
流程：
curl -sSL http://acs-public-mirror.oss-cn-hangzhou.aliyuncs.com/docker-engine/internet | sh -
docker pull elasticsearch:2
六、执行一件安装脚本：
cd bearded-avenger/deployment/ubuntu14
bash bootstrap_elasticsearch.sh
成功安装的截图：
chengong.png
七、尝试爬取黑IP黑域名等数据到elasticsearch
su - cif
csirtg-smrt --client cif --fireball -r /etc/cif/rules/default/csirtg.yml -f port-scanners -d
成功执行的截图：
feed.png
八、查看elasticsearch数据
目前elasticsearch还没有安装head、kopf等插件，只能用curl命令
可拷贝其他elasticsearch中的plugins目录到本地的指定目录。
root@elk:~/bearded-avenger/test# find / -name elasticsearch.in.sh
/var/lib/docker/aufs/diff/b8d77a9eee69729f60d454e22b262abd93ebd135fedf92faff4b7e5d950a1194/usr/share/elasticsearch/bin/elasticsearch.in.sh
/var/lib/docker/aufs/mnt/1b866f98ba81754fdee761eb64bb3782594d3ac4890ffa271d9f5c6abc929594/usr/share/elasticsearch/bin/elasticsearch.in.sh
root@elk:~/bearded-avenger/test# cd /var/lib/docker/aufs/mnt/1b866f98ba81754fdee761eb64bb3782594d3ac4890ffa271d9f5c6abc929594/usr/share/elasticsearch/
root@elk:/var/lib/docker/aufs/mnt/1b866f98ba81754fdee761eb64bb3782594d3ac4890ffa271d9f5c6abc929594/usr/share/elasticsearch# ll
total 52
drwxr-xr-x 13 root root 4096 Feb 17 08:05 ./
drwxr-xr-x 72 root root 4096 Feb 17 07:05 ../
drwxr-xr-x 2 root root 4096 Feb 17 08:05 bin/
drwxr-xr-x 3 bind ssh 4096 Feb 7 15:54 config/
drwxr-xr-x 2 bind ssh 4096 Feb 7 15:53 data/
drwxr-xr-x 2 root root 4096 Feb 7 15:53 lib/
drwxr-xr-x 2 bind ssh 4096 Feb 7 15:53 logs/
drwxr-xr-x 5 root root 4096 Feb 7 15:53 modules/
-rw-r--r-- 1 root root 150 Jan 3 06:51 NOTICE.txt
drwxr-xr-x 4 bind ssh 4096 Feb 17 08:02 plugins/
-rw-r--r-- 1 root root 8700 Jan 3 06:51 README.textile
root@elk:/var/lib/docker/aufs/mnt/1b866f98ba81754fdee761eb64bb3782594d3ac4890ffa271d9f5c6abc929594/usr/share/elasticsearch# cd plugins/
root@elk:/var/lib/docker/aufs/mnt/1b866f98ba81754fdee761eb64bb3782594d3ac4890ffa271d9f5c6abc929594/usr/share/elasticsearch/plugins# ll
total 16
drwxr-xr-x 4 bind ssh 4096 Feb 17 08:02 ./
drwxr-xr-x 13 root root 4096 Feb 17 08:05 ../
drwxr-xr-x 5 bind ssh 4096 Feb 17 07:59 head/
drwxr-xr-x 8 bind ssh 4096 Feb 17 07:59 kopf/
九、其他配置
修改elasticsearch.in.sh 文件中的配置信息，保证发挥elasticsearch的最大特性
将此处两个值设置相同切小于等于最大内存数的1/2，此处可设置成8g
其他参数根据自己需求进行修改
es.png
网友评论