2018年的最后几天平台迭代到了V6版本,代号Veronica(维罗妮卡)。(Veronica名字取自电影复仇者联盟2,电影中的Veronica是由布鲁斯·班纳(绿巨人)和托尼·斯塔克(钢铁侠)共同完成,Veronica系统包含多个功能模块,反浩克装甲是其中之一,反浩克装甲的作用相当于外勤特工,其余功能模块起着支持、补给作用。除反浩克外其余已知的功能模块包括:独立卫星通信,独立操作,战损支持(出现战损可调用备件更换),监狱式隔离仓等,有点跑题了....之所以取名Veronica ,是因为运维平台跟电影中有很多向像之处,首先平台本身独立工作,且并不具备任何运维功能,全部依靠调用外部API接口外部应用,同时平台本身可以通过分布式部署,使不同的节点一同来处理公共的任务,诸如:监控作业,项目发布,数据采集(采集这块我用到了Salt,在Veronica 中,Salt master也是多主架构的部署方式,即由不同的Veronica 节点来进行配置管理、信息采集、同步模块等,这个下期会说到),统计,计算及一些耗时任务,比如创建虚拟机。
下文给大家一个思路,就是借助Celery分布式特性,通过多节点运行数据监控,然后存储到Redis,最后由前端Dashboard读取数据。先上一张效果图:
image.pngimage.png
一、 环境准备
1.Celery 3.1.26,官网http://docs.celeryproject.org/en/latest/
2.Celery-with-redis 3.0
3.QUEUE及CELERY_RESULT_BACKEND 采用了REDIS 3.2.0 (后面会换Rabbitmq,官方潜力推介)
4.Django 1.6.6
5.Django-celery (3.1.17)
二、食用方法
1.Celery:
我的环境是3台服务器,一台主控,两台代理(处理任务),每台代理跑4个队列(如图1中的25-1---25-4及26-1---26-4)
代理25服务器的启动命令 python manage.py celery worker -n 25 -l info -Q 25-1,25-2,25-3,25-4
代理26服务器的启动命令 python manage.py celery worker -n 26 -l info -Q 26-1,26-2,26-3,26-4
主控24服务器的启动命令python manage.py celery beat -n 24 -l info 及 python manage.py celery worker -n 24 -l info -Q default
代码结构如下:
image.png
然后3台分别在settings.py中注册定时任务模块:
CELERY_IMPORTS = ('cmdb.mycelery.tasks.vm_task',
'cmdb.mycelery.schedule.cmdb_cron.cmdb_f',
'cmdb.mycelery.schedule.openfalconagent_cron.openfalcon_f',
'cmdb.mycelery.schedule.salt_cron.salt_f',
'cmdb.mycelery.schedule.dashboard_cron.openfalcon_f',
)
2.Djcelery:
使用了djcelery,可以在任务中方便的直接操作Django数据库,而且最终的任务可以在Django的后台中查看和修改相关的任务。但是我们需要通过运维平台来调用djcelery的models来进行作业的增、删、改,而不是让运维小伙伴通过ADMIN后台去修改。
后台代码:
@csrf_exempt
@login_required
def schedule_interval_ajax(request):
"""获取interval,组成select"""
interval_dict = {}
intervalschedule_list = celery_models.IntervalSchedule.objects.all()
for i in intervalschedule_list:
interval_dict[i.id] = str(i.every) + ' ' + str(i.period)
return HttpResponse(simplejson.dumps(interval_dict, ensure_ascii=False))
@csrf_exempt
@login_required
def schedule_crontab_ajax(request):
"""获取crontab,组成select"""
crontab_dict = {}
crontabschedule_list = celery_models.CrontabSchedule.objects.all()
for i in crontabschedule_list:
crontab_dict[i.id] = str(i.minute) + ' ' + str(i.hour) + ' ' + str(i.day_of_week) + ' ' + str(i.day_of_month) + ' ' + str(i.month_of_year)
return HttpResponse(simplejson.dumps(crontab_dict, ensure_ascii=False))
@csrf_exempt
@login_required
@my_permissionVerify
def schedule_edit_ajax(request):
"""修改作业"""
if request.method == 'POST':
schedule_type = request.POST['schedule_type']
try:
schedule_crontab_id = request.POST['schedule_crontab']
except:
schedule_crontab_id = ''
try:
schedule_interval_id = request.POST['schedule_interval']
except:
schedule_interval_id = ''
if request.POST['schedule_enabled'] == 'true':
schedule_enabled_id = 1
else:
schedule_enabled_id = 0
schedule_args = request.POST['schedule_args']
schedule_kwargs = request.POST['schedule_kwargs']
schedule_name = request.POST['schedule_name']
schedule_queue = request.POST['schedule_queue']
try:
celery_models.PeriodicTask.objects.filter(name=schedule_name).update(interval = schedule_interval_id,
name = schedule_name,
crontab = schedule_crontab_id,
enabled = schedule_enabled_id,
args = schedule_args,
kwargs = schedule_kwargs,
queue = schedule_queue)
"""保存"""
celery_models.PeriodicTask.objects.get(name=schedule_name).save()
return HttpResponse('{"status":1}')
except Exception as err:
print err
return HttpResponse('{"status":0, "err":%s}'% err)
else:
name = request.GET['schedule_name']
periodictask_dict = {}
interval_dict = {}
crontab_dict = {}
periodictask_info = celery_models.PeriodicTask.objects.filter(name=name)
for i in periodictask_info:
periodictask_dict['name'] = i.name
periodictask_dict['task'] = i.task
interval_dict['id'] = i.interval_id
interval_dict['context'] = str(i.interval)
crontab_dict['id'] = i.crontab_id
crontab_dict['context'] = str(i.crontab)
periodictask_dict['interval_dict'] = interval_dict
periodictask_dict['crontab_dict'] = crontab_dict
periodictask_dict['queue'] = i.queue
periodictask_dict['args'] = i.args
periodictask_dict['kwargs'] = i.kwargs
periodictask_dict['enabled'] = i.enabled
return HttpResponse(simplejson.dumps(periodictask_dict, ensure_ascii=False))
JS代码:
function schedule_type_hidden (obj){
if (obj.value=="1"){
document.getElementById("add_schedule_interval").style.display ="block";
document.getElementById("add_schedule_crontab").style.display ="none";
}else{
document.getElementById("add_schedule_interval").style.display ="none";
document.getElementById("add_schedule_crontab").style.display ="block";
}
}
$(function(){
$("#schedule_interval_select").focus(function(){
$.ajax({
async: false,
type: "POST",
url : "../schedule_interval_ajax/",
data : $("#submit_form").serialize(),
cache: false,
dataType: "json",
beforeSend:function(){
Metronic.blockUI({animate: true});
},
success: function(obj) {
for (var id in obj){
$("#schedule_interval_select").append("<option value="+id+">"+obj[id]+"</option>");
}
$("#schedule_interval_select").unbind("focus");
},
complete: function() {
Metronic.unblockUI();
},
});
return false;
});
});
$(function(){
$("#schedule_crontab_select").focus(function(){
$.ajax({
async: false,
type: "POST",
url : "../schedule_crontab_ajax/",
data : $("#submit_form").serialize(),
cache: false,
dataType: "json",
beforeSend:function(){
Metronic.blockUI({animate: true});
},
success: function(obj) {
for (var id in obj){
$("#schedule_crontab_select").append("<option value="+id+">"+obj[id]+"</option>");
}
$("#schedule_crontab_select").unbind("focus");
},
complete: function() {
Metronic.unblockUI();
},
});
return false;
});
});
$(function(){
$("#edit_schedule_interval_select").focus(function(){
$.ajax({
async: false,
type: "POST",
url : "../schedule_interval_ajax/",
data : $("#submit_form").serialize(),
cache: false,
dataType: "json",
beforeSend:function(){
Metronic.blockUI({animate: true});
},
success: function(obj) {
$("#edit_schedule_interval_select").empty()
for (var id in obj){
$("#edit_schedule_interval_select").append("<option value="+id+">"+obj[id]+"</option>");
}
$("#edit_schedule_interval_select").unbind("focus");
},
complete: function() {
Metronic.unblockUI();
},
});
return false;
});
});
$(function(){
$("#edit_schedule_crontab_select").focus(function(){
$.ajax({
async: false,
type: "POST",
url : "../schedule_crontab_ajax/",
data : $("#submit_form").serialize(),
cache: false,
dataType: "json",
beforeSend:function(){
Metronic.blockUI({animate: true});
},
success: function(obj) {
$("#edit_schedule_crontab_select").empty()
for (var id in obj){
$("#edit_schedule_crontab_select").append("<option value="+id+">"+obj[id]+"</option>");
}
$("#edit_schedule_crontab_select").unbind("focus");
},
complete: function() {
Metronic.unblockUI();
},
});
return false;
});
});
function schedule_edit(){
$(".fa-edit").bind('click', function() {
var schedule_name =$(this).parent().parent().find('td').eq(0).attr("id");
$.ajax({
type: "GET",
url: "../schedule_edit_ajax/?schedule_name="+schedule_name,
cache: false,
async:true,
dataType: "json",
beforeSend:function(){
Metronic.blockUI({animate: true});
},
complete: function() {
Metronic.unblockUI();
},
success: function(obj) {
$('#edit_schedule_name').val(obj['name'])
$('#edit_schedule_queue').val(obj['queue'])
$('#edit_schedule_args').val(obj['interval_id'])
$('#edit_schedule_args').val(obj['args'])
$('#edit_schedule_kwargs').val(obj['kwargs'])
if (obj['interval_dict']['id'] == null){
$('#edit_schedule_type').val("2")
document.getElementById("edit_add_schedule_interval").style.display ="none";
document.getElementById("edit_add_schedule_crontab").style.display ="block";
$("#edit_schedule_crontab_select").append("<option value="+obj['crontab_dict']['id']+">"+obj['crontab_dict']['context']+"</option>");
}
if (obj['crontab_dict']['id'] == null){
$('#edit_schedule_type').val("1")
document.getElementById("edit_add_schedule_interval").style.display ="block";
document.getElementById("edit_add_schedule_crontab").style.display ="none";
$("#edit_schedule_interval_select").append("<option value="+obj['interval_dict']['id']+">"+obj['interval_dict']['context']+"</option>");
}
if (obj['enabled'] == true ){
$('#edit_schedule_enabled').bootstrapSwitch('toggleState');
$('#edit_schedule_enabled').bootstrapSwitch('state', true);
$("#edit_schedule_enableds").val('true')
}else{
$('#edit_schedule_enabled').bootstrapSwitch('toggleState');
$('#edit_schedule_enabled').bootstrapSwitch('state', false);
$("#edit_schedule_enableds").val('false')
}
}
});
$("#edit_schedule_enabled").on('switchChange.bootstrapSwitch', function(e, status) {
if (status == false){
$("#edit_schedule_enableds").val('false');
}else{
$("#edit_schedule_enableds").val('true');
}
});
$('#responsive2 form').submit(function(){
$.ajax({
type: "POST",
data: $('#responsive2 form').serialize(),
url: "../schedule_edit_ajax/",
cache: false,
async:true,
dataType: "json",
beforeSend:function(){
Metronic.blockUI({animate: true});
},
complete: function() {
Metronic.unblockUI();
},
success: function(obj) {
if (obj['status'] == "1"){
alert('定时调度修改成功');
$('#responsive2').modal('hide');
$('#responsive2 form')[0].reset();
}else {
alert('定时调度修改失败,错误为:'+obj['err']);
}
}
});
return false;
});
});
$('#responsive2').on('hide.bs.modal', function () {
location.reload();
});
}
$(document).ready(function(){
schedule_edit();
})
3.最后:
后面要做的就是通过异步的任务队列对获取你想要的数据,图上我的作业是通过OPENFALCON的接口获取负载排名、IO排名,进出流量排名等。然后存储到Redis,在由Dashboard通过局部刷新分别从Redis中读取数据。这样做的好处,不会因为访问的客户端多,导致频繁发起对各个监控系统的请求。同时可以将监控作业分散在若干机器内,实现横向的扩展(绝大部分商业监控的办法)。
获取报警作业,代码如下:
r = redis_conect_db5()
@task()
def triggers_list():
cursor.execute(sql)
triggers =cursor.fetchall()
triggers_list=[]
for i in triggers:
triggers_dict={}
triggers_dict['host'] = i[0]
triggers_dict['description'] = i[1]
triggers_dict['lastchange'] = Calculate_date_today(str(i[2]))
triggers_dict['priority'] = i[3]
triggers_dict['metric'] = ''.join(i[4].split('/')[1:]).replace(',','<br>')
triggers_dict['cond'] = i[5]
triggers_list.append(triggers_dict)
cursor.close()
conn.close()
try:
r.set("dashboard:triggers_list", json.dumps(triggers_list))
except Exception as err:
print err
return triggers_list
前台局部刷新获取数据,代码如下:
r = redis_conect_db5()
def triggers_list():
try:
print 'triggers from redis'
triggers_list = json.loads(r.get("dashboard:triggers_list"))
except:
triggers_list = []
return triggers_list
网友评论