利用zabbix完成一套完整的自动化监控

作者: 繁华丶凋零 | 来源:发表于2021-02-26 09:59 被阅读0次

利用zabbix完成一套完整的自动化监控
zabbix
Zabbix使用LLD自动发现规则发现监控docker容器（下）
Oracle数据库添加到Zabbix监控
zabbix 小知识
zabbix使用percona-monitoring-plugi
【zabbix】zabbix配置邮件告警通知
Linux监控平台介绍、zabbix监控介绍、安装zabbix（
zabbix 键值
zabbix搭建安全服务器集群

前景：
开发环境巨多（并行环境10个以上）
设计一套自动化监控系统

监控项涉及：
服务器内存，/home挂载盘，nas服务，进程

数据来源：
1.服务器信息已按组名落地分类在 mongo 表里，可直接抽出数据
2.服务器进程信息统一存放在 ansible部署机各环境目录下

设计方案：

各服务器搭建 zabbix_agent,塞入手写脚本和配置
2.zabbix自制吗，模板，创建自动发现规则（port自动发现，process自动发现）
3.shell 脚本处理服务器进程信息并按服务器区分，生成文件分发到每台服务器 support用户下
4.python脚本定时更新主机宏

代码1：zabbix脚本

#脚本名discovery_port.sh
#res=`cat /patrol/zabbix/mysql_port|grep -v "^#"`
res=`echo $1| sed "s/,/\n/g"`;
port=($res)
printf '{\n'
printf '\t"data":[\n'
for key in ${!port[@]}
do
    port=${port[${key}]}
    process=`cat /home/support/port_process.conf | grep -w $port | awk -F" " '{print $4}'|head -n1`
    hostname=`cat /home/support/port_process.conf | grep -w $port | awk -F" " '{print $1}'|head -n1`
    ip=`cat /home/support/port_process.conf | grep -w $port | awk -F" " '{print $2}'|head -n1`
    user=`cat /home/support/port_process.conf | grep -w $port | awk -F" " '{print $3}'|head -n1`
    if [[ "${#port[@]}" -gt 1 && "${key}" -ne "$((${#port[@]}-1))" ]];
then
        printf '\t {\n'
        printf "\t\t\t\"{#HOST_PORT}\":\"${port[${key}]}\",\"{#HOST_PORT_PROCESS}\":\"$process\",\"{#HOST_PORT_HOSTNAME}\":\"$hostname\",\"{#HOST_PORT_IP}\":\"$ip\",\"{#HOST_PORT_USER}\":\"$user\"},\n"
#        printf "\t\t\t\"{#HOST_PORT}\":\"${port[${key}]}\"},\n"         
else [[ "${key}" -eq "((${#port[@]}-1))" ]]
        printf '\t {\n'
        printf "\t\t\t\"{#HOST_PORT}\":\"${port[${key}]}\",\"{#HOST_PORT_PROCESS}\":\"$process\",\"{#HOST_PORT_HOSTNAME}\":\"$hostname\",\"{#HOST_PORT_IP}\":\"$ip\",\"{#HOST_PORT_USER}\":\"$user\"}\n"
#        printf "\t\t\t\"{#HOST_PORT}\":\"${port[${key}]}\"}\n"
fi
done
printf '\t ]\n'
printf '}\n'

#脚本名discovery_process.sh
#res=`cat /patrol/zabbix/mysql_port|grep -v "^#"`
res=`echo $1| sed "s/,/\n/g"`;
port=($res)
printf '{\n'
printf '\t"data":[\n'
for key in ${!port[@]}
do
    process=${port[${key}]}
    hostname=`cat /home/support/port_process.conf | grep -w $process | awk -F" " '{print $1}'|head -n1`
    ip=`cat /home/support/port_process.conf | grep -w $process | awk -F" " '{print $2}'|head -n1`
    user=`cat /home/support/port_process.conf | grep -w $process | awk -F" " '{print $3}'|head -n1`
    
    if [[ "${#port[@]}" -gt 1 && "${key}" -ne "$((${#port[@]}-1))" ]];
then
        printf '\t {\n'
        printf "\t\t\t\"{#HOST_PROCESS}\":\"${port[${key}]}\",\"{#HOST_PROCESS_HOSTNAME}\":\"$hostname\",\"{#HOST_PROCESS_IP}\":\"$ip\",\"{#HOST_PROCESS_USER}\":\"$user\",\"{#HOST_PROCESS_MONITOR}\":\"$process_monitor\"},\n"
   
else [[ "${key}" -eq "((${#port[@]}-1))" ]]
        printf '\t {\n'
        printf "\t\t\t\"{#HOST_PROCESS}\":\"${port[${key}]}\",\"{#HOST_PROCESS_HOSTNAME}\":\"$hostname\",\"{#HOST_PROCESS_IP}\":\"$ip\",\"{#HOST_PROCESS_USER}\":\"$user\",\"{#HOST_PROCESS_MONITOR}\":\"$process_monitor\"}\n"
fi
done
printf '\t ]\n'
printf '}\n'

这里自动发现进程脚本设置多个原因是更新主机宏时候发现字符串长度有限制，最大不能超过254，所以设计多个自动发现规则

#脚本名discovery_process1.sh
#res=`cat /patrol/zabbix/mysql_port|grep -v "^#"`
res=`echo $1| sed "s/,/\n/g"`;
port=($res)
printf '{\n'
printf '\t"data":[\n'
for key in ${!port[@]}
do
    process=${port[${key}]}
    hostname=`cat /home/support/port_process.conf | grep -w $process | awk -F" " '{print $1}'|head -n1` 
    ip=`cat /home/support/port_process.conf | grep -w $process | awk -F" " '{print $2}'|head -n1`
    user=`cat /home/support/port_process.conf | grep -w $process | awk -F" " '{print $3}'|head -n1`

    if [[ "${#port[@]}" -gt 1 && "${key}" -ne "$((${#port[@]}-1))" ]];then
        printf '\t {\n'
        printf "\t\t\t\"{#HOST_PROCESS1}\":\"${port[${key}]}\",\"{#HOST_PROCESS_HOSTNAME1}\":\"$hostname\",\"{#HOST_PROCESS_IP1}\":\"$ip\",\"{#HOST_PROCESS_USER1}\":\"$user\",\"{#HOST_PROCESS_MONITOR1}\":\"$process_monitor\"},\n"
   
    else [[ "${key}" -eq "((${#port[@]}-1))" ]]
        printf '\t {\n'
        printf "\t\t\t\"{#HOST_PROCESS1}\":\"${port[${key}]}\",\"{#HOST_PROCESS_HOSTNAME1}\":\"$hostname\",\"{#HOST_PROCESS_IP1}\":\"$ip\",\"{#HOST_PROCESS_USER1}\":\"$user\",\"{#HOST_PROCESS_MONITOR1}\":\"$process_monitor\"}\n"
fi
done
printf '\t ]\n'
printf '}\n'

#脚本名discovery_process2.sh
#res=`cat /patrol/zabbix/mysql_port|grep -v "^#"`
res=`echo $1| sed "s/,/\n/g"`;
port=($res)
printf '{\n'
printf '\t"data":[\n'
for key in ${!port[@]}
do
    process=${port[${key}]}
    hostname=`cat /home/support/port_process.conf | grep -w $process | awk -F" " '{print $1}'|head -n1`
    ip=`cat /home/support/port_process.conf | grep -w $process | awk -F" " '{print $2}'|head -n1`
    user=`cat /home/support/port_process.conf | grep -w $process | awk -F" " '{print $3}'|head -n1`

    if [[ "${#port[@]}" -gt 1 && "${key}" -ne "$((${#port[@]}-1))" ]];
then
        printf '\t {\n'
        printf "\t\t\t\"{#HOST_PROCESS2}\":\"${port[${key}]}\",\"{#HOST_PROCESS_HOSTNAME2}\":\"$hostname\",\"{#HOST_PROCESS_IP2}\":\"$ip\",\"{#HOST_PROCESS_USER2}\":\"$user\",\"{#HOST_PROCESS_MONITOR2}\":\"$process_monitor\"},\n"
   
else [[ "${key}" -eq "((${#port[@]}-1))" ]]
        printf '\t {\n'
        printf "\t\t\t\"{#HOST_PROCESS2}\":\"${port[${key}]}\",\"{#HOST_PROCESS_HOSTNAME2}\":\"$hostname\",\"{#HOST_PROCESS_IP2}\":\"$ip\",\"{#HOST_PROCESS_USER2}\":\"$user\",\"{#HOST_PROCESS_MONITOR2}\":\"$process_monitor\"}\n"
fi
done
printf '\t ]\n'
printf '}\n'

#脚本名discovery_process3.sh
#res=`cat /patrol/zabbix/mysql_port|grep -v "^#"`
res=`echo $1| sed "s/,/\n/g"`;
port=($res)
printf '{\n'
printf '\t"data":[\n'
for key in ${!port[@]}
do
    process=${port[${key}]}
    hostname=`cat /home/support/port_process.conf | grep -w $process | awk -F" " '{print $1}'|head -n1`
    ip=`cat /home/support/port_process.conf | grep -w $process | awk -F" " '{print $2}'|head -n1`
    user=`cat /home/support/port_process.conf | grep -w $process | awk -F" " '{print $3}'|head -n1`

    if [[ "${#port[@]}" -gt 1 && "${key}" -ne "$((${#port[@]}-1))" ]];
then
        printf '\t {\n'
        printf "\t\t\t\"{#HOST_PROCESS3}\":\"${port[${key}]}\",\"{#HOST_PROCESS_HOSTNAME3}\":\"$hostname\",\"{#HOST_PROCESS_IP3}\":\"$ip\",\"{#HOST_PROCESS_USER3}\":\"$user\",\"{#HOST_PROCESS_MONITOR3}\":\"$process_monitor\"},\n"
   
else [[ "${key}" -eq "((${#port[@]}-1))" ]]
        printf '\t {\n'
        printf "\t\t\t\"{#HOST_PROCESS3}\":\"${port[${key}]}\",\"{#HOST_PROCESS_HOSTNAME3}\":\"$hostname\",\"{#HOST_PROCESS_IP3}\":\"$ip\",\"{#HOST_PROCESS_USER3}\":\"$user\",\"{#HOST_PROCESS_MONITOR3}\":\"$process_monitor\"}\n"
fi
done
printf '\t ]\n'
printf '}\n'

代码2：zabbix配置文件

#配置文件名： discovery.conf.j2
UserParameter=monitor_port.discovery[*],/home/zabbix/dev/zabbix/bin/discovery_port.sh $1  
UserParameter=monitor_process.discovery[*],/home/zabbix/dev/zabbix/bin/discovery_process.sh $1
UserParameter=monitor_process.discovery1[*],/home/zabbix/dev/zabbix/bin/discovery_process1.sh $1
UserParameter=monitor_process.discovery2[*],/home/zabbix/dev/zabbix/bin/discovery_process2.sh $1
UserParameter=monitor_process.discovery3[*],/home/zabbix/dev/zabbix/bin/discovery_process3.sh $1

#配置文件名：zabbix_agentd.conf.j2
PidFile=/home/zabbix/dev/zabbix/zabbix_agentd.pid
LogFile=/home/zabbix/dev/zabbix/logs/zabbix_agentd.log
LogFileSize=0
ListenPort=10050
StartAgents=3
ListenIP=0.0.0.0
Server=1.2.3.4
ServerActive=1.2.3.4
Hostname={{ ansible_host }}
Include=/home/zabbix/dev/zabbix/etc/zabbix_agentd.conf.d

代码3：更新宏脚本

# -*- coding: utf-8 -*-
# from zabbix_api import ZabbixAPI
import sys, os, re
import time
import configparser
import logging
import pymongo
from pymongo import MongoClient
import math
from multiprocessing import Process,Lock
import threading
from zabbix_api import ZabbixAPI



BASE_DIR = '/home/deploy/app/update_monitor_hong'

zapi = ZabbixAPI(server='http://xxx/', path="", log_level=0)
zapi.login('admin', 'xxxx')

client = pymongo.MongoClient('xxxxx')
db = client['xxx']
db.authenticate('xxx','xxx')
table = db['serverpool']

logger = logging.getLogger()
logger.setLevel(logging.ERROR)
logfile = '/home/deploy/app/update_monitor_hong/logs/api.log'
fh = logging.FileHandler(logfile, mode='w')
fh.setLevel(logging.ERROR)
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)

     
def zabbix_all_info():
    group_list = all_info_get()
    clean_command = '[ -f /home/deploy/app/update_monitor_hong/process.list_tmp ] && rm /home/deploy/app/update_monitor_hong/process.list_tmp'
    os.system(clean_command)
    for group in group_list:
        dir = '/home/deploy/ansible_workspace/environments/' + group
        dir_file = dir + '/process.list'
        if os.path.exists(dir_file):
            add_command = 'cat ' + dir_file + '>> /home/deploy/app/update_monitor_hong/process.list_tmp'
            os.system(add_command)
        vars_file = dir + '/vars.yml'
        if os.path.exists(vars_file):
            add_vars_command = 'bash' + ' /home/deploy/app/update_monitor_hong/project/mq_redis_nginx.sh ' + vars_file + ' ' + group
            print(add_vars_command)
            os.system(add_vars_command)  

    change_command = 'bash ' + ' /home/deploy/app/update_monitor_hong/project/get_zabbix_info.sh'
    os.system(change_command) 
    logger.debug('achieve ' + group + ' process.list info success')

def all_info_get():
    group_list = [] #所有未结项史诗组       
    all_group_info = table.find({"status": {'$ne': "idle"}}, {"status": "1"})
    for group in all_group_info:
        group_list.append(group['status'])
    group_list = set(group_list)
    logger.debug('shengcheng epic_group_lists success ')
    return group_list


def all_host_get():
    host_list = [] #所有未结项史诗组       
    all_host_info = table.find({"status": {'$ne': "idle"}}, {"status": "1"})
    for host in all_host_info:
        host_list.append(host['_id'])
#    group_list = set(group_list)
    logger.debug('shengcheng epic_host_lists success ')
    return host_list



def all_idel_get():
    idel_list = [] #所有未结项史诗组
    table = db['serverpool']
    all_idel_info = table.find({"status": "idle"}, {"status": "1"})
    for idel_ip in all_idel_info:
        idel_list.append(idel_ip['_id'])
    idel_list = set(idel_list)
    logger.debug('using idel_host_lists are: ' + idel_list)
    return idel_list

def get_host_id(host_name):    
    hostid = zapi.host.get({"filter": {"host": host_name}})[0]['hostid']

def add_hong(host_name,hong_name,hong_value): #新增宏
    hostid = zapi.host.get({"filter": {"host": host_name}})[0]['hostid']
    zapi.usermacro.create({
        "hostid": hostid,
        "macro": hong_name,
        "value": hong_value
    })
    logger.debug('add ' + host_name + hong_name + ' success')

def update_hong(host_name,hong_name,hong_value):  #根据主机和宏名称更新宏
    hostmacroid = get_hostmacroid(host_name,hong_name)
    zapi.usermacro.update({
        "hostmacroid": hostmacroid,
        "value": hong_value
    })
    logger.debug('update ' + host_name + hong_name + ' success')

def update_hong_id(hong_id,hong_value):  #根据宏id更新宏
    zapi.usermacro.update({
        "hostmacroid": hong_id,
        "value": hong_value
    })
    logger.debug('update ' + hong_id + ' success')

def get_hostmacroid(host_name,hong_name):  #获取单个主机单个宏id
    hostid = zapi.host.get({"filter": {"host": host_name}})[0]['hostid']
    host_macro_id_all = zapi.usermacro.get({ "hostids":hostid })
    for host_macro_line in host_macro_id_all:
        if host_macro_line['macro'] == hong_name:
            host_macro_id = host_macro_line['hostmacroid']        
    logger.debug('get ' + host_name + hong_name + ' id success')
    return host_macro_id

def get_hostmacroid_value(host_name,hong_name):  #获取单个宏值
    hostid = zapi.host.get({"filter": {"host": host_name}})[0]['hostid']
    host_macro_id_all = zapi.usermacro.get({ "hostids":hostid })
    for host_macro_line in host_macro_id_all:
        if host_macro_line['macro'] == hong_name:
            host_macro_id_value = host_macro_line['value']
    logger.debug('get ' + host_name + hong_name + ' id success')
    return host_macro_id_value     

def get_hostmacroid_list(host_name):  #获取单台主机所有宏id
    host_macros_list = []
    hostid = zapi.host.get({"filter": {"host": host_name}})[0]['hostid']
    host_macros = zapi.usermacro.get({"output":"extend", "hostids":hostid})
    for i in range(len(host_macros)):
        host_macros_list.append(host_macros[i]['hostmacroid'])
    logger.debug('get ' + host_name + ' all hongids success')
    return host_macros_list

def get_hostmacroname_list(host_name):  #获取单台主机所有宏名称
    host_macros_name_list = []
    hostid = zapi.host.get({"filter": {"host": host_name}})[0]['hostid']
    host_name_macros = zapi.usermacro.get({"output":"extend", "hostids":hostid})
    for i in range(len(host_name_macros)):
        host_macros_name_list.append(host_name_macros[i]['macro'])
    logger.debug('get ' + host_name + ' all hongnames success')
    return host_macros_name_list
   
def delete_hostmacroid(host_name):
    host_macros_list = get_hostmacroid_list(host_name)
    if len(host_macros_list) > 0:
        zapi.usermacro.delete(host_macros_list)

def get_group_id(group_name):
    groupid = zapi.hostgroup.get({"filter": {"name": group_name}})[0]['groupid']
    return groupid

def create_host(host_name,group_name):
    group_id = get_group_id(group_name)
    EPIC_templateid = zapi.template.get({"filter": {"host": "Linux OS EPIC"}})[0]['templateid']
    zapi.host.create({"host": host_name, "interfaces": [
            {"type": 1, "main": 1, "useip": 1, "ip": host_name, "dns": "", "port": "10050"}],
                               "groups": [{"groupid": group_id}], "tags": [{"tag": "Host name", "value": host_name}],
                               "templates": [{"templateid": EPIC_templateid}]})
def get_zabbix_all_hosts():
    all_zabbix_allgroup_hosts_list = []
    all_zabbix_allhosts_info = zapi.host.get({"output": "extend", })
    for i in range(len(all_zabbix_allhosts_info)):
        all_zabbix_allgroup_hosts_list.append(all_zabbix_allhosts_info[i]['host'])
    return(all_zabbix_allgroup_hosts_list)

#此函数是处理字符主机宏字符串长于255的情况
def process_value_list(process_value):
    process_value_value_list = process_value.strip(',').split(',')
    return_value_list = []
    process_value_reset = ''
    process_value_json = {}
    while len(process_value) > 255:
        i = 0
        while len(process_value_reset) < 255:
                k = process_value_value_list[i]
                process_value_reset = process_value_reset + k + ','
                process_value_json['end'] = i
                i = i + 1
        j = process_value_json['end']
        process_value_value_list_new = process_value_value_list[0:j]
        process_value_value_list = process_value_value_list[j:]
        process_value_new = ",".join(process_value_value_list_new)
        process_value_reset = ''
        return_value_list.append(process_value_new)
        process_value_linshi = process_value_new + ','
        process_value = process_value.replace(process_value_linshi,'')

    else:
        return_value_list.append(process_value)

    return(return_value_list)




def update_zabbix_info():
    zabbix_hosts = get_zabbix_all_hosts()
    host_list = all_host_get()
    exist_list = []
    hong_process = '{$HOST_PROCESS}'
    hong_port = '{$HOST_PORT}'
#    hong_name_list = ['{$HOST_PROCESS}','{$HOST_PROCESS1}','{$HOST_PROCESS2}','{$HOST_PROCESS3}','{$HOST_PORT}']
    for line in open("/home/deploy/app/update_monitor_hong/host.list"):
        exist_list.append(line.replace('\n','').replace('\r',''))
    for i in host_list:
        if i not in exist_list and i in zabbix_hosts:
            try:
                delete_hostmacroid(i)
                i = str(i) 
                out = i + 'delete free hostmacro'
                print(out)
            except Exception as e:
                i = str(i)
                out = i + e
                print(out)            

    for line in open("/home/deploy/app/update_monitor_hong/process.list"):
        if line.split(" ")[0] in host_list:
            host_list.remove(line.split(" ")[0])

        host = line.split(" ")[0]
        if host not in zabbix_hosts:
            create_host(host,'EPIC_servers')
            log = 'create ' + ' ' + host + ' success'
            print(log)

        hong_process_value = line.split(" ")[1]        
        hong_port_value = line.split(" ")[2]

        hong_process_value = hong_process_value.replace('\n','').replace('\r','')
        hong_port_value = hong_port_value.replace('\n','').replace('\r','')
        print(host)     
        print(hong_port_value)
        print(hong_process_value)
##################################################################################
        hong_process_name = '{$HOST_PROCESS}'
        hong_port_name = '{$HOST_PORT}'
        hostmacroname = get_hostmacroname_list(host)
        hong_process_value_list = process_value_list(hong_process_value)
        print(hong_process_value_list)
        for i,process_value in enumerate(hong_process_value_list):
            i = str(i)
            if i == str(0):
                hong_process_name = '{$HOST_PROCESS}'
            else: 
                hong_process_name = '{$HOST_PROCESS' + i + '}'
            
            if hong_process_name in hostmacroname:
                now_process_value = get_hostmacroid_value(host,hong_process_name)
                if now_process_value != hong_process_name:
                    update_hong(host,hong_process_name,process_value) 
            else:
                add_hong(host,hong_process_name,process_value)
   
        if hong_port_name in hostmacroname:
            now_port_value = get_hostmacroid_value(host,hong_port_name) #当前宏值

            if now_port_value != hong_port_value:
                update_hong(host,hong_port_name,hong_port_value)

        else:
            add_hong(host,hong_port_name,hong_port_value)



    for no_value_host in host_list:
        hong_process_value = ''
        hong_port_value = ''    
        hong_process_name = '{$HOST_PROCESS}'
        hong_port_name = '{$HOST_PORT}'
        hostmacroname = get_hostmacroname_list(no_value_host)
        if hong_process_name in hostmacroname:
            now_process_value = get_hostmacroid_value(no_value_host,hong_process_name) #当前宏值

            if now_process_value != hong_process_value:
                update_hong(no_value_host,hong_process_name,hong_process_value)

        else:
            add_hong(no_value_host,hong_process_name,hong_process_value)

        if hong_port_name in hostmacroname:
            now_port_value = get_hostmacroid_value(no_value_host,hong_port_name) #当前宏值

            if now_port_value != hong_port_value:
                update_hong(no_value_host,hong_port_name,hong_port_value)

        else:
            add_hong(no_value_host,hong_port_name,hong_port_value)


zabbix_all_info()
update_zabbix_info()
              
        
#        for idel in all_idel_info:
#            hongid_list = get_hostmacroid_list(idel)
#            if hongid_list:
#                print(idel)
#                for hong_one_id in hongid_list:
#                    update_hong_id(hong_one_id,' ')
#                    out = 'update '+ idel + '主机宏成功'   
#                    print(out)

#生成宏信息脚本1
#此脚本涉及隐私，不方便透露

#生成宏信息脚本2
#此脚本涉及隐私，不方便透露

#定时任务
*/3 8-23 * * * cd /home/deploy/app/update_monitor_hong/project; python3 process_show.py > /home/deploy/app/update_monitor_hong/logs/process_show_epic.log

代码4：供数脚本

#更新服务器磁盘，内存信息脚本
# -*- coding: utf-8 -*-
import pymongo
from pymongo import MongoClient
from zabbix_api import ZabbixAPI


zapi = ZabbixAPI(server='http:///', path="", log_level=0)
zapi.login('xxx', 'xxx')


def get_all_mem(host):
    value_tmem = zapi.item.get({"filter": {"host": host ,"name": 'Total memory'}})[0]['lastvalue']
    value_last = change_size(value_tmem ,'no')
    return(value_last)
    
def get_all_disk(host):
    value_tdisk = zapi.item.get({"filter": {"host": host ,"name": 'DISK /home'}})[0]['lastvalue']
    value_last = change_size(value_tdisk ,'no')
    return(value_last)

def change_size(size,panduan):
    size = str(size)
    size = int(size)
    if panduan == 'yes':
        return(round(size/1024/1024/1024,2))
    else:
        return(round(size/1024/1024/1024))


#client = pymongo.MongoClient('172.19.192.23')
#db = client['test']
#db.authenticate('Mark','Cad__448')
#table = db['serverpool']
#list_all = []
#json_all = table.find({})
#for ip_info in json_all:
#    ip = ip_info['_id']
#    disk = get_all_disk(ip)
#    mem = get_all_mem(ip)
#    table.update({"_id": ip },{"$set":{"mem": mem }})
#    table.update({"_id": ip },{"$set":{"disk": disk }})

def create_usergroup(group_name):
    groupid = get_group_id(group_name)
    zapi.usergroup.create({"name": group_name, "rights": [{"permission":'2', "id": groupid }] })

def create_user(group_name):
    usergroupid = get_usergroupid(group_name)
    groupid = get_group_id(group_name)
    url = 'http://xxx/latest.php?groupids%5B%5D=' + groupid + '&application=&select=&show_without_data=1&filter_set=1'
    zapi.user.create({"alias": group_name, "passwd": group_name, "usrgrps": [{"usrgrpid": usergroupid}],"url": url})

def get_usergroup():
    all_usergroup_list = []
    all_usergroups = zapi.usergroup.get({"output": "extend"})
    for i in range(len(all_usergroups)):
        all_usergroup_list.append(all_usergroups[i]['name'])
    return(all_usergroup_list)    

def get_usergroupid(group_name):
    usergroupid = zapi.usergroup.get({"filter": {"name": group_name}})[0]['usrgrpid']
    return(usergroupid)    

def get_user():
    all_users_list = []
    all_users = zapi.user.get({"output": "extend"})    
    for i in range(len(all_users)):
        all_users_list.append(all_users[i]['alias'])      
    return(all_users_list)

def get_all_zabbix_list():
    all_zabbix_groups_list = []
    all_zabbix_group_info = zapi.hostgroup.get({"output": "extend"})
    for i in range(len(all_zabbix_group_info)):
        all_zabbix_groups_list.append(all_zabbix_group_info[i]['name'])
    return(all_zabbix_groups_list)

def get_group_inventory_list(group_name):
    if group_name == 'EPIC_servers':
        group_name = 'idle'
    group_ip_list = [] #组 ip列表
    one_group_info = table.find({"status": group_name }, {"status": "1"})
    for ip_info in one_group_info:
        group_ip_list.append(ip_info['_id'])
    return(group_ip_list)

def get_zabbix_all_hosts():
    all_zabbix_allgroup_hosts_list = []
    all_zabbix_allhosts_info = zapi.host.get({"output": "extend", })
    for i in range(len(all_zabbix_allhosts_info)):
        all_zabbix_allgroup_hosts_list.append(all_zabbix_allhosts_info[i]['host'])
    return(all_zabbix_allgroup_hosts_list)

def get_group_id(group_name):
    groupid = zapi.hostgroup.get({"filter": {"name": group_name}})[0]['groupid']
    return groupid

def get_host_id(host_name):
    hostid = zapi.host.get({"filter": {"host": host_name}})[0]['hostid']
    return hostid


def get_group_hosts_list(group_name):
    group_id = get_group_id(group_name)
    group_hosts_list = []
    group_hosts_info = zapi.host.get({"selectGroups": "extend", "groupids": [group_id]})
    for i in range(len(group_hosts_info)):
        group_hosts_list.append(group_hosts_info[i]['host'])
    return group_hosts_list


def create_group(group_name):
    zapi.hostgroup.create({"name": group_name})


def change_host_group(host_name, group_name):
    host_id = get_host_id(host_name)
    group_id = get_group_id(group_name)
    zapi.host.update({"hostid": host_id, "groups": [group_id]})




def get_host_id(host_name):
    hostid = zapi.host.get({"filter": {"host": host_name}})[0]['hostid']
    return hostid

def get_group_id(group_name):
    groupid = zapi.hostgroup.get({"filter": {"name": group_name}})[0]['groupid']
    return groupid

def create_host(host_name,group_name):
    group_id = get_group_id(group_name)
    EPIC_templateid = zapi.template.get({"filter": {"host": "Linux OS EPIC"}})[0]['templateid']
    zapi.host.create({"host": host_name, "interfaces": [
            {"type": 1, "main": 1, "useip": 1, "ip": host_name, "dns": "", "port": "10050"}],
                               "groups": [{"groupid": group_id}], "tags": [{"tag": "Host name", "value": host_name}],
                               "templates": [{"templateid": EPIC_templateid}]})
       
def update_group(group_name):
    all_zabbix_groups_list = get_all_zabbix_list()
    group_inventory_list = get_group_inventory_list(group_name)
    zabbix_all_hosts = get_zabbix_all_hosts()
#    group_hosts_list = get_group_hosts_list(group_name)
    if group_name not in all_zabbix_groups_list: #组不存在则判断是否该创建                      
       if get_group_inventory_list(group_name): #组存在机器
           create_group(group_name)
       group_hosts_list = get_group_hosts_list(group_name)    
    else:
       group_hosts_list = get_group_hosts_list(group_name)
       if group_inventory_list: 
           for inventory_host in group_inventory_list:
               if inventory_host in zabbix_all_hosts: #机器存在于zabbix总览
                   if inventory_host not in group_hosts_list: #机器不存在于对应表
                       change_host_group(inventory_host, group_name)
               else:
                   create_host(inventory_host, group_name)
           if group_name != 'EPIC_servers':
               for zabbix_group_host in group_hosts_list: #循环zabbix组机器
                   if zabbix_group_host not in group_inventory_list: #如果不在inventory,送去 idel
                       change_host_group(zabbix_group_host,'EPIC_servers')                                 
           else: 
               for zabbix_group_host in group_hosts_list:
                   if zabbix_group_host not in group_inventory_list:
                       change_host_group(zabbix_group_host,'guer_servers')    

client = pymongo.MongoClient('xxx')
db = client['xxx']
db.authenticate('xx','xxxx')
table = db['serverpool']

all_used_group = []
all_user_info = table.find({})
for i in all_user_info:
    group_name = i['status']
    all_used_group.append(group_name)
all_used_group_2 = {}.fromkeys(all_used_group).keys()
print(all_used_group_2)
for group_name in all_used_group_2:
    if group_name == 'idle':
        group_name = 'EPIC_servers'
    print(group_name)
    update_group(group_name)
    all_used_group.append(group_name)

list_all = []
json_all = table.find({})
for ip_info in json_all:
    ip = ip_info['_id']
    disk = get_all_disk(ip)
    mem = get_all_mem(ip)
    print(ip)
    print(disk)
    print(mem)
    table.update({"_id": ip },{"$set":{"mem": mem }})
    table.update({"_id": ip },{"$set":{"disk": disk }})

usergroups = get_usergroup()
users = get_user()
for group_name in all_used_group_2:
    if group_name == 'idle':
        group_name = 'EPIC_servers'
    if group_name not in usergroups:
        create_usergroup(group_name)
    if group_name not in users:
        create_user(group_name)

#提供 服务器 内存，磁盘，进程监控状态脚本
from zabbix_api import ZabbixAPI
import os
import pymongo
from pymongo import MongoClient
from multiprocessing import Process,Lock
import threading

zapi = ZabbixAPI(server='http://xxx/', path="", log_level=0)
zapi.login('xxx', 'xxx')

client = pymongo.MongoClient('xxxx')
db = client['xxx']
db.authenticate('xxx','xxx')
table = db['serverpool']


#global ip_info_lists


def change_size(size,panduan):
    size = str(size)
    size = int(size)
    if panduan == 'yes':
        return(round(size/1024/1024/1024,2))
    else:
        return(round(size/1024/1024/1024))

def get_group_inventory_list(group_name):
    if group_name == 'EPIC_servers':
        group_name = 'idle'
    group_ip_list = [] #组 ip列表
    one_group_info = table.find({"status": group_name }, {"status": "1"})
    for ip_info in one_group_info:
        group_ip_list.append(str(ip_info['_id']))
    return(group_ip_list)

def get_ip_jsons(group_name):
    group_ip_list = get_group_inventory_list(group_name)
    total_jsons = {}
    ip_info_lists = []
    for ip in group_ip_list:
        dir_cmd = '/home/devops/app/epic_update_monitor/monitor_value/' + ip
        if os.path.exists(dir_cmd):
#            runGetStatusThread = threading.Thread(target=get_ip_info,args=(ip ,))
#            runGetStatusThread.start()
            
            ip_info_lists.append(get_ip_info(ip))
    total_jsons['data'] = ip_info_lists
    return(total_jsons)

def get_monitor_value(ip,monitor):
    try:
        monitor_value = zapi.item.get({"filter": {"host": ip ,"name": monitor}})[0]['lastvalue']
#        logger.warning('get ' + ip + ' ' + monitor + ' items info success')
        return(monitor_value)
    except Exception as e:
#        logger.warning('get ' + ip + ' ' + monitor + ' items info fail')
#        logger.warning(str(e))
        return('none')

def get_ip_info(ip):
    all = zapi.item.get({"filter": {"host": ip}})
    all_items = []
    all_items_value = {}
    ip_monitor_values = {}
    process_monitor_lists = []
    for i in range(len(all)):
        name = all[i]['name']
        all_items.append(name)    
        lastvalue = all[i]['lastvalue']
        all_items_value[name] = lastvalue

    value_tmem = all_items_value['Available memory']
    value_tmem_last = change_size(value_tmem ,'yes')

    value_tdisk = all_items_value['DISK /home free']
    value_tdisk_last = change_size(value_tdisk ,'yes')

    if value_tmem_last > int(5):
        if value_tdisk_last > int(5):
            machine_status = int(1)
            statusDesc = 'normal'
        else:
            machine_status = int(0)
            statusDesc = 'available disk less 5G'
    else:
        if value_tdisk_last > int(5):
            machine_status = int(0)
            statusDesc = 'availabe memory less 5G'    
        else:
            machine_status = int(0)
            statusDesc = 'availabe memory and disk less 5G'

    ip_monitor_values['ip'] = ip
    ip_monitor_values['memory'] = value_tmem_last
    ip_monitor_values['disk'] = value_tdisk_last
    ip_monitor_values['machine_status'] = machine_status
    ip_monitor_values['statusDesc'] = statusDesc

    file = '/home/devops/app/epic_update_monitor/monitor_value/' + ip
    for line in open(file):
        line
        process_monitor_json = {}
        length = len(line.split(" "))
        process = line.split(" ")[3].replace('\n','').replace('\r','')
        if length > int(4):
            port = line.split(" ")[4].replace('\n','').replace('\r','')
        else:
            port = 'None'   
        
        if port == 'None':
            process_monitor = 'process ' + process + ' status'
            if process_monitor not in all_items:
                continue
            value_process = int(all_items_value[process_monitor])
            value_port = 'None'
            portStatus = 'None'
            if value_process > int(0):
                processStatus = int(1)  
                processDesc = 'online'
            else:
                processStatus = int(0)
                processDesc = 'close'
        else:
            port_monitor = 'port ' + port + ' status on ' + process
            if port_monitor not in all_items:
                continue 
            value_port = int(all_items_value[port_monitor])
            if value_port == int(1):
                value_process = int(1)
                processStatus = int(1)
                portStatus = int(1)
                processDesc = 'online'
            else:
                value_process = int(0)
                processStatus = int(0)
                portStatus = int(0)
                processDesc = 'close'

        changemq_monitor = 'process ' + process + ' changemq_status'
        if 'node-modules' in process or 'mq_ip' in process or 'redis_ip' in process or 'msc_nginx_vip' in process or 'rmb-msc-pc' in process or 'cut-release' in process:
            process_monitor_json['changemq_status'] = 'None'
        elif changemq_monitor not in all_items:
            process_monitor_json['changemq_status'] = 'no'
        else:
            value_changemq_monitor = int(all_items_value[changemq_monitor]) 
            if value_changemq_monitor == int(1):
                process_monitor_json['changemq_status'] = 'yes'
            else:
                process_monitor_json['changemq_status'] = 'no'
#            try:
#                value_changemq_monitor = int(get_monitor_value(ip,changemq_monitor))
#                if value_changemq_monitor == int(1):
#                    process_monitor_json['changemq_status'] = 'yes'
#                else:
#                    process_monitor_json['changemq_status'] = 'no'
#            except:
#                process_monitor_json['changemq_status'] = 'no'




     
        process_monitor_json['name'] = process
        process_monitor_json['processStatus'] = value_process
        process_monitor_json['port'] = port
        process_monitor_json['portStatus'] = value_port 
        process_monitor_json['processDesc'] = processDesc
        process_monitor_lists.append(process_monitor_json)



#nas
    nas_dict = {}
    value_nfs = int(get_monitor_value(ip,'nfs status'))
    if value_nfs > int(0):
        nfsStatus = int(1)
        nfsDesc = 'online'
    else:
        nfsStatus = int(0)
        nfsDesc = 'close'
    nas_dict['name'] = 'nfs'
    nas_dict['processStatus'] = value_nfs
    nas_dict['port'] = '2049'
    nas_dict['portStatus'] = value_nfs
    nas_dict['processDesc'] = nfsDesc
    process_monitor_lists.append(nas_dict)


    ip_monitor_values['process'] = process_monitor_lists
    return(ip_monitor_values)
#    ip_info_lists.append(ip_monitor_values)

#抽数接口脚本
# -*- coding: UTF-8 -*-
from flask import Flask, render_template, request, redirect, url_for
from flask_cors import CORS
import update_export
import json
import update_monitor_value
import paramiko
import update_ST_monitor_value
import total_env_info


ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect( 'xxxx','22','xxx','xxxx' )



app = Flask(__name__)
show_time = update_export
show_monitor = update_monitor_value
show_st_monitor = update_ST_monitor_value
show_diskmem_monitor = total_env_info
@app.route('/api/EPIC_show/free_machine',methods=['POST', 'GET'])
def show_free_info():
    if request.method == 'POST':
        group_name = 'idle'
        data = show_time.update_mongo(group_name)
        return(json.dumps(data))


@app.route('/api/EPIC_show/group_name',methods=['POST', 'GET'])
def show_group_info():
    if request.method == 'POST':
        group_name = request.json['group_name']
        data = show_time.update_mongo(group_name)
        return(json.dumps(data))

@app.route('/api/EPIC_show/free_oracle_machine',methods=['POST', 'GET'])
def show_oracle_free_info():
    if request.method == 'POST':
        group_name = 'idle'
        data = show_time.update_oracle_mongo(group_name)
        return(json.dumps(data))


@app.route('/api/EPIC_show/group_oracle_name',methods=['POST', 'GET'])
def show_oracle_group_info():
    if request.method == 'POST':
        group_name = request.json['group_name']
        data = show_time.update_oracle_mongo(group_name)
        return(json.dumps(data))

@app.route('/api/EPIC_show/url_info',methods=['POST', 'GET'])
def show_zabbix_url_info():
    if request.method == 'POST':
        group_name = request.json['group_name']
        data = show_time.show_url_info(group_name)
        return(json.dumps(data))

@app.route('/api/EPIC_show/monitor_info',methods=['POST', 'GET'])
def show_zabbix_monitor_info():
    if request.method == 'POST':
        group_name = request.json['group_name']
        data = show_monitor.get_ip_jsons(group_name)
        return(json.dumps(data))

@app.route('/api/EPIC_show/st_monitor_info',methods=['POST', 'GET'])
def show_zabbix_st_monitor_info():
    if request.method == 'POST':
        group_name_list = ['ST-2','ST-4','MEM','UAT-1','UAT-2','UAT-3','UAT-4']
        group_name = request.json['group_name']
        if group_name not in group_name_list:
            data = {'info':'not current env'}
        else:
            data = show_st_monitor.get_group_info(group_name)

        return(json.dumps(data))

@app.route('/api/EPIC_show/st_memdisk_info',methods=['POST', 'GET'])
def show_zabbix_st_memdisk_info():
    if request.method == 'POST':
        group_name_list = ['ST-2','ST-4','MEM','UAT-1','UAT-2','UAT-3','UAT-4']
        group_name = request.json['group_name']
        if group_name not in group_name_list:
            data = {'info':'not current env'}
        else:
            data = show_diskmem_monitor.get_group_info(group_name)

        return(json.dumps(data))


@app.route('/api/EPIC_show/repair',methods=['POST', 'GET'])
def repair():
    #{group:ST-79234 ,process:all/tbs-dp-report ,host:172.17.196.89 ,}  
    if request.method == 'POST':
        env = request.json['group_name']
        process = request.json['process']     
        host = request.json['host']
        start_script = '/home/deploy/app/epic_update_monitor/project/process_start/all_start.sh'
        start_log = '/home/deploy/app/epic_update_monitor/project/process_start/logs/start_' + env + '.log'
#        start_command = 'bash ' + start_script + ' ' + env + ' ' + process + ' ' + host + ' >> ' + start_log + ' ' + '2>&1'
        start_command = 'bash ' + start_script + ' ' + env + ' ' + process + ' ' + host + ' >> ' + start_log
        try:
            ssh.exec_command(start_command)
            xxxx = {}
            xxxx['info'] = 'repair ' + process + ' success'
            return(json.dumps(xxxx))
        except Exception as e:
            yyyy = {}
            yyyy['info'] = 'repair fail'
            return(json.dumps(yyyy))

if __name__ == '__main__':
    CORS(app, supports_credentials=True)
    app.run(host='0.0.0.0', port="30852",debug=True)

利用zabbix完成一套完整的自动化监控
前景：开发环境巨多（并行环境10个以上）设计一套自动化监控系统监控项涉及：服务器内存，/home挂载盘，na...
zabbix
自动化运维技术--运维框架自动化监控 zabbix architecture1 zabbix architect...
Zabbix使用LLD自动发现规则发现监控docker容器（下）
本篇是使用Zabbix监控docker容器下篇。利用ZABBIX自动发现监控功能，在部署zabbix agent客...
Oracle数据库添加到Zabbix监控
使用zabbix监控oracle数据库一、概述本文将给大家介绍如何利用zabbix+python监控oracl...
zabbix 小知识
完整的Zabbix监控流程： Agentd需要安装到被监控的主机上，它负责定期收集各项数据，并发送到zabbix ...
zabbix使用percona-monitoring-plugi
摘要利用 Zabbix 监控MySQL的时候，zabbix官网自带的插件只提供了很少的监控参数，对于运维和DBA...
【zabbix】zabbix配置邮件告警通知
Zabbix监控服务端、客户端都已经部署完成，被监控主机已经添加，Zabiix监控运行正常，通过查看Zabbix监...
Linux监控平台介绍、zabbix监控介绍、安装zabbix（
Linux监控平台介绍 Linux监控平台介绍 zabbix监控介绍 zabbix监控介绍zabbix监控介绍安...
zabbix 键值
zabbix键值简介： zabbix监控项是从主机收集的数据信息，zabbix监控项都是基于键值进行监控，键值有包...
zabbix搭建安全服务器集群
Zabbix介绍 Zabbix监控组件主要包括：Zabbix Server、Zabbix Proxy、Zabbix...