java 业务指标监控

作者: 天草二十六_简村人 | 来源:发表于2020-04-21 15:46 被阅读0次

java 业务指标监控
监控笔记
监控基础概念
业务监控运维体系架构
Zabbix 服务器性能指标参考(学习笔记十七)
《数据思维》/03指标体系
《Prometheus监控实战》读书笔记
我们自搭监控系统的grafana的视图设计
数据如何驱动业务增长
CAT使用总结

1、总体设计

image.png

业务方使用Slf4jReporter生成指标数据到日志文件里，后面就交由服务端来接收和处理并展示了。

2、引入jar包

<dependency>
            <groupId>io.dropwizard.metrics</groupId>
            <artifactId>metrics-core</artifactId>
        </dependency>

        <dependency>
            <groupId>io.dropwizard.metrics</groupId>
            <artifactId>metrics-annotation</artifactId>
        </dependency>

        <dependency>
            <groupId>com.github.davidb</groupId>
            <artifactId>metrics-influxdb</artifactId>
            <version>0.9.3</version>
        </dependency>

3、logback-spring.xml增加日志输入

<appender name="metricsFILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <FileNamePattern>${LOG_HOME}/metrics_%d{yyyyMMdd}.log</FileNamePattern>
            <!--日志文件保留天数-->
            <MaxHistory>7</MaxHistory>
        </rollingPolicy>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%d{HH:mm:ss} %-5p %c:%L - %m%n</pattern>
        </encoder>
        <!--日志文件最大的大小-->
        <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
            <MaxFileSize>${FILE_SIZE}MB</MaxFileSize>
        </triggeringPolicy>
    </appender>

    <logger name="metrics" level="INFO" additivity="false">
        <appender-ref ref="metricsFILE" />
    </logger>

4、指标配置类MetricConfig

import com.codahale.metrics.*;
import metrics_influxdb.HttpInfluxdbProtocol;
import metrics_influxdb.InfluxdbReporter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.concurrent.TimeUnit;

@Configuration
public class MetricConfig {

    @Bean
    public MetricRegistry registry() {
        return new MetricRegistry();
    }

    @Bean
    public Slf4jReporter slf4jReporter(MetricRegistry registry) {
        Slf4jReporter reporter = Slf4jReporter.forRegistry(registry)
                .convertRatesTo(TimeUnit.SECONDS)
                .convertDurationsTo(TimeUnit.MILLISECONDS)
                .build();

        reporter.start(10, TimeUnit.SECONDS);
        return reporter;
    }

    /**
     * TPS 计算器
     *
     * @param registry
     * @return
     */
    @Bean
    public Meter requestMeter(MetricRegistry registry) {
        return registry.meter("request");
    }

    /**
     * 直方图
     *
     * @param registry
     * @return
     */
    @Bean
    public Histogram responseSizes(MetricRegistry registry) {
        return registry.histogram("response-sizes");
    }

    /**
     * 计数器
     *
     * @param registry
     * @return
     */
    @Bean
    public Counter pendingJobs(MetricRegistry registry) {
        return registry.counter("requestCount");
    }

    /**
     * 计时器
     *
     * @param registry
     * @return
     */
    @Bean
    public Timer responses(MetricRegistry registry) {
        return registry.timer("executeTime");
    }


    @Bean(name = "influxdbReporter")
    public ScheduledReporter influxdbReporter(MetricRegistry registry) {
        ScheduledReporter reporter = InfluxdbReporter.forRegistry(registry)
                .protocol(new HttpInfluxdbProtocol("http",
                        "localhost",
                        8086, "", "", "metrics"))
                .convertRatesTo(TimeUnit.SECONDS)
                .convertDurationsTo(TimeUnit.MILLISECONDS)
                .filter(MetricFilter.ALL)
                .skipIdleMetrics(false)
                .build();

        reporter.start(10, TimeUnit.SECONDS);

        return reporter;
    }
}

5、使用示例：

   @Resource
    private Meter requestMeter;

    @Resource
    private Histogram responseSizes;

    @Resource
    private Counter pendingJobs;

    @Resource
    private Timer responses;

    @RequestMapping("/test/sign")
    @ResponseBody
    public String getSign(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        requestMeter.mark();

        pendingJobs.inc();

        responseSizes.update(new Random().nextInt(10));

        final Timer.Context context = responses.time();
        try {
            return "success";
        } finally {
            context.stop();
        }
    }

效果见下图：

image.png

每隔10秒打印出指标数据的日志。

19:11:37 INFO  metrics:373 - type=COUNTER, name=requestCount, count=0
19:11:37 INFO  metrics:373 - type=HISTOGRAM, name=response-sizes, count=0, min=0, max=0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0
19:11:37 INFO  metrics:373 - type=METER, name=request, count=0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second
19:11:37 INFO  metrics:373 - type=TIMER, name=executeTime, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
19:11:47 INFO  metrics:373 - type=COUNTER, name=requestCount, count=1
19:11:47 INFO  metrics:373 - type=HISTOGRAM, name=response-sizes, count=1, min=5, max=5, mean=5.0, stddev=0.0, median=5.0, p75=5.0, p95=5.0, p98=5.0, p99=5.0, p999=5.0
19:11:47 INFO  metrics:373 - type=METER, name=request, count=1, mean_rate=0.04772524178633343, m1=0.014712537947741825, m5=0.0032510706679223173, m15=0.0011018917421948848, rate_unit=events/second
19:11:47 INFO  metrics:373 - type=TIMER, name=executeTime, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
19:11:57 INFO  metrics:373 - type=COUNTER, name=requestCount, count=1
19:11:57 INFO  metrics:373 - type=HISTOGRAM, name=response-sizes, count=1, min=5, max=5, mean=5.0, stddev=0.0, median=5.0, p75=5.0, p95=5.0, p98=5.0, p99=5.0, p999=5.0
19:11:57 INFO  metrics:373 - type=METER, name=request, count=1, mean_rate=0.032306681295300385, m1=0.012453894499523116, m5=0.003144487893819254, m15=0.0010897162674033895, rate_unit=events/second
19:11:57 INFO  metrics:373 - type=TIMER, name=executeTime, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
19:12:07 INFO  metrics:373 - type=COUNTER, name=requestCount, count=2
19:12:07 INFO  metrics:373 - type=HISTOGRAM, name=response-sizes, count=2, min=5, max=7, mean=6.156210465768618, stddev=0.987723792557591, median=7.0, p75=7.0, p95=7.0, p98=7.0, p99=7.0, p999=7.0
19:12:07 INFO  metrics:373 - type=METER, name=request, count=2, mean_rate=0.04883621909301433, m1=0.025254532045303885, m5=0.0062924699865950515, m15=0.002179567069063387, rate_unit=events/second
19:12:07 INFO  metrics:373 - type=TIMER, name=executeTime, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
19:12:17 INFO  metrics:373 - type=COUNTER, name=requestCount, count=2
19:12:17 INFO  metrics:373 - type=HISTOGRAM, name=response-sizes, count=2, min=5, max=7, mean=6.156210465768618, stddev=0.987723792557591, median=7.0, p75=7.0, p95=7.0, p98=7.0, p99=7.0, p999=7.0
19:12:17 INFO  metrics:373 - type=METER, name=request, count=2, mean_rate=0.03925160662498755, m1=0.02137749984701412, m5=0.006086178282834525, m15=0.0021554837014424494, rate_unit=events/second
19:12:17 INFO  metrics:373 - type=TIMER, name=executeTime, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
19:12:27 INFO  metrics:373 - type=COUNTER, name=requestCount, count=2
19:12:27 INFO  metrics:373 - type=HISTOGRAM, name=response-sizes, count=2, min=5, max=7, mean=6.156210465768618, stddev=0.987723792557591, median=7.0, p75=7.0, p95=7.0, p98=7.0, p99=7.0, p999=7.0
19:12:27 INFO  metrics:373 - type=METER, name=request, count=2, mean_rate=0.032810010401197694, m1=0.018095662944349354, m5=0.00588664962556148, m15=0.0021316664456582147, rate_unit=events/second
19:12:27 INFO  metrics:373 - type=TIMER, name=executeTime, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
19:12:37 INFO  metrics:373 - type=COUNTER, name=requestCount, count=2
19:12:37 INFO  metrics:373 - type=HISTOGRAM, name=response-sizes, count=2, min=5, max=7, mean=6.156210465768618, stddev=0.987723792557591, median=7.0, p75=7.0, p95=7.0, p98=7.0, p99=7.0, p999=7.0
19:12:37 INFO  metrics:373 - type=METER, name=request, count=2, mean_rate=0.028187596113694386, m1=0.01531764798217201, m5=0.005693662295739435, m15=0.0021081123612784834, rate_unit=events/second
19:12:37 INFO  metrics:373 - type=TIMER, name=executeTime, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
19:12:47 INFO  metrics:373 - type=COUNTER, name=requestCount, count=2
19:12:47 INFO  metrics:373 - type=HISTOGRAM, name=response-sizes, count=2, min=5, max=7, mean=6.156210465768618, stddev=0.987723792557591, median=7.0, p75=7.0, p95=7.0, p98=7.0, p99=7.0, p999=7.0
19:12:47 INFO  metrics:373 - type=METER, name=request, count=2, mean_rate=0.024705614814530406, m1=0.012966109085216199, m5=0.005507001843146522, m15=0.0020848185403616864, rate_unit=events/second
19:12:47 INFO  metrics:373 - type=TIMER, name=executeTime, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
19:12:57 INFO  metrics:373 - type=COUNTER, name=requestCount, count=4
19:12:57 INFO  metrics:373 - type=HISTOGRAM, name=response-sizes, count=4, min=0, max=7, mean=2.0488679402655547, stddev=2.6066574932478366, median=1.0, p75=5.0, p95=7.0, p98=7.0, p99=7.0, p999=7.0
19:12:57 INFO  metrics:373 - type=METER, name=request, count=4, mean_rate=0.04397865428643308, m1=0.01097557438357367, m5=0.005326460848075398, m15=0.0020617821070978764, rate_unit=events/second
19:12:57 INFO  metrics:373 - type=TIMER, name=executeTime, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
19:13:07 INFO  metrics:373 - type=COUNTER, name=requestCount, count=4
19:13:07 INFO  metrics:373 - type=HISTOGRAM, name=response-sizes, count=4, min=0, max=7, mean=2.0488679402655547, stddev=2.6066574932478366, median=1.0, p75=5.0, p95=7.0, p98=7.0, p99=7.0, p999=7.0
19:13:07 INFO  metrics:373 - type=METER, name=request, count=4, mean_rate=0.03962227676951814, m1=0.03871569903135633, m5=0.011653980026690198, m15=0.0042427837018434575, rate_unit=events/second
19:13:07 INFO  metrics:373 - type=TIMER, name=executeTime, count=0, min=0.0, max=0.0, mean=0.0, stddev=0.0, median=0.0, p75=0.0, p95=0.0, p98=0.0, p99=0.0, p999=0.0, mean_rate=0.0, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds

后期补充下后面几个技术栈。

java 业务指标监控
1、总体设计业务方使用Slf4jReporter生成指标数据到日志文件里，后面就交由服务端来接收和处理并展示了。...
监控笔记
监控分类系统业务监控方法探针（probing）内省（introspection）监控指标指标设计方法...
监控基础概念
监控指标：硬件指标软件指标业务指标监控系统：采样：sensor（传感器）存储展示报警采样：周期...
业务监控运维体系架构
业务大盘业务指标和系统指标、服务指标不同，需要业务方根据不同的业务自行上报监控数据。业务大盘作为业务运维系统的使...
Zabbix 服务器性能指标参考(学习笔记十七)
服务器性能及业务监控指标，可根据实际情况调整，下表供大家参考。监控类别监控子对象监控维度监控指标警告严重服务器...
《数据思维》/03指标体系
什么是指标体系？指标体系是从不同维度梳理业务，把指标有系统地组织起来。指标体系有什么用？ • 监控业务情况； ...
《Prometheus监控实战》读书笔记
Prometheus是一个开源的监控系统。一、监控简介监控不仅仅只是系统的技术指标，还可以是业务指标。确保为客...
我们自搭监控系统的grafana的视图设计
一、监控体系业务指标监控应用监控（jvm、接口访问量及耗时、线程池、数据库连接池、hystrix）中间件--...
数据如何驱动业务增长
一、报表体系 1、报表体系的意义数据报表体系是指在业务每天推进的过程中、对业务指标的监控、对业务流程的监控、对业...
CAT使用总结
简介 CAT基于Java开发的实时应用监控平台，包括实时应用监控，业务监控。 CAT支持的监控消息类型包括： Tr...