Kafka监控

CMAK

当前环境：

[$] java -version
openjdk version "11.0.9" 2020-10-20 LTS
OpenJDK Runtime Environment Zulu11.43+21-CA (build 11.0.9+11-LTS)
OpenJDK 64-Bit Server VM Zulu11.43+21-CA (build 11.0.9+11-LTS, mixed mode)

下载并安装：

git clone https://github.com/yahoo/CMAK.git && cd CMAK
./sbt clean dist
unzip target/universal/cmak-3.0.0.5.zip && cd cmak-3.0.0.5
vim conf/application.conf

修改关键配置：

kafka-manager.zkhosts="127.0.0.1:2181"
kafka-manager.zkhosts=${?ZK_HOSTS}
cmak.zkhosts="127.0.0.1:2181"

浏览器访问页面：

http://127.0.0.1:9000/

Burrow

Go环境搭建：

curl https://dl.google.com/go/go1.15.5.linux-amd64.tar.gz > go1.15.5.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.15.5.linux-amd64.tar.gz

设置环境变量：

[#] vim /etc/profile
export PATH=$PATH:/usr/local/go/bin
[#] source /etc/profile

验证：

[#] go version
go version go1.15.5 linux/amd64

下载Burrow源码，并安装依赖：

git clone https://github.com/linkedin/Burrow.git && cd Burrow
go mod tidy

如果速度缓慢，可以配置Go国内镜像：

[#] vim /etc/profile
export GOPROXY=https://mirrors.aliyun.com/goproxy/
[#] source /etc/profile

执行编译：

go install

修改配置：

vim config/burrow.toml

[general]
pidfile="burrow.pid"
stdout-logfile="burrow.out"
access-control-allow-origin="mysite.example.com"

[logging]
filename="logs/burrow.log"
level="info"
maxsize=100
maxbackups=30
maxage=10
use-localtime=false
use-compression=true

[zookeeper]
servers=[ "127.0.0.1:2181" ]
timeout=6
root-path="/burrow"

[client-profile.test]
client-id="burrow-test"
kafka-version="0.10.0"

[cluster.local]
class-name="kafka"
servers=[ "127.0.0.1:9092" ]
client-profile="test"
topic-refresh=120
offset-refresh=30

[consumer.local]
class-name="kafka"
cluster="local"
servers=[ "127.0.0.1:9092" ]
client-profile="test"
group-denylist="^(console-consumer-|python-kafka-consumer-|quick-).*$"
group-allowlist=""

[consumer.local_zk]
class-name="kafka_zk"
cluster="local"
servers=[ "127.0.0.1:2181" ]
zookeeper-path="/kafka-cluster"
zookeeper-timeout=30
group-denylist="^(console-consumer-|python-kafka-consumer-|quick-).*$"
group-allowlist=""

[httpserver.default]
address=":8000"

[storage.default]
class-name="inmemory"
workers=20
intervals=15
expire-group=604800
min-distance=1

#[notifier.default]
#class-name="http"
#url-open="http://someservice.example.com:1467/v1/event"
#interval=60
#timeout=5
#keepalive=30
#extras={ api_key="REDACTED", app="burrow", tier="STG", fabric="mydc" }
#template-open="conf/default-http-post.tmpl"
#template-close="conf/default-http-delete.tmpl"
#method-close="DELETE"
#send-close=true
#threshold=1

运行：

[#] go env | grep "GOPATH"
GOPATH="/root/go"
[#] /root/go/bin/Burrow --config-dir ./config

执行请求，查看Kafka集群状态：

curl -s http://localhost:8000/v3/kafka | python3 -m json.tool

{
    "error": false,
    "message": "cluster list returned",
    "clusters": [
        "local"
    ],
    "request": {
        "url": "/v3/kafka",
        "host": "remer"
    }
}

查看消费列表状态：

curl -s http://localhost:8000/v3/kafka/local/consumer | python3 -m json.tool

{
    "error": false,
    "message": "consumer list returned",
    "consumers": [
        "console-consumer-91113",
        "burrow-local",
        "messageGroup"
    ],
    "request": {
        "url": "/v3/kafka/local/consumer",
        "host": "remer"
    }
}

查看消费状态：

curl -s http://localhost:8000/v3/kafka/local/consumer/group-id/status | python3 -m json.tool

{
    "error": false,
    "message": "consumer status returned",
    "status": {
        "cluster": "local",
        "group": "group-id",
        "status": "NOTFOUND",
        "complete": 1,
        "partitions": [],
        "partition_count": 0,
        "maxlag": null,
        "totallag": 0
    },
    "request": {
        "url": "/v3/kafka/local/consumer/group-id/status",
        "host": "remer"
    }
}

更多参考：Burrow-HTTP-Endpoint

参考

Burrow-HTTP-Endpoint
Kafka监控工具汇总
CMAK