简介
分布式系统
是由一组通过网络进行通信、为了完成共同的任务而协调工作的计算机节点组成的系统。分布式系统的出现是为了用廉价的、普通的机器完成单个计算机无法完成的计算、存储任务。其目的是利用更多的机器，处理更多的数据。
特性：
分布性、对等性、并发性、缺乏全局时钟、故障必然会发生
典型问题：
通信异常、网络分区、3态(成功、失败、超时)、节点故障
CAP原则又称CAP定理
指的是在一个分布式系统中，Consistency（一致性）、 Availability（可用性）、Partition tolerance（分区容错性），三者不可兼得
P: 分区容错性
CP 一致性
AP 可用性
BASE原则
BA: 基本可用
S: 软状态
E: 最终一致性
因果一致性、会话一致性、单调读一致性、读己之所写一致性
保证分布式系统一致性的多种协议
2PC: 2Phase-Commit,请求和执行
3PC: 3Phase-Commit,CanCommit-->PreCommit-->DoCommit
Paxos:
Google Chubby, 分布式锁服务，GFS/BigTable都用到了Chubby
分布式协作、元数据存储、Master选择
-------------------------------------------------------------------
-------------------------------------------------------------------
HDFS/HBase,Zookeeper
ZooKeeper是一个开放源码的分布式应用程序协调服务，是Google的Chubby一个开源的实现，它是集群的管理者，监视着集群中各个节点的状态根据节点提交的反馈进行下一步合理操作。最终，将简单易用的接口和性能高效、功能稳定的系统提供给用户
ZK是一个典型的分布式数据一致性解决方案，分布式应用程序可以基于它实现数据的发布/订阅、负载均衡、名称服务、分布式协调/通知、集群管理、Master选举、分布式锁和分布式队列；
基本概念：
集群角色: Leader,Follower,Observer
Leader: 选举产生，读/写；
Follower: 参与选举，可被选举，读服务；
Observer: 参与选举，不可被选举，提供读服务；
会话:zk中，客户端<-->服务端，TCP长链接；
sessionTimeout
数据节点(ZNode):zk数据模型中的数据单元；zk数据都存储在内存中，数据模型为树状结构(ZNode Tree)；每个ZNode都会保存自己的数据于内存中；
持久节点: 仅显示删除才消失
临时节点: 会话终止即自动消失
版本(Version):ZK会为每个ZNode维护一个称之为Stat的数据结构，记录了当前ZNode的3个数据版本
version: 当前版本
cversion: 当前znode的子节点的版本
aversion: 当前znode的ACL的版本
ACL: ZK使用ACL机制进行权限控制
CREATE, READ, WRITE, DELETE, ADMIN
事件监听器(Watcher):
ZK上，由用户指定的触发机制，在某些事件产生时，ZK能够将其通知给相关的客户端；
ZAB协议: Zookeeper Atomic Broadcast, ZK原子广播协议:
ZAB协议中存在3种状态:
Looking
Following
Leading
四个阶段:
选举: election
发现: disco
同步:
广播:
安装：
部署方式：单机模式、伪分布式、分布式
http://zookeeper.apache.org
基于Java语言--jdk环境需要配置
epel仓库
zoo.cfg配置文件的参数，下面有
指定主机的语法格式:
server.ID=IP:port
ID:各主机的数字标识，一般从1开始
IP:各主机的IP
节点信息: stat
cZxid = 0x0 节点被创建时的事务ID
ctime = Thu Jan 01 08:00:00 CST 1970
mZxid = 0x0 节点最后一次被更新的事务ID
mtime = Thu Jan 01 08:00:00 CST 1970
pZxid = 0x0 当前节点下的子节点最后一次被修改时的事务ID
cversion = -1 子节点的版本号
dataVersion = 0 当前节点数据的版本号
aclVersion = 0 acl的版本号，修改节点权限
ephemeralOwner = 0x0 创建临时节点的时候，会有一个sessionId 。 该值存储的就是这个sessionid
dataLength = 0 数据值长度
numChildren = 1 子节点数
Client:
Watcher，一次性地触发通知机制；
当服务器端的节点触发指定事件的时候会触发watcher。服务端会向客户端发送一个事件通知。
一旦触发一次通知后，该watcher就失效
zkCli命令
create,ls,ls2,stat,delete,rmr,get,set,...
监控zk的四字命令
ruok,stat,srvr,conf,cons,wchs,envl...
zoo.cfg配置文件的参数
基本参数
clientPort=2181
dataDir=/data/zookeeper
dataLogDir: 事务日志文件路径；
initLimit=5
tickTime=2000
syncLimit=2
存储配置:
preAllocSize: 为事务日志预先分配的磁盘空间量；默认65535kB；65M
snapCount: 每多少次事务后执行一次快照操作，每事务的平均大小在100字节；
autopurge.snapRetainCount:
autopurge.purgeInterval: purge操作的时间间隔，0表示不启动；
fsync.warningthresholdms: zk进行事务日志fsync操作时消耗的时长报警阈值；
weight.X=N : 判断quorum时投票权限，默认1；
网络配置:
maxClientCnxns: 每客户端IP的最大并发连接数；
clientPortAddress: zk监听IP地址；
minSessionTimeout:
maxSessionTimeout:
集群配置
initLimit: Follower连入Leader并完成数据同步的时长；
syncLimit: 心跳检测的最大延迟；
leaderServes: 默认zk的leader接收读写请求，额外还要负责协调各Follower发来的事务等，因此，为使得leader集中处理zk集群内部信息，建议不让leader直接提供服务；
cnxTimeout: Leader选举期间，各服务器创建TCP连接的超时时长；
ellectionAig: 选举算法，目前仅支持FastLeaderElection算法一种；
server.id=[hostname]:port:port[:observer]
集群内各服务器的属性参数
第一个port: follower与leader进行通信和数据同步时所使用端口；
第二个port: leader选举时使用的端口；
observer: 定义指定的服务器为observer；
注意: 运行为集群模式时，每个节点在其数据目录中应该有一个myid文件，其内容仅为当前server的id；
应用场景
数据发布/订阅
负载均衡
命名服务
分布式协调/通知
集群管理
Master选举
配置
3台主机（一主两从）
192.168.25.11
192.168.25.12
192.168.25.13
centos7 1804
NAT+仅主机
安装包: telnet wget ntpdate net-tools lrzsz vim
安装epel源: epel-release-latest-7.noarch.rpm
集群--时间一定要同步
crontab -e
*/5 * * * * ntpdate time3.aliyun.com && hwclock -w
/etc/hosts
192.168.25.11 node1.fgq.com node1
192.168.25.12 node2.fgq.com node2
192.168.25.13 node3.fgq.com node3
192.168.25.14 node4.fgq.com node4
192.168.25.15 node5.fgq.com node5
3个节点都操作，以node2为例
[root@node2 ~]# mkdir -p /fgq/base-env/
[root@node2 ~]# cd /fgq/base-env/
[root@node2 base-env]# rz
上传 jdk-8u152-linux-x64.tar.gz 和 zookeeper-3.4.13.tar.gz
[root@node2 base-env]# tar zxf jdk-8u152-linux-x64.tar.gz
[root@node2 base-env]# ln -s jdk1.8.0_152 jdk
[root@node2 base-env]# vim /etc/profile
最下方添加信息：
export JAVA_HOME=/fgq/base-env/jdk
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
[root@node2 base-env]# source /etc/profile
[root@node2 base-env]# java -version
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)
[root@node2 base-env]# tar zxf zookeeper-3.4.13.tar.gz
[root@node2 base-env]# ln -s zookeeper-3.4.13 zookeeper
node1上操作单实例
[root@node1 ~]# cd /fgq/base-env/zookeeper/conf
[root@node1 conf]# cp zoo_sample.cfg zoo.cfg
添加信息：
dataDir=/fgq/base-env/zookeeper/data/
dataLogDir=/fgq/base-env/zookeeper/log/
server.1=192.168.25.11:28888:38888
[root@node1 conf]# mkdir /fgq/base-env/zookeeper/{data,log}
[root@node1 conf]# cd ..
[root@node1 zookeeper]# ./bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /fgq/base-env/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@node1 zookeeper]# ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 50 :::2181 :::*
LISTEN 0 50 :::38418 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
[root@node1 zookeeper]# ./bin/zkCli.sh
[root@node1 zookeeper]# ./bin/zkCli.sh
Connecting to localhost:2181
... ...
2019-03-15 11:39:14,722 [myid:] - INFO [main:ZooKeeper@442] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@42110406
Welcome to ZooKeeper!
2019-03-15 11:39:14,743 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1029] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-03-15 11:39:14,822 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
... ...
[zk: localhost:2181(CONNECTED) 0] ?
ZooKeeper -server host:port cmd args
stat path [watch]
set path data [version]
ls path [watch]
delquota [-n|-b] path
ls2 path [watch]
setAcl path acl
setquota -n|-b val path
history
redo cmdno
printwatches on|off
delete path [version]
sync path
listquota path
rmr path
get path [watch]
create [-s] [-e] path data acl
addauth scheme auth
quit
getAcl path
close
connect host:port
[zk: localhost:2181(CONNECTED) 1] ls /
[zookeeper]
# 递归显示
[zk: localhost:2181(CONNECTED) 2] ls2 /
[zookeeper]
cZxid = 0x0
ctime = Thu Jan 01 08:00:00 CST 1970
mZxid = 0x0
mtime = Thu Jan 01 08:00:00 CST 1970
pZxid = 0xb
cversion = 3
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 1
# 创建/test 内容test
[zk: localhost:2181(CONNECTED) 12] create /test "test"
Created /test
[zk: localhost:2181(CONNECTED) 13] ls /
[zookeeper, test]
[zk: localhost:2181(CONNECTED) 14] ls /test
[]
# 显示 /test 属性
[zk: localhost:2181(CONNECTED) 15] stat /test
cZxid = 0x13
ctime = Fri Mar 15 11:51:14 CST 2019
mZxid = 0x13
mtime = Fri Mar 15 11:51:14 CST 2019
pZxid = 0x13
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 4
numChildren = 0
[zk: localhost:2181(CONNECTED) 17] ls2 /
[zookeeper, test]
cZxid = 0x0
ctime = Thu Jan 01 08:00:00 CST 1970
mZxid = 0x0
mtime = Thu Jan 01 08:00:00 CST 1970
pZxid = 0x13
cversion = 4
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 2
[zk: localhost:2181(CONNECTED) 19] ls2 /test
[]
cZxid = 0x13
ctime = Fri Mar 15 11:51:14 CST 2019
mZxid = 0x13
mtime = Fri Mar 15 11:51:14 CST 2019
pZxid = 0x13
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 4
numChildren = 0
# 更新/test数据
[zk: localhost:2181(CONNECTED) 5] get /test
test
cZxid = 0x13
ctime = Fri Mar 15 11:51:14 CST 2019
mZxid = 0x13
mtime = Fri Mar 15 11:51:14 CST 2019
pZxid = 0x13
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 4
numChildren = 0
[zk: localhost:2181(CONNECTED) 6] set /test "test path"
cZxid = 0x13
ctime = Fri Mar 15 11:51:14 CST 2019
mZxid = 0x16
mtime = Fri Mar 15 14:13:52 CST 2019
pZxid = 0x13
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 9
numChildren = 0
[zk: localhost:2181(CONNECTED) 7] get /test
test path
cZxid = 0x13
ctime = Fri Mar 15 11:51:14 CST 2019
mZxid = 0x16
mtime = Fri Mar 15 14:13:52 CST 2019
pZxid = 0x13
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 9
numChildren = 0
# 删除/test
[zk: localhost:2181(CONNECTED) 8] delete /test
[zk: localhost:2181(CONNECTED) 9] ls /
[zookeeper]
# 创建/test子目录
[zk: localhost:2181(CONNECTED) 10] create /test "test application"
Created /test
[zk: localhost:2181(CONNECTED) 11] create /test/node1 "node1.fgq.com"
Created /test/node1
[zk: localhost:2181(CONNECTED) 12] create /test/node2 "node2.fgq.com"
Created /test/node2
[zk: localhost:2181(CONNECTED) 13] ls /test
[node2, node1]
[zk: localhost:2181(CONNECTED) 14] ls2 /test
[node2, node1]
cZxid = 0x18
ctime = Fri Mar 15 14:19:10 CST 2019
mZxid = 0x18
mtime = Fri Mar 15 14:19:10 CST 2019
pZxid = 0x1a
cversion = 2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 16
numChildren = 2
# 删除/test，此处有子目录，使用rmr删除
[zk: localhost:2181(CONNECTED) 18] delete /test
Node not empty: /test
[zk: localhost:2181(CONNECTED) 19] rmr /test
[zk: localhost:2181(CONNECTED) 20] ls /
[zookeeper]
# 创建临时目录(节点)，临时目录不能有子目，退出后，再看就没有了
持久目录(节点)，只有通过rmr/delete才会删除持久目录
[zk: localhost:2181(CONNECTED) 16] create -e /test "test app"
Created /test
[zk: localhost:2181(CONNECTED) 17] create -e /test/node1 "test app node1"
Ephemerals cannot have children: /test/node1
[zk: localhost:2181(CONNECTED) 18] quit
[root@node1 zookeeper]# ./bin/zkCli.sh
[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper]
集群配置
node2和node3上
[root@node2 ~]# mkdir /fgq/base-env/zookeeper/{data,log}
[root@node2 ~]# mkdir /fgq/base-env/zookeeper/{data,log}
node1
[root@node1 ~]# cd /fgq/base-env/zookeeper/conf/
[root@node1 conf]# vim zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/fgq/base-env/zookeeper/data/
dataLogDir=/fgq/base-env/zookeeper/log/
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=192.168.25.11:2888:3888
server.2=192.168.25.12:2888:3888
server.3=192.168.25.13:2888:3888
[root@node1 conf]# echo 1 > ../data/myid
[root@node1 conf]# scp zoo.cfg node2:/fgq/base-env/zookeeper/conf/
[root@node1 conf]# scp zoo.cfg node3:/fgq/base-env/zookeeper/conf/
node2和node3
[root@node2 ~]# echo 2 > /fgq/base-env/zookeeper/data/myid
[root@node3 ~]# echo 3 > /fgq/base-env/zookeeper/data/myid
# node1先停止上面运行的单实例
[root@node1 ~]# /fgq/base-env/zookeeper/bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /fgq/base-env/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
# 再启动
[root@node1 ~]# /fgq/base-env/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /fgq/base-env/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@node1 ~]# ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 50 :::2181 :::*
LISTEN 0 50 :::40973 :::*
LISTEN 0 50 ::ffff:192.168.25.11:3888 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
# node2上启动
[root@node2 ~]# /fgq/base-env/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /fgq/base-env/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@node2 ~]# ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 50 :::2181 :::*
LISTEN 0 50 :::39529 :::*
LISTEN 0 50 ::ffff:192.168.25.12:3888 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
# node3上启动
[root@node3 ~]# /fgq/base-env/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /fgq/base-env/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
You have new mail in /var/spool/mail/root
[root@node3 ~]# ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 50 :::2181 :::*
LISTEN 0 50 :::35242 :::*
LISTEN 0 50 ::ffff:192.168.25.13:3888 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
等过了一会之后，zookeeper自动就会分配主从(leader/follower)
看配置文件中设置的2888端口出现在哪个机器上，哪个机器上就是leader
[root@node1 ~]# ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 50 :::2181 :::*
LISTEN 0 50 ::ffff:192.168.25.11:2888 :::*
LISTEN 0 50 :::40973 :::*
LISTEN 0 50 ::ffff:192.168.25.11:3888 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
测试
[root@node1 ~]# telnet 192.168.25.11 2181
Trying 192.168.25.11...
Connected to 192.168.25.11.
Escape character is '^]'.
输入信息: ruok
imokConnection closed by foreign host.
[root@node1 ~]# telnet 192.168.25.11 2181
Trying 192.168.25.11...
Connected to 192.168.25.11.
Escape character is '^]'.
输入信息: stat
Zookeeper version: 3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
Clients:
/192.168.25.11:60252[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x100000000
Mode: leader 显示192.168.25.11--node1节点是leader
Node count: 4
Proposal sizes last/min/max: -1/-1/-1
Connection closed by foreign host.
[root@node1 ~]# telnet 192.168.25.12 2181
Trying 192.168.25.12...
Connected to 192.168.25.12.
Escape character is '^]'.
stat
Zookeeper version: 3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
Clients:
/192.168.25.11:43600[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x25
Mode: follower 显示192.168.25.12--node2节点是follower
Node count: 4
Connection closed by foreign host.
[root@node1 ~]# telnet 192.168.25.13 2181
Trying 192.168.25.13...
Connected to 192.168.25.13.
Escape character is '^]'.
stat
Zookeeper version: 3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
Clients:
/192.168.25.11:48678[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x100000000
Mode: follower 显示192.168.25.13--node3节点是follower
Node count: 4
Connection closed by foreign host.
网友评论