spark的集群配置

这里我们介绍 standalone 模式环境搭建
首先在所有的节点上安装spark

参考文章安装 spark
https://www.jianshu.com/p/b0d88e5dd503

到 spark的conf的目录下，查看配置文件

[river@s201 spark]$ cd /soft/spark/conf/
[river@s201 conf]$ ll
total 36
-rw-r--r--. 1 river river  996 Oct 29 14:36 docker.properties.template
-rw-r--r--. 1 river river 1105 Oct 29 14:36 fairscheduler.xml.template
-rw-r--r--. 1 river river 2025 Oct 29 14:36 log4j.properties.template
-rw-r--r--. 1 river river 7801 Oct 29 14:36 metrics.properties.template
-rw-r--r--. 1 river river  865 Oct 29 14:36 slaves.template
-rw-r--r--. 1 river river 1292 Oct 29 14:36 spark-defaults.conf.template
-rwxr-xr-x. 1 river river 4221 Oct 29 14:36 spark-env.sh.template
[river@s201 conf]$

配置master节点的slaves
修改该目录下 slaves.template 文件为 slaves

mv slaves.template slaves

编辑slaves 文件
添加从节点主机名

s202
s203
s204

我的hosts 配置如下

[river@s201 conf]$ cat /etc/hosts
127.0.0.1 localhost
192.168.172.201 s201
192.168.172.202 s202
192.168.172.203 s203
192.168.172.204 s204

同步slaves 文件到从节点

[river@s201 conf]$ scp slaves river@s202:/soft/spark/conf/  
slaves                                                                         100%  871   736.9KB/s   00:00    
[river@s201 conf]$ scp slaves river@s203:/soft/spark/conf/
slaves                                                                         100%  871   535.0KB/s   00:00    
[river@s201 conf]$ scp slaves river@s204:/soft/spark/conf/
slaves                                                                         100%  871   530.9KB/s   00:00    
[river@s201 conf]$

接下来启动spark集群

[river@s201 conf]$ /soft/spark/sbin/start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /soft/spark/logs/spark-river-org.apache.spark.deploy.master.Master-1-s201.out
s204: starting org.apache.spark.deploy.worker.Worker, logging to /soft/spark/logs/spark-river-org.apache.spark.deploy.worker.Worker-1-s204.out
s202: starting org.apache.spark.deploy.worker.Worker, logging to /soft/spark/logs/spark-river-org.apache.spark.deploy.worker.Worker-1-s202.out
s203: starting org.apache.spark.deploy.worker.Worker, logging to /soft/spark/logs/spark-river-org.apache.spark.deploy.worker.Worker-1-s203.out
s201: starting org.apache.spark.deploy.worker.Worker, logging to /soft/spark/logs/spark-river-org.apache.spark.deploy.worker.Worker-1-s201.out

通过jps 查看进程

[river@s201 conf]$ jps
66208 Jps
2148 SecondaryNameNode
2310 ResourceManager
1943 NameNode
66072 Master
66156 Worker

[river@s202 soft]$ jps
1781 DataNode
49061 Jps
48987 Worker
1902 NodeManager

可以看到主节点上已经有 master 和 worker了，从节点也有了 worker
说明已经启动成功了

我们再登录到web来看看集群状况