Spark搭建 之 Standalone集群

本文主线 系统 => 集群 => 测试

本文基于Spark搭建 之 单机模式以及Spark第一个应用

系统

保证集群ssh互通

hostname

sudo hostname master
# sudo hostname slave01
# sudo hostname slave02

sudo vim /etc/hostname
# master or slave01 or slave02

hosts

sudo vim /etc/hosts
# 192.168.1.29 master
# 192.168.1.30 slave01
# 192.168.1.31 slave02

集群

# master
cd /opt/services

wget https://mirror.tuna.tsinghua.edu.cn/apache/spark/spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz

tar xf spark-2.4.7-bin-hadoop2.7.tgz

mv spark-2.4.7-bin-hadoop2.7 spark

cd spark

cp conf/slaves.template conf/slaves

vim conf/slaves
# slave01
# slave02

cp conf/spark-env.sh.template conf/spark-env.sh

vim conf/spark-env.sh
# export JAVA_HOME=/opt/services/jdk
# SPARK_MASTER_PORT=master
# SPARK_MASTER_PORT=7077

scp -r /opt/services/spark op@slave01:/opt/services

scp -r /opt/services/spark op@slave02:/opt/services

方法1

# master
/opt/services/spark/sbin/start-all.sh

方法2

# master
/opt/services/spark/sbin/start-master.sh

# slave01 & slave02
/opt/services/spark/sbin/start-slave.sh spark://master:7077

• http://192.168.1.29:8080

测试

# slave01
echo apple >> /opt/services/spark/README.md
# slave02
echo blue >> /opt/services/spark/README.md

• 基于Spark第一个应用

/opt/services/spark/bin/spark-submit --class "SimpleApp" --master spark://master:7077 ~/HelloSpark.jar
# Lines with a: 61, Lines with b: 31

参考

• Spark Standalone Mode