配置 Spark 集群环境配置

spark-2.4.0-bin-hadoop2.7.tgz上传到192.168.11.10机器上

解压缩spark压缩到到 /usr/local

tar -zxvf spark-2.4.0-bin-hadoop2.7.tgz -C /usr/local

cd /usr/local

mv spark-2.4.0-bin-hadoop2.7/ spark

修改spark-env.sh配置文件

把SPARK_HOME/conf/下的spark-env.sh.template文件复制为spark-env.sh

cd /usr/local/spark/conf

mv spark-env.sh.template spark-env.sh

vi spark-env.sh

添加以下内容

export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)
export JAVA_HOME=/usr/local/java8
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export SPARK_MASTER_IP=192.168.11.10
export SPARK_MASTER_HOST=master
export SPARK_MASTER_PORT=7077
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=5 -Dspark.history.fs.logDirectory=hdfs://master:9000/spark-app_history"

修改slaves配置文件

cd /usr/local/spark/conf

mv slaves.template slaves

vi slaves

添加以下从节点

master

修改 spark-defaults.conf

cd /usr/local/spark/conf

mv spark-defaults.conf.template spark-defaults.conf

vi spark-defaults.conf

spark.eventLog.enabled     true
spark.eventLog.dir     hdfs://master:9000/spark-app-history
spark.eventLog.compress true

在HDFS上创建spark-app-history目录

hdfs dfs -mkdir /spark-app-history

添加 Spark路径

vi /etc/profile

export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin:

source /etc/profile

启动集群

cd /usr/local/spark/sbin

cp start-all.sh start-spark-all.sh

cp stop-all.sh stop-spark-all.sh

start-spark-all.sh

访问 http://192.168.11.10:8080/ 页面，