WordCount

spark2maven编写scala的WordCount程序

环境描述

scala.version 2.11.2
jdk.version 1.8
spark.version 2.2.0
maven.version 3.5.4

配置本地maven

修改conf下的settings.xml文件
<localRepository>F:\m2</localRepository>

创建maven项目

image.png image.png image.png

main下新建scala的Directory文件夹

image.png

设置根目录权限Mark Directory as 到Sources Root（可以新建java或者scala的class类）

image.png

设置sdk为2.11

image.png

新建scala类WordCount

image.png
image.png

下载依赖，编写pom.xml文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <properties>
        <project.version>1.0-SNAPSHOT</project.version>
        <scala.version>2.11.2</scala.version>
        <jdk.version>1.8</jdk.version>
        <spark.version>2.2.0</spark.version>
    </properties>

    <groupId>spark.test</groupId>
    <artifactId>spark.test.test_one</artifactId>
    <packaging>jar</packaging>
    <version>${project.version}</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
    </dependencies>

</project>

编写WordCount程序

import org.apache.spark.{SparkConf, SparkContext}

object WordCount {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("WordCount").setMaster("local")
    val sc = new SparkContext(conf)

    val lines = sc.textFile("F:/scala_test/spark.txt",1)
    val words = lines.flatMap(_.split(" ")).filter(word => word != " ")
    //val words = lines.flatMap { line => line.split(" ") }
    val pairs = words.map(word => (word, 1))
    //val pairs = words.map { word => (word, 1) }
    val wordscount = pairs.reduceByKey(_ + _)

    wordscount.collect.foreach(println)
    sc.stop()
  }
}

调试日志级别

因为打印出日志太多，不好观看，可以配置一下日志级别。
在scala目录WordCount同级目录下新建file文件名称为log4j.properties
log4j.rootCategory=INFO, console更改为
log4j.rootCategory=ERROR, console
（或者在spark-2.2.0-bin-hadoop2.6\spark-2.2.0-bin-hadoop2.6\conf下把log4j.properties.template拷贝出来修改一下）

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Set everything to be logged to the console
log4j.rootCategory=ERROR, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

报错

run1

找不到working directory目录

image.png image.png

working directory目录更改为$MODULE_DIR$

image.png

run2

数据出来了，但是还有个错误，没有找到hadoop的环境变量。

image.png

下载zip并解压到任意目录
https://github.com/srccodes/hadoop-common-2.2.0-bin

image.png
配置环境变量
image.png

结果

Process finished with exit code 0
0说明程序正常执行完毕，1说明程序出错。

image.png