spark的maven项目构建（基于idea 和maven）

环境预安装

• idea安装
• java安装
• scala安装
• idea安装scala插件

1.新建一个maven项目

开始创建项目体系结构
File --> Project

image
选择org.scala-tools.archetypes:scala-archetype-simple image
maven的设置
image

项目名称

image

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>test.gzstrong</groupId>
    <artifactId>Maven_BigData</artifactId>
    <version>1.0-SNAPSHOT</version>
    <inceptionYear>2008</inceptionYear>

    <properties>
      <scala.version>2.11.0</scala.version>
      <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <repositories>
      <repository>
        <id>scala-tools.org</id>
        <name>Scala-Tools Maven2 Repository</name>
        <url>http://scala-tools.org/repo-releases</url>
      </repository>
    </repositories>

    <pluginRepositories>
      <pluginRepository>
        <id>scala-tools.org</id>
        <name>Scala-Tools Maven2 Repository</name>
        <url>http://scala-tools.org/repo-releases</url>
      </pluginRepository>
    </pluginRepositories>

    <dependencies>
      <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>${scala.version}</version>
      </dependency>
      <dependency>
        <groupId>org.specs</groupId>
        <artifactId>specs</artifactId>
        <version>1.2.5</version>
        <scope>test</scope>
      </dependency>

      <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.3.0</version>
      </dependency>
    </dependencies>


    <build>
      <sourceDirectory>src/main/scala</sourceDirectory>
      <testSourceDirectory>src/test/scala</testSourceDirectory>
      <plugins>
        <plugin>
          <groupId>org.scala-tools</groupId>
          <artifactId>maven-scala-plugin</artifactId>
          <executions>
            <execution>
              <goals>
                <goal>compile</goal>
                <goal>testCompile</goal>
              </goals>
            </execution>
          </executions>
          <configuration>
            <scalaVersion>${scala.version}</scalaVersion>
            <args>
              <arg>-target:jvm-1.5</arg>
            </args>
          </configuration>
        </plugin>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-eclipse-plugin</artifactId>
          <configuration>
            <downloadSources>true</downloadSources>
            <buildcommands>
              <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand>
            </buildcommands>
            <additionalProjectnatures>
              <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature>
            </additionalProjectnatures>
            <classpathContainers>
              <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer>
              <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer>
            </classpathContainers>
          </configuration>
        </plugin>
      </plugins>
    </build>
    <reporting>
      <plugins>
        <plugin>
          <groupId>org.scala-tools</groupId>
          <artifactId>maven-scala-plugin</artifactId>
          <configuration>
            <scalaVersion>${scala.version}</scalaVersion>
          </configuration>
        </plugin>
      </plugins>
    </reporting>
  </project>

2.目录结构如下:

image

3.在src的main目录下新建java和scala文件夹

4.同时修改java和scala文件夹为源码文件夹

image

5.在src的test目录下新建java和scala文件夹

image

6.同时修改java和scala文件夹为测试文件夹

image

然后设置scal为项目的sdk

7.写个spark

SparkApp.scala

package test.gzstrong

import org.apache.spark.{SparkConf, SparkContext}

object SparkApp{
    def main(args: Array[String]): Unit = {
        val conf = new SparkConf().setAppName("test").setMaster("local[2]")
        val sc = new SparkContext(conf)
        val data = Array(1, 2,2,1,2, 3, 4, 5)
        val distData = sc.parallelize(data)
        val pairs = distData.map((_, 1))
        val counts = pairs.reduceByKey((a, b) => a + b)
        counts.collect().foreach(println)
        sc.stop()
    }
}

8.运行

运行