win10下idea使用sbt方式编译scala源码
虽然集群使用Linux，Ubuntu的桌面版也有非常好的体验感受，但是windows下写码编译打包成jar文件然后再提交到集群运行也是不错的，故本文记录win10下如何搭建环境
准备工作
idea官网下载 https://www.jetbrains.com/idea/
sbt官网下载 https://www.scala-sbt.org/download.html
一些配置
打开idea，右上角File->Settings
找到Plugins(插件)，选择安装Jetbrains的插件
安装idea插件按钮.png
在搜索框搜索Scala什么版本无所谓，可能就是为了代码高亮并且有智能提示和联想。因为在之后sbt编译的配置中不一定使用这个版本。
下载Scala插件.png
插件下载完成之后会提示restart idea，那就restart咯~
新建一个Project，选择sbt
新建项目.png
安装环境需要java
jdk-sbt-scala.png
打开Settings,找到Build Tools，进行对sbt的一些自定义
可以使用在官网下载的sbt-1.1.6.zip，解压，解压后的文件路径请添加到环境变量
然后如下图，Launcher选择Custom，选择刚刚解压的文件中sbt/bin/sbt-launch.jar
配置sbt.png
如果能够用Google的小伙伴这一步无所谓，不能用就换源，可以换成阿里的~
找到解压sbt的目录，进入conf文件夹，新建文本文档改名为repo.properties
[repositories]
local
aliyun: http://maven.aliyun.com/nexus/content/groups/public/
typesafe: http://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
sonatype-oss-releases
maven-central
sonatype-oss-snapshots
完成之后，切换至Terminal
sbt有许多命令
sbt clean
sbt compile
sbt package
sbt assembly
···
我习惯使用package命令，将scala编译后生成的内容打成jar包，提交到集群运行
到此为止，可以键入sbt sbtVersion查看sbt的版本，这个过程将陷入漫长的等待。
第一次会比较慢，可以去喝杯咖啡或者开一波黑，之后就会比较快了。
键入 sbt sbtVersion 的运行结果如果是这样就表示成了！
D:\IDEAProjects\SparkSample>sbt sbtVersion
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[info] Loading settings from idea.sbt ...
[info] Loading global plugins from C:\Users\zhongfuze\.sbt\1.0\plugins
[info] Loading settings from assembly.sbt ...
[info] Loading project definition from D:\IDEAProjects\SparkSample\project
[info] Loading settings from build.sbt ...
[info] Set current project to SparkSample (in build file:/D:/IDEAProjects/SparkSample/)
[info] 1.1.6
在src/main/scala里新建HelloScala.scala
object HelloScala {
def main(args: Array[String]): Unit = {
println("Hello Scala!")
}
}
键入 sbt package 的运行结果如果是这样就表示成了！
生成的jar包 在根目录/target/scala-2.11/xxxxx.jar
D:\IDEAProjects\SparkSample>sbt package
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[info] Loading settings from idea.sbt ...
[info] Loading global plugins from C:\Users\zhongfuze\.sbt\1.0\plugins
[info] Loading settings from assembly.sbt ...
[info] Loading project definition from D:\IDEAProjects\SparkSample\project
[info] Loading settings from build.sbt ...
[info] Set current project to SparkSample (in build file:/D:/IDEAProjects/SparkSample/)
[info] Compiling 1 Scala source to D:\IDEAProjects\SparkSample\target\scala-2.11\classes ...
[info] Done compiling.
[info] Packaging D:\IDEAProjects\SparkSample\target\scala-2.11\sparksample_2.11-1.0.jar ...
[info] Done packaging.
[success] Total time: 4 s, completed 2018-7-24 16:12:19
分割线
到这里为止只是能够能打包而已，但是coding的时候是需要各种各样的配置的呢
而且为了便于维护各种各样的依赖jar，properties，config，并且有时需要决定某些jar文件需要被打包，有的jar文件集群里已经存在了，不需要打包，要满足这种需求，就使用sbt assembly命令
在项目根目录/project下，与build.properties的同级目录
新建assembly.sbt，内容如下
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.7")
Note:sbt-assembly的版本取决于sbt的版本，不同的版本区间版本号不一样的！！！
assembly-sbt.png
接着找到根目录下build.sbt
这里可以自定义许多内容，添加依赖等等
更具体的用法请参考github-sbt-assembly
https://github.com/sbt/sbt-assembly
name := "SparkSample"
version := "1.0"
organization := "com.zhong.PRM"
scalaVersion := "2.11.8"
assemblyJarName in assembly := "PRM.jar"
test in assembly := {}
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
assemblyMergeStrategy in assembly := {
case PathList("javax", "servlet", xs@_*) => MergeStrategy.first
case PathList(ps@_*) if ps.last endsWith ".class" => MergeStrategy.first
case PathList(ps@_*) if ps.last endsWith ".xml" => MergeStrategy.first
case PathList(ps@_*) if ps.last endsWith ".properties" => MergeStrategy.first
case "application.conf" => MergeStrategy.concat
case "unwanted.txt" => MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" % "provided"
排除jar包
provided 表示打包的时候可以不包含这个jar包
sbt-assembly是根据项目配置的libraryDependencies依赖进行打包的，不需要打包的依赖可以设置”provided”进行排除
[build.sbt]
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" % "provided"
排除scala库的jar包
在项目根目录下创建assembly.sbt文件并添加以下配置（注：sbt-assembly相关的配置，可以配置在项目根目录/build.sbt中，也可以在项目根目录下的assembly.sbt文件中）：
[assembly.sbt]
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
明确排除某一指定jar包
[assembly.sbt]
assemblyExcludedJars in assembly := {
val cp = (fullClasspath in assembly).value
cp filter {_.data.getName == “compile-0.1.0.jar”}
}
多个文件共享相同的相对路径
如果多个文件共享相同的相对路径（例如，多个依赖项JAR中名为application.conf的资源），则默认策略是验证所有候选项具有相同的内容，否则出错。可以使用以下内置策略之一或编写自定义策略在每个路径的基础上配置此行为：
MergeStrategy.deduplicate是上面描述的默认值
MergeStrategy.first以类路径顺序选择第一个匹配文件
MergeStrategy.last选择最后一个
MergeStrategy.singleOrError在冲突时出现错误消息
MergeStrategy.concat简单地连接所有匹配的文件并包含结果
MergeStrategy.filterDistinctLines也会连接，但在此过程中会遗漏重复项
MergeStrategy.rename重命名源自jar文件的文件
MergeStrategy.discard只是丢弃匹配的文件
路径名称到合并策略的映射是通过设置assemblyMergeStrategy完成的，可以按如下方式扩充
assemblyMergeStrategy in assembly := {
case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first
case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
case "application.conf" => MergeStrategy.concat
case "unwanted.txt" => MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
放一些配置文件做备忘
[plugins.sbt]
logLevel := Level.Warn
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
[build.sbt]
name := "lanke"
version := "1.0"
scalaVersion := "2.11.8"
assemblyJarName in assembly := "lanke.jar"
test in assembly := {}
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
assemblyMergeStrategy in assembly := {
case PathList("javax", "servlet", xs@_*) => MergeStrategy.first
case PathList(ps@_*) if ps.last endsWith ".class" => MergeStrategy.first
case PathList(ps@_*) if ps.last endsWith ".xml" => MergeStrategy.first
case PathList(ps@_*) if ps.last endsWith ".properties" => MergeStrategy.first
case "application.conf" => MergeStrategy.concat
case "unwanted.txt" => MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
resolvers ++= Seq(
"kompics" at "http://kompics.sics.se/maven/repository/"
)
javacOptions ++= Seq("-encoding", "UTF-8", "-source", "1.7", "-target", "1.7")
resolvers ++= Seq(
"libs-releases" at "http://artifactory.jd.com/libs-releases",
"libs-snapshots" at "http://artifactory.jd.com/libs-snapshots",
"plugins-releases" at "http://artifactory.jd.com/plugins-releases",
"plugins-snapshots" at "http://artifactory.jd.com//plugins-snapshots"
)
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.11" % "2.3.0" % "provided",
"org.apache.spark" % "spark-sql_2.11" % "2.3.0" % "provided",
"org.apache.spark" % "spark-streaming_2.11" % "2.3.0" % "provided",
"org.apache.spark" % "spark-hive_2.11" % "2.3.0" % "provided",
"org.apache.spark" % "spark-repl_2.11" % "2.3.0" % "provided",
"org.apache.spark" % "spark-tags_2.11" % "2.3.0" % "provided"
)
libraryDependencies += "com.yammer.metrics" % "metrics-core" % "2.2.0"
libraryDependencies += "com.typesafe" % "config" % "1.2.1"
libraryDependencies += "net.liftweb" % "lift-json_2.11" % "3.0"
libraryDependencies += "com.huaban" % "jieba-analysis" % "1.0.2"
resolvers += "Sonatype OSS Releases" at "http://oss.sonatype.org/content/repositories/releases/"
libraryDependencies += "com.thesamet" %% "kdtree" % "1.0.4"
libraryDependencies += "com.soundcloud" % "cosine-lsh-join-spark_2.10" % "1.0.1"
libraryDependencies += "org.tensorflow" %% "spark-tensorflow-connector" % "1.6.0"
libraryDependencies += "org.scalaj" %% "scalaj-http" % "2.4.0"
[tools/sbt/conf repo.properties]
[repositories]
local
my-ivy-proxy-releases: http://artifactory.jd.com/ivy-release/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
my-maven-proxy-releases: http://artifactory.jd.com/libs-releases/
[respositories]
[repositories]
local
aliyun: http://maven.aliyun.com/nexus/content/groups/public/
typesafe: http://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
sonatype-oss-releases
maven-central
sonatype-oss-snapshots
网友评论