dr-elephant 问题

1 ./compile.sh ./compile.conf 构建 2.0.15

node-sass install caught in infinite loop

gyp verb tmpdir == cwd automatically will remove dev files after to save disk space
gyp verb command install [ '7.8.0' ]
gyp verb download using dist-url https://npm.taobao.org/dist
gyp verb install input version string "7.8.0"
gyp verb install installing version: 7.8.0
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp verb install version not already installed, continuing with install 7.8.0
gyp verb ensuring nodedir is created /data/bigData/dr-elephant-2.0.15/web/node_modules/node-sass/.node-gyp/7.8.0
gyp WARN EACCES user "root" does not have permission to access the dev dir "/data/bigData/dr-elephant-2.0.15/web/node_modules/node-sass/.node-gyp/7.8.0"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/data/bigData/dr-elephant-2.0.15/web/node_modules/node-sass/.node-gyp"

resolve：

# npm -g --unsafe-perm install node-sass

参考

2 ./bin/start.sh app-conf/ 启动报错

$ create index yarn_app_result_i5 on yarn_app_result (job_def_id); （job_def_id varchar(800)）

 [42000][1071] Specified key was too long; max key length is 3072 bytes

resolve:
索引长度是有限制的, innodb引擎utf8mb4编码的varchar占4个字节，3072 / 4 = 768.

使用create index yarn_app_result_i5 on yarn_app_result (job_def_id(768));创建索引
或 将字段长度改为768

3 spark 启动historyserver

./sbin/start-history-server.sh

报异常

Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:278)
    at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.io.FileNotFoundException: Log directory specified does not exist: file:/tmp/spark-events Did you configure the correct one through spark.history.fs.logDirectory?
    at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:214)
    at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:160)
    at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:156)
    at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:78)
    ... 6 more
Caused by: java.io.FileNotFoundException: File file:/tmp/spark-events does not exist
    at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
    at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:204)
    ... 9 more

将spark.fs.history.logDirectory配置到conf/spark-defaults.conf目录里，发现并不起作用，

查看start-history-server.sh脚本内容，发现这个参数可以作为启动脚本的第一个参数传递：

if [ $# != 0 ]; then  
  echo "Using command line arguments for setting the log directory is deprecated. Please "  
  echo "set the spark.history.fs.logDirectory configuration option instead."  
  export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$1"  
fi  

故使用：

./start-history-server.sh  hdfs://localhost:9000/history

4 ./compile.sh ./compile.conf ( dr-elephant-2.0.15)

• sbt 0.13.2 spark 1.6.3 scala 2.10.4 OK

• sbt 0.13.2 spark 2.2.1 scala 2.10.4

java.lang.IllegalStateException: impossible to get artifacts when data has not been loaded. IvyNode = org.slf4j#slf4j-api;1.7.6
    at org.apache.ivy.core.resolve.IvyNode.getArtifacts(IvyNode.java:809)
    at org.apache.ivy.core.resolve.IvyNode.getSelectedArtifacts(IvyNode.java:786)
    at org.apache.ivy.core.report.ResolveReport.setDependencies(ResolveReport.java:235)
    at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:235)
    at org.apache.ivy.Ivy.resolve(Ivy.java:517)
    at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:211)
    at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:126)
    at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:125)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
    at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)

• sbt 1.1.1 spark 1.6.3 scala 2.10.4

java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
    at sbt.Help$.briefDetail(Command.scala:232)
    at sbt.Help$.<init>(Command.scala:214)
    at sbt.Help$.<clinit>(Command.scala)
    at sbt.Command$.command$default$2(Command.scala:87)
    at sbt.BuiltinCommands$.defaults(Main.scala:363)
    at sbt.xMain.run(Main.scala:74)
    at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:57)
    at xsbt.boot.Launch$.withContextLoader(Launch.scala:77)
    at xsbt.boot.Launch$.run(Launch.scala:57)
    at xsbt.boot.Launch$$anonfun$explicit$1.apply(Launch.scala:45)
    at xsbt.boot.Launch$.launch(Launch.scala:65)
    at xsbt.boot.Launch$.apply(Launch.scala:16)
    at xsbt.boot.Boot$.runImpl(Boot.scala:32)
    at xsbt.boot.Boot$.main(Boot.scala:21)
    at xsbt.boot.Boot.main(Boot.scala)
Error during sbt execution: java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;

• sbt: 0.13.7 spark: 2.2.1 scala: 2.11.8 / 2.10.4

java.lang.UnsupportedOperationException: Position.start on class scala.reflect.internal.util.OffsetPosition
    at scala.reflect.internal.util.Position.start(Position.scala:114)
    at sbt.internals.parser.SbtParser.sbt$internals$parser$SbtParser$$convertStatement$1(SbtParser.scala:142)
    at sbt.internals.parser.SbtParser$$anonfun$4.apply(SbtParser.scala:147)
    at sbt.internals.parser.SbtParser$$anonfun$4.apply(SbtParser.scala:147)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
    at sbt.internals.parser.SbtParser.splitExpressions(SbtParser.scala:147)
    at sbt.internals.parser.SbtParser.<init>(SbtParser.scala:60)
    at sbt.EvaluateConfigurations$.splitExpressions(EvaluateConfigurations.scala:227)
    at sbt.EvaluateConfigurations$.parseConfiguration(EvaluateConfigurations.scala:69)
    at sbt.EvaluateConfigurations$.evaluateSbtFile(EvaluateConfigurations.scala:107)
    at sbt.Load$.sbt$Load$$loadSettingsFile$1(Load.scala:710)
    at sbt.Load$$anonfun$sbt$Load$$memoLoadSettingsFile$1$1.apply(Load.scala:715)
    at sbt.Load$$anonfun$sbt$Load$$memoLoadSettingsFile$1$1.apply(Load.scala:714)
    at scala.Option.getOrElse(Option.scala:120)
    at sbt.Load$.sbt$Load$$memoLoadSettingsFile$1(Load.scala:714)
    at sbt.Load$$anonfun$loadFiles$1$2.apply(Load.scala:721)
    at sbt.Load$$anonfun$loadFiles$1$2.apply(Load.scala:721)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at sbt.Load$.loadFiles$1(Load.scala:721)
    at sbt.Load$.discoverProjects(Load.scala:732)
    at sbt.Load$.discover$1(Load.scala:545)
    at sbt.Load$.loadTransitive(Load.scala:574)
    at sbt.Load$.loadProjects$1(Load.scala:442)
    at sbt.Load$.loadUnit(Load.scala:446)
    at sbt.Load$$anonfun$18$$anonfun$apply$11.apply(Load.scala:281)
    at sbt.Load$$anonfun$18$$anonfun$apply$11.apply(Load.scala:281)
    at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(BuildLoader.scala:91)
    at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(BuildLoader.scala:90)
    at sbt.BuildLoader.apply(BuildLoader.scala:140)
    at sbt.Load$.loadAll(Load.scala:334)
    at sbt.Load$.loadURI(Load.scala:289)
    at sbt.Load$.load(Load.scala:285)
    at sbt.Load$.load(Load.scala:276)
    at sbt.Load$.apply(Load.scala:130)
    at sbt.Load$.buildPluginDefinition(Load.scala:819)
    at sbt.Load$.buildPlugins(Load.scala:785)
    at sbt.Load$.plugins(Load.scala:773)
    at sbt.Load$.loadUnit(Load.scala:431)
    at sbt.Load$$anonfun$18$$anonfun$apply$11.apply(Load.scala:281)
    at sbt.Load$$anonfun$18$$anonfun$apply$11.apply(Load.scala:281)
    at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(BuildLoader.scala:91)
    at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(BuildLoader.scala:90)
    at sbt.BuildLoader.apply(BuildLoader.scala:140)
    at sbt.Load$.loadAll(Load.scala:334)
    at sbt.Load$.loadURI(Load.scala:289)
    at sbt.Load$.load(Load.scala:285)
    at sbt.Load$.load(Load.scala:276)
    at sbt.Load$.apply(Load.scala:130)
    at sbt.Load$.defaultLoad(Load.scala:36)
    at sbt.BuiltinCommands$.doLoadProject(Main.scala:481)
    at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:475)
    at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:475)
    at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:58)
    at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:58)
    at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:60)
    at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:60)
    at sbt.Command$.process(Command.scala:92)
    at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:98)
    at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:98)
    at sbt.State$$anon$1.process(State.scala:184)
    at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:98)
    at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:98)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.MainLoop$.next(MainLoop.scala:98)
    at sbt.MainLoop$.run(MainLoop.scala:91)
    at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:70)
    at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:65)
    at sbt.Using.apply(Using.scala:24)
    at sbt.MainLoop$.runWithNewLog(MainLoop.scala:65)
    at sbt.MainLoop$.runAndClearLast(MainLoop.scala:48)
    at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:32)
    at sbt.MainLoop$.runLogged(MainLoop.scala:24)
    at sbt.StandardMain$.runManaged(Main.scala:53)
    at sbt.xMain.run(Main.scala:28)
    at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:57)
    at xsbt.boot.Launch$.withContextLoader(Launch.scala:77)
    at xsbt.boot.Launch$.run(Launch.scala:57)
    at xsbt.boot.Launch$$anonfun$explicit$1.apply(Launch.scala:45)
    at xsbt.boot.Launch$.launch(Launch.scala:65)
    at xsbt.boot.Launch$.apply(Launch.scala:16)
    at xsbt.boot.Boot$.runImpl(Boot.scala:32)
    at xsbt.boot.Boot$.main(Boot.scala:21)
    at xsbt.boot.Boot.main(Boot.scala)
[error] java.lang.UnsupportedOperationException: Position.start on class scala.reflect.internal.util.OffsetPosition