自定义spark metric source

作者: 望山不是山 | 来源:发表于2018-12-20 23:05 被阅读0次

自定义spark metric source
k8s官方文档实践系列－将日志记录和指标添加到示例
Skywalking-04：扩展Metric监控信息
25-SparkStreaming01
spark 学习笔记
7、Flume高阶自定义组件_Flume自定义Source
尚硅谷大数据技术之Flume
spark xgboost损失函数和评价函数的修改
learn spark
spark2.1 sql 自定义udf以及spark sql a

Executor启动命令

/usr/soft/jdk1.8/bin/java -cp /usr/soft/spark-2.0.1-bin-hadoop2.7/conf/:/usr/soft/spark-2.0.1-bin-hadoop2.7/jars/*:/usr/soft/hadoop/etc/hadoop/ -Xmx1024M -Dspark.driver.port=57067 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@hadoop01:57067 --executor-id 1 --hostname 192.168.116.180 --cores 4 --app-id app-20181211230052-0005 --worker-url spark://Worker@192.168.116.180:53016

日志如下

2018-12-11 22:15:37 INFO  CoarseGrainedExecutorBackend:2608 - Started daemon with process name: 16451@hadoop01
2018-12-11 22:15:37 INFO  SignalUtils:54 - Registered signal handler for TERM
2018-12-11 22:15:37 INFO  SignalUtils:54 - Registered signal handler for HUP
2018-12-11 22:15:37 INFO  SignalUtils:54 - Registered signal handler for INT
2018-12-11 22:15:40 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-12-11 22:15:40 INFO  SecurityManager:54 - Changing view acls to: root
2018-12-11 22:15:40 INFO  SecurityManager:54 - Changing modify acls to: root
2018-12-11 22:15:40 INFO  SecurityManager:54 - Changing view acls groups to: 
2018-12-11 22:15:40 INFO  SecurityManager:54 - Changing modify acls groups to: 
2018-12-11 22:15:40 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
2018-12-11 22:15:43 INFO  TransportClientFactory:267 - Successfully created connection to hadoop01/192.168.116.180:40360 after 954 ms (0 ms spent in bootstraps)
2018-12-11 22:15:43 INFO  SecurityManager:54 - Changing view acls to: root
2018-12-11 22:15:43 INFO  SecurityManager:54 - Changing modify acls to: root
2018-12-11 22:15:43 INFO  SecurityManager:54 - Changing view acls groups to: 
2018-12-11 22:15:43 INFO  SecurityManager:54 - Changing modify acls groups to: 
2018-12-11 22:15:43 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
2018-12-11 22:15:44 INFO  TransportClientFactory:267 - Successfully created connection to hadoop01/192.168.116.180:40360 after 7 ms (0 ms spent in bootstraps)
2018-12-11 22:15:44 INFO  DiskBlockManager:54 - Created local directory at /tmp/spark-fc435845-8ad8-49d3-85df-1c8562de05de/executor-e318fab1-90a7-462f-88f1-4eb2d5712e21/blockmgr-cfb0dac2-54eb-4da2-b985-34f63d65277c
2018-12-11 22:15:44 INFO  MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2018-12-11 22:15:45 INFO  CoarseGrainedExecutorBackend:54 - Connecting to driver: spark://CoarseGrainedScheduler@hadoop01:40360
2018-12-11 22:15:45 INFO  WorkerWatcher:54 - Connecting to worker spark://Worker@192.168.116.180:53016
2018-12-11 22:15:45 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.116.180:53016 after 8 ms (0 ms spent in bootstraps)
2018-12-11 22:15:45 INFO  WorkerWatcher:54 - Successfully connected to spark://Worker@192.168.116.180:53016
2018-12-11 22:15:45 INFO  CoarseGrainedExecutorBackend:54 - Successfully registered with driver
2018-12-11 22:15:45 INFO  Executor:54 - Starting executor ID 0 on host 192.168.116.180
2018-12-11 22:15:46 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 48702.
2018-12-11 22:15:46 INFO  NettyBlockTransferService:54 - Server created on 192.168.116.180:48702
2018-12-11 22:15:46 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2018-12-11 22:15:46 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(0, 192.168.116.180, 48702, None)
2018-12-11 22:15:46 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(0, 192.168.116.180, 48702, None)
2018-12-11 22:15:46 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(0, 192.168.116.180, 48702, None)
2018-12-11 22:15:46 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 0
2018-12-11 22:15:46 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 1
2018-12-11 22:15:46 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 2
2018-12-11 22:15:46 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 3
2018-12-11 22:15:46 INFO  Executor:54 - Running task 2.0 in stage 0.0 (TID 2)
2018-12-11 22:15:46 INFO  Executor:54 - Running task 1.0 in stage 0.0 (TID 1)
2018-12-11 22:15:46 INFO  Executor:54 - Running task 0.0 in stage 0.0 (TID 0)
2018-12-11 22:15:46 INFO  Executor:54 - Running task 3.0 in stage 0.0 (TID 3)
2018-12-11 22:15:46 INFO  Executor:54 - Fetching spark://hadoop01:40360/jars/spark-refact_2.12.jar with timestamp 1544537727225
2018-12-11 22:15:47 INFO  TransportClientFactory:267 - Successfully created connection to hadoop01/192.168.116.180:40360 after 49 ms (0 ms spent in bootstraps)
2018-12-11 22:15:47 INFO  Utils:54 - Fetching spark://hadoop01:40360/jars/spark-refact_2.12.jar to /tmp/spark-fc435845-8ad8-49d3-85df-1c8562de05de/executor-e318fab1-90a7-462f-88f1-4eb2d5712e21/spark-3ac85576-200e-4062-8118-aa70e135a9c5/fetchFileTemp8785665110169178609.tmp
2018-12-11 22:15:48 INFO  Utils:54 - Copying /tmp/spark-fc435845-8ad8-49d3-85df-1c8562de05de/executor-e318fab1-90a7-462f-88f1-4eb2d5712e21/spark-3ac85576-200e-4062-8118-aa70e135a9c5/-13608901521544537727225_cache to /usr/soft/spark-2.0.1-bin-hadoop2.7/work/app-20181211221528-0003/0/./spark-refact_2.12.jar
2018-12-11 22:15:48 INFO  Executor:54 - Adding file:/usr/soft/spark-2.0.1-bin-hadoop2.7/work/app-20181211221528-0003/0/./spark-refact_2.12.jar to class loader
2018-12-11 22:15:49 INFO  TorrentBroadcast:54 - Started reading broadcast variable 0
2018-12-11 22:15:50 INFO  TransportClientFactory:267 - Successfully created connection to hadoop01/192.168.116.180:48263 after 21 ms (0 ms spent in bootstraps)
2018-12-11 22:15:51 INFO  MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 2011.0 B, free 366.3 MB)
2018-12-11 22:15:52 INFO  TorrentBroadcast:54 - Reading broadcast variable 0 took 2743 ms
2018-12-11 22:15:52 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 3.1 KB, free 366.3 MB)
2018-12-11 22:15:56 INFO  Executor:54 - Finished task 3.0 in stage 0.0 (TID 3). 1127 bytes result sent to driver
2018-12-11 22:15:56 INFO  Executor:54 - Finished task 1.0 in stage 0.0 (TID 1). 1127 bytes result sent to driver
2018-12-11 22:15:56 INFO  Executor:54 - Finished task 0.0 in stage 0.0 (TID 0). 1084 bytes result sent to driver
2018-12-11 22:15:56 INFO  Executor:54 - Finished task 2.0 in stage 0.0 (TID 2). 1084 bytes result sent to driver
2018-12-11 22:15:56 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 4
2018-12-11 22:15:56 INFO  Executor:54 - Running task 4.0 in stage 0.0 (TID 4)