1. 借鉴

flume启动报错 java.lang.NoClassDefFoundError: org/htrace/Trace
安装Flume遇到的问题及解决
Flume配置lzo

2. 开始

2020-05-12 05:03:23,630 (conf-file-poller-0) [ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:146)] Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
        at org.apache.flume.sink.hdfs.HDFSEventSink.getCodec(HDFSEventSink.java:312)
        at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:237)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:411)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:141)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

很明显缺少jar包，同时我使用lzo压缩，操作的jar包也要多一个。
将以下jar包拷贝到flume的lib下即可

share/hadoop/common目录

• hadoop-common-2.7.2.jar
• hadoop-lzo-0.4.20.jar【如果没有使用lzo压缩，可忽略】

share/hadoop/tools/lib 目录

• hadoop-auth-2.7.2.jar
• htrace-core-3.1.0-incubating.jar
• commons-io-2.4.jar【这个包可能会有冲突，删除低版本】
• commons-configuration-1.6.jar

share/hadoop/hdfs目录

• hadoop-hdfs-2.7.2.jar

2. 如果使用lzo压缩，会报一个没有该压缩类型的错误，此时需要将hadoop中的core-site.xml文件放到--conf下

<configuration>
       <property>
            <name>fs.defaultFS</name>
            <value>hdfs://hadoop01:9000</value>
       </property>
       <property>
            <name>hadoop.tmp.dir</name>
            <value>/opt/datas/tmp</value>
       </property>
        <property>
            <name>io.compression.codecs</name>
            <value>
                org.apache.hadoop.io.compress.GzipCodec,
                org.apache.hadoop.io.compress.DefaultCodec,
                org.apache.hadoop.io.compress.BZip2Codec,
                org.apache.hadoop.io.compress.SnappyCodec,
                com.hadoop.compression.lzo.LzoCodec,
                com.hadoop.compression.lzo.LzopCodec
            </value>
        </property>

        <property>
            <name>io.compression.codec.lzo.class</name>
            <value>com.hadoop.compression.lzo.LzoCodec</value>
        </property>
</configuration>

如下是我启动flume的配置

/opt/envs/flume-1.7.0/bin/flume-ng agent --conf /opt/envs/flume-1.7.0/conf --conf-file /opt/envs/flume-1.7.0/conf/kafka-flume-hdfs.conf --name a1 -Dflume.root.logger=INFO,console

3. 找不到lzo 本地包

 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.RuntimeException: native-lzo library not available
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:451)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: native-lzo library not available
        at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
        at org.apache.flume.sink.hdfs.HDFSCompressedDataStream.open(HDFSCompressedDataStream.java:97)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:249)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:232)
        at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:668)
        at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
        at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:665)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more

解决方案：

1. 停止flume
2. yum install lzo

3. 大功告成