Phoenix：
phoenix.connection.consistency
HBase：
hbase.tmp.dir
hbase.local.dir
hbase.regionserver.handler.count 3*48
hbase.ipc.server.callqueue.handler.factor
hbase.ipc.server.callqueue.read.ratio
hbase.ipc.server.callqueue.scan.ratio
hbase.client.write.buffer 2097152*4
hbase.client.max.perserver.tasks
hbase.client.max.perregion.tasks
hbase.client.localityCheck.threadPoolSize
hbase-site.xml
<property>
<name>hbase.ipc.client.tcpnodelay</name>
<value>true</value>
</property>
<property>
<name>hbase.ipc.server.tcpnodelay</name>
<value>true</value>
</property>
<property>
<name>dfs.client.read.shortcircuit</name>
<value>true</value>
</property>
<property>
<name>dfs.client.read.shortcircuit.skip.checksum</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.avoid.read.stale.datanode</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.avoid.write.stale.datanode</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.stale.datanode.interval</name>
<value>60000</value>
</property>
<property>
<name>dfs.namenode.write.stale.datanode.ratio</name>
<value>0.3f</value>
</property>
hdfs相关配置：
dfs.datanode.synconclose 设为true，当为false时，系统重启或断电时有可能数据丢失，默认值是false。
当写操作完成之后，缓存中的block不会立即被写入磁盘，如果要同步将缓存的block写入磁盘，用户需要将“hdfs-site.xml”中的dfs.datanode.synconclose设置为true。更改此设置后，对性能可能存在影响。
dfs.datanode.sync.behind.writes=FALSE 如果是true，写之后，DN将指示操作系统把队列中的数据全部立即写磁盘。和常用的OS策略不同，它们可能在触发写磁盘之前等待30
sdfs.namenode.avoid.write.stale.datanode —— default: true
dfs.namenode.avoid.read.stale.datanode —— default: true
dfs.namenode.stale.datanode.interval —— default: 30 seconds
默认是true，超过30s未收到heartbeat的datanode，namenode会将之判为最低优先级的读写
Set the connection property Consistency to timeline in the JDBC connect string
-XX:+UseCMSCompactAtFullCollection 与 -XX:CMSFullGCsBeforeCompaction=1
-XX:CMSInitiatingOccupancyFraction=70 和-XX:+UseCMSInitiatingOccupancyOnly
-XX:+CMSScavengeBeforeRemark
core-site.xml
<property>
<name>ipc.server.tcpnodelay</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.stale.datanode.interval</name>
<value>true</value>
</property>
hdfs-site.xml
<property>
<name>dfs.namenode.avoid.read.stale.datanode</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.avoid.write.stale.datanode</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.stale.datanode.interval</name>
<value>60000</value>
</property>
<property>
<name>dfs.namenode.write.stale.datanode.ratio</name>
<value>0.3f</value>
</property>
<!--python -c 'import math ; print int(math.log(16) * 20)' = 60-->
<property>
<name>dfs.namenode.service.handler.count</name>
<value>60</value>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>60</value>
<description>The number of server threads for the namenode.</description>
</property>
<property>
<name>dfs.datanode.handler.count</name>
<value>60</value>
</property>
<property>
<name>dfs.datanode.max.transfer.threads</name>
<value>409600</value>
</property>
<!--http://muziyuchen.com/hdfs-6/-->
<property>
<name>dfs.client.block.write.locateFollowingBlock.retries</name>
<value>30</value>
</property>
yarn-site.xml
<!--Yarn Agg Configuration-->
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/disk1/yarnlogs,/home/disk2/yarnlogs,/home/disk3/yarnlogs,/home/disk4/yarnlogs,/home/disk5/yarnlogs,/home/disk6/yarnlogs,/home/disk7/yarnlogs,/home/disk8/yarnlogs</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>864000</value>
</property>
<property>
<name>yarn.log-aggregation.retain-check-interval-seconds</name>
<value>-1</value>
</property>
<property>
<name>yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds</name>
<value>10800</value>
</property>
<property>
<name>yarn.nodemanager.delete.debug-delay-sec</name>
<value>10800</value>
</property>
<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>10800</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/hadoop/yarn</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
<value>logs</value>
</property>
<!--Yarn Log Dirs-->
<property>
<name>yarn.nodemanager.disk-health-checker.min-healthy-disks</name>
<value>0.25</value>
</property>
<property>
<name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
<value>98.1</value>
</property>
<property>
<name>yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb</name>
<value>0</value>
</property>
<property>
<name>yarn.resourcemanager.fs.state-store.uri</name>
<value>${hadoop.tmp.dir}/yarn/system/rmstore</value>
</property>
<property>
<name>yarn.resourcemanager.leveldb-state-store.path</name>
<value>${hadoop.tmp.dir}/yarn/system/rmstore</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>${hadoop.tmp.dir}/nm-local-dir</value>
</property>
<property>
<name>yarn.nodemanager.recovery.dir</name>
<value>${hadoop.tmp.dir}/yarn-nm-recovery</value>
</property>
<property>
<name>yarn.timeline-service.leveldb-timeline-store.path</name>
<value>${hadoop.tmp.dir}/yarn/timeline</value>
</property>
<property>
<name>yarn.timeline-service.leveldb-state-store.path</name>
<value>${hadoop.tmp.dir}/yarn/timeline</value>
</property>
网友评论