美文网首页大数据
大数据||MapReduce数据类型

大数据||MapReduce数据类型

作者: 明明德撩码 | 来源:发表于2018-02-13 08:23 被阅读78次

数据类型都实现了Writable接口,以便这些类型定义的数据可以被序列化进行网络传输和文件存储。

基本数据类型

BooleanWritable:标准布尔型数值
ByteWritable :单字节数值
DoubleWritable:双字节数值
FloatWritable:浮点型
IntWritable:整型数
LongWritable:长整型
Text
NullWritable:当<key,value>中的key或value为空使用。

MapReduce数据类型都实现了writable和comparable

实现了writable类型都要进行磁盘读写

实现了comparable类型数据还进行排序

LongWritable 数据类型样例

/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.hadoop.io;


import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;

/** A WritableComparable for longs. */
@InterfaceAudience.Public
@InterfaceStability.Stable
public class LongWritable implements WritableComparable<LongWritable> {
  private long value;

  public LongWritable() {}

  public LongWritable(long value) { set(value); }

  /** Set the value of this LongWritable. */
  public void set(long value) { this.value = value; }

  /** Return the value of this LongWritable. */
  public long get() { return value; }

  @Override
  public void readFields(DataInput in) throws IOException {
    value = in.readLong();
  }

  @Override
  public void write(DataOutput out) throws IOException {
    out.writeLong(value);
  }

  /** Returns true iff <code>o</code> is a LongWritable with the same value. */
  @Override
  public boolean equals(Object o) {
    if (!(o instanceof LongWritable))
      return false;
    LongWritable other = (LongWritable)o;
    return this.value == other.value;
  }

  @Override
  public int hashCode() {
    return (int)value;
  }

  /** Compares two LongWritables. */
  @Override
  public int compareTo(LongWritable o) {
    long thisValue = this.value;
    long thatValue = o.value;
    return (thisValue<thatValue ? -1 : (thisValue==thatValue ? 0 : 1));
  }

  @Override
  public String toString() {
    return Long.toString(value);
  }

  /** A Comparator optimized for LongWritable. */ 
  public static class Comparator extends WritableComparator {
    public Comparator() {
      super(LongWritable.class);
    }

    @Override
    public int compare(byte[] b1, int s1, int l1,
                       byte[] b2, int s2, int l2) {
      long thisValue = readLong(b1, s1);
      long thatValue = readLong(b2, s2);
      return (thisValue<thatValue ? -1 : (thisValue==thatValue ? 0 : 1));
    }
  }

  /** A decreasing Comparator optimized for LongWritable. */ 
  public static class DecreasingComparator extends Comparator {
    
    @Override
    public int compare(WritableComparable a, WritableComparable b) {
      return -super.compare(a, b);
    }
    @Override
    public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
      return -super.compare(b1, s1, l1, b2, s2, l2);
    }
  }

  static {                                       // register default comparator
    WritableComparator.define(LongWritable.class, new Comparator());
  }

}

相关文章

  • 【MapReduce】图形化纵览整个流程

    MapReduce作为分布式数据处理的框架,下面用一张图来诠释整个数据的处理流程。 数据类型在MapReduce中...

  • 大数据||MapReduce数据类型

    数据类型都实现了Writable接口,以便这些类型定义的数据可以被序列化进行网络传输和文件存储。 基本数据类型 B...

  • 数据仓库工具Hive

    数据仓库工具Hive Hive产生背景 直接使用MapReduce处理大数据,问题: MapReduce开放难度大...

  • java

    数据类型分为:8大基础数据类型和3大引用数据类型。 基础数据类型和引用数据类型的区别: 1,基本数据类型变量声明之...

  • Java中的基本数据类型

    基本数据类型 Java中的数据类型分为 内置数据类型和引用数据类型两种。内置数据类型分为4大类,包含 整数型(by...

  • Hive

    HadoopHDFS 存储数据YARN 资源管理MapReduce 处理数据Mapreduce is hard...

  • Hadoop 【基础】| MapReduce简介

    MapReduce,并行计算框架,用于处理大规模的数据集 一、MapReduce原理 采用分而治之的思想,将一个大...

  • MapReduce(5)数据类型和格式

    ##08.18 mapreduce 类型与格式 map函数 reduce函数 combiner函数 partiti...

  • 数据结构与算法(第一章)

    数据类型 系统定义的数据类型,也就是我们常说的基本数据类型,在java里面就是八大数据类型(int、float、d...

  • JavaScript基本数据类型

    JavaScript数据类型有2大分类:一是基本数据类型,二是特殊数据类型其中,基本数据类型包括以下3种:(1)数...

网友评论

    本文标题:大数据||MapReduce数据类型

    本文链接:https://www.haomeiwen.com/subject/glxdtftx.html