016 大数据中的职业和工作角色-综合指南

016 Careers and Job Roles in Big Data – A Comprehensive Guide

1. Objective

1. 目标

This tutorial on Careers and job roles in Big Data will help you in understanding the need of Big Data, the market trend for Big Data, various companies hiring Big Data Professionals from fresher to experienced professionals, different profiles in Big data to grow the career in like Hadoop developer, Hadoop admin, Hadoop architect, Hadoop tester and Big data analyst along with their roles and responsibilities, skills and experience required for different Big Data profiles.

这个关于职业和工作角色的教程 大数据 将帮助您了解大数据的需求、大数据的市场趋势、各大公司从刚入行专业人员到经验丰富的专业人员, 大数据的不同概况，为职业发展提供了 Hadoop开发人员、 Hadoop 管理员、 Hadoop 架构师、 Hadoop 测试人员和大数据分析师，以及不同大数据配置文件所需的角色和职责、技能和经验.

Careers and Job Roles in Big Data - A Comprehensive Guide

Careers and Job Roles in Big Data – A Comprehensive Guide

2. Introduction to Careers and job roles in Big data

2. 大数据职业和就业

Huge data is getting generated every second and if gathered all at daily basis, it leads to 2.5 quintillion bytes of data which is being generated every day.

每秒钟都会产生大量数据，如果每天都收集到这些数据，那么每天会产生 2.5 quintillion 字节的数据.

90 % of this data is being generated in just last 2 years which has led to huge explosion of data. You can imagine from the day of its inception till last 2 years, generation of data has contributed only 10 % of the total data available in the planet. So data generation is exponential with time and not linear.

90% 的数据是在过去两年产生的，这直接导致了数据大爆炸. 你可以想象，从数据诞生之日到过去两年，数据的产生只贡献了地球上所有可用数据的 10%. 因此，数据的产生是指数级而非线性的.

So as we have seen the various sources of data and how much data is getting generated every day, we can conclude that this data is growing at the rate of 40 % compounded annually. If we calculate total data that will be generated by 2020, it comes to around 45 ZB that shows how Big data is growing and would be the most demanding technology in coming years. This would give the Big data professionals a very bright future to work with.

因此，正如我们已经看到的各种数据来源以及每天产生多少数据一样，我们可以得出结论，这些数据以每年 40% 的复合速度增长. 如果我们计算到 2020 年将产生的 大约 45 个 ZB 总数据大数据这将是未来几年最苛刻的技术. 这将给大数据专业人士带来非常光明的未来.

A research by MarketsandMarkets estimates that Hadoop and Big Data Analytics market is anticipated to reach $13.9 billion by the end of 2017.

MarketsandMarkets 的一项研究估计，到 2017年底，Hadoop 和大数据分析市场预计将达到 139 亿美元.

Hadoop jobs are not only being offered by IT companies but all types of companies are hiring High paid Hadoop candidates including financial firms, retail organizations, banks, healthcare organizations etc. There is huge demand for Hadoop developer jobs and Hadoop administration jobs amongst start-ups that are building Hadoop directly into their business plans. Companies like EMC Corporation, Apple, Facebook, Google, Oracle, Hortonworks, IBM, Microsoft, Cisco, etc. have several Hadoop job openings with various positions like Hadoop Developers, Hadoop Testers, Hadoop Architects and Hadoop Administrators in almost all cities of India with most demand being in Bangalore, Pune, Mumbai, Hyderabad, Chennai, New Delhi and NCR.

Hadoop 工作不仅由 IT 公司提供，所有类型的公司都在招聘高薪的 Hadoop 候选人，包括金融公司、零售组织、银行、医疗保健组织等. 初创企业对 Hadoop 开发人员作业和 Hadoop 管理作业有着巨大的需求，他们正在将 Hadoop 直接构建到他们的业务计划中 EMC Corporation 、 Apple 、 Facebook 、 Google 、 Oracle 、 Hortonworks 、 IBM 、 Microsoft 、 Cisco 等公司都有 Hadoop 职位空缺，包括 Hadoop 开发人员、 Hadoop 测试人员、在印度几乎所有城市，Hadoop 架构师和 Hadoop 管理员的需求最多的是班加罗尔、浦那、孟买、海得拉巴、钦奈、新德里和 NCR.

One of the most common questions being asked is “What are the various jobs available in Hadoop”. Here is a brief overview on the various Hadoop job descriptions.

被问到的最常见的问题之一是 “Hadoop 中有哪些工作”. 以下是对各种 Hadoop 工作描述的简要概述.

3. Hadoop Developer

3. Hadoop 开发

Hadoop developer jobs responsibilities are to write programs as per the system designs and must have fair knowledge about the coding and programming. Task of Hadoop developer is similar to software developer but in Big Data domain. Job of Hadoop developer also includes understanding and working to come up with solutions to problems, designing and architecting along with strong documenting skills.

Hadoop开发人员的工作职责是根据系统设计编写程序，并且必须对编码和编程有充分的了解. Hadoop 开发人员的任务类似于软件开发人员，但在大数据领域，Hadoop 开发人员的工作还包括理解和工作，提出问题的解决方案，设计和架构，以及强大的记录技能.

i. Hadoop Developer Roles and Responsibilities

i. Hadoop 开发人员的和职责

• Defining job flows

• Managing and Reviewing Hadoop Log Files

• Cluster Coordination services through Zookeeper

• Managing Hadoop jobs through scheduler

• Supporting MapReduce programs running on the Hadoop cluster

• 定义工作流程

• 管理和审查 Hadoop 日志文件

• 分组协调服务管理员

• 通过调度器管理 Hadoop 作业

• 支持MapReduceHadoop 集群上运行的程序

ii. Skills Required for Hadoop Developer:

Ii.Hadoop 开发人员所需技能:

• Expertise in writing Pig Latin Scripts and HiveQL

• Proven Knowledge of workflow/ schedulers like Oozie

• Ability to deal with Data Loading tools like Flume and Sqoop

• 写作方面的专业知识 Pig脚本和 HiveQL (译者注: Pig 国内使用的已经比较少，我是在 12-13 年使用作为主要数据开发工具，目前 Hive、Spark 依然是主流)

• 对 Oozie 等工作流/调度器有丰富的知识 (译者注: 可关注 Azkaban、Airflow)

• 能够处理数据加载工具，如 Flume和 Sqoop

iii. Experience required for a Hadoop developer profile

Iii.Hadoop 开发人员配置文件所需的经验

0-5 years
Hadoop developer resume must be a blend of above skills to be selected by companies hiring Hadoop developer in India and abroad and get high range of Hadoop developer salary. For more details on Hadoop developer certification by Cloudera, visit Details of Cloudera certification courses.

Hadoop 开发人员简历必须是由在印度和国外雇佣 Hadoop 开发人员的公司选择的上述技能的混合，并获得高范围的 Hadoop 开发人员工资.有关 Cloudera 认证 Hadoop 开发人员的更多详细信息，请访问Cloudera 认证课程详情.

4. Hadoop Architect

4. Hadoop 架构师

Hadoop architect jobs responsibilities are to design how the system should work and then manage the complete life cycle of a Hadoop solution. He must have subject matter expertise and delivery experience working on popular Hadoop distribution platforms like Cloudera, HortonWorks, and MapR.

Hadoop 架构师的工作职责是设计系统应该如何工作，然后管理 Hadoop 解决方案的完整生命周期.他必须拥有在 Cloudera 、 HortonWorks 和 MapR 等流行 Hadoop 分发平台上工作的主题专业知识和交付经验.

i. Hadoop Architect Roles and Responsibilities

Hadoop 架构师角色和职责

• Designing the Technical architecture and application customization

• Undertaking the necessary Requirement analysis

• Undertaking the deployment of proposed solution design

• 技术架构设计与应用定制

• 进行必要的需求分析

• 承担提议的解决方案设计的部署

ii. Skills Required for Hadoop Architect

Ii.Hadoop 架构师所需技能

• Indepth knowledge about Hadoop Architecture and HDFS including YARN

• Hive, Pig

• Java MapReduce

• HBase

• 深入了解Hadoop架构 和 HDFS包括 Yarn

• Hive, Pig

• Java MapReduce

• HBase

iii. Experience required for a Hadoop architect profile

Iii.Hadoop 架构师配置文件所需的经验

8+ years
Hadoop architect resume must be a blend of above skills to be selected by companies hiring Hadoop architects in India and abroad.

8 年以上
Hadoop 架构师简历必须是由在印度和国外雇佣 Hadoop 架构师的公司选择的上述技能的混合.

5. Hadoop Administrator

5. Hadoop 管理员

The Hadoop Administration jobs responsibilities are similar to the System administrator job. Hadoop admin roles and responsibilities include setting up Hadoop clusters, backup, recovery and maintenance of the same. Good knowledge of hardware systems and Hadoop architecture is required by Hadoop administrator.

Hadoop 管理作业职责与系统管理员作业相似.Hadoop 管理员角色和职责包括设置 Hadoop 集群、备份、恢复和维护 Hadoop 集群.Hadoop 管理员需要对硬件系统和 Hadoop 体系结构有很好的了解.

i. Hadoop admin roles and responsibilities:

Hadoop 管理员角色和职责

• HDFS maintenance and support

• Hadoop Clusters planning and screening

• Tracking of all the connectivity and security issues

• Setting up new Hadoop users

• HDFS 维护和支持

• Hadoop 集群规划与筛选

• 跟踪所有连接和安全问题

• 建立新的 Hadoop 用户

ii. Skills Required for Hadoop administrator:

Ii.Hadoop 管理员所需技能:

• In-depth knowledge of Hbase

• Hands on experience in Oozie, HCatalog, Hive

• Strong scripting skills in Linux environment

• 深入了解 Hbase

• 在 Oozie，HCatalog 的实践经验, Hive

• 强大的脚本编写技能 Linux环境

iii. Experience required for a Hadoop admin profile

Iii.Hadoop 管理配置文件所需的经验

0-5 years
Hadoop administrator resume must be a blend of above skills to be selected by companies hiring Hadoop administrators in India and abroad.

0-5 年
Hadoop管理员简历必须是印度和国外招聘 Hadoop 管理员的公司选择的上述技能的混合.

6. Hadoop Tester

6. Hadoop 测试工程师

Hadoop tester jobs responsibilities are to ensure that the process being designed is in sync with the process being required by the organization. His main role is to find and fix bugs in Hadoop applications.

Hadoop 测试人员的工作职责是确保正在设计的流程与组织要求的流程同步.他的主要职责是发现和修复 Hadoop 应用程序中的错误.

i. Hadoop Tester Roles and Responsibilities

Hadoop 测试工程师的角色和职责

• Main responsibility includes troubleshooting and finding defects in the program and reporting them to Hadoop developer to take corrective action

• Reporting all the Positive and negative test cases in Hadoop/Pig/Hive components

• 主要职责包括故障排除和发现程序中的缺陷，并向 Hadoop 开发人员报告以采取纠正措施

• 报告 Hadoop/Pig/Hive 组件中的所有阳性和阴性测试用例

ii. Skills required for Hadoop Tester

Ii.Hadoop 测试人员所需技能

• Testing Knowledge of JUnit, MRUnit framework

• Knowledge of Java to test MapReduce Jobs

• Hands on knowledge of Hive, Pig

• JUnit 、 MRUnit 框架的测试知识

• Java 测试知识 MapReduce工作岗位

• Hive、Pig 的知识

iii. Experience required for a Hadoop tester profile

Iii.Hadoop 测试工程师 配置文件所需的经验

0-5 years
Hadoop tester resume must be a blend of above skills to be selected by companies hiring Hadoop testers in India and abroad.

0-5 年
Hadoop 测试人员简历必须是由在印度和国外雇佣 Hadoop 测试人员的公司选择的上述技能的混合.

译者注: 如对Hadoop 大数据测试工程师感兴趣的同学，建议从白盒测试工程师做起，自动化、性能、脚本、不同数据格式自动处理等，掌握开发、脚本功底会让你的工作如虎添翼

7. Hadoop Analyst

7. Hadoop 分析工程师

Hadoop analyst jobs responsibilities are to analyze large amounts of data and bring out insights that could be used by the companies for their betterment.

Hadoop 分析师的工作职责是分析大量数据，并提出公司可以用来改进的见解.

i. Hadoop analyst Roles and Responsibilities

Hadoop 分析员角色和职责

• Developing new insights from the available data using scripting language

• Doing A/B testing as per different hypotheses to impact different Key Performance Indicators

• 使用脚本语言从可用数据中开发新的见解

• 根据不同的假设进行 A/B 测试，以影响不同的关键绩效指标

ii. Skills required for Hadoop Analyst

Ii.Hadoop 分析师所需技能

• Hands on knowledge of Hive, Pig

• In depth Flume knowledge and SQL command

• Hive、Pig

• 在深度 Flume知识和 SQL 命令

译者注: 一般数据分析师不需要了解 Flume，但强大的 SQL 调优能力是必须的，否则会被架构师、集群管理员请去喝茶。

iii. Experience required for a Hadoop analyst profile

Iii.Hadoop 分析师档案所需的经验

0-5 years
Hadoop analyst resume must be a blend of the above skills to be selected by companies hiring Hadoop analysts in India and abroad.

0-5 年
Hadoop 分析师简历必须是由在印度和国外雇佣 Hadoop 分析师的公司选择的上述技能的混合.

https://data-flair.training/blogs/careers-job-roles-big-data-comprehensive-guide