美文网首页Hadoop
115.Kerberos的安装配置与启用

115.Kerberos的安装配置与启用

作者: 大勇任卷舒 | 来源:发表于2022-08-14 16:12 被阅读0次
    • 在CM服务器上安装KDC服务
    $ sudo yum -y install krb5-server krb5-libs krb5-auth-dialog krb5-workstation
    • /etc/krb5.conf配置修改
    [ec2-user@ip-xxx-xx-xx-xx ~]$ sudo vim /etc/krb5.conf
# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log
[libdefaults]
 dns_lookup_realm = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
 default_realm = FAYSON.COM
 #default_ccache_name = KEYRING:persistent:%{uid}
[realms]
 FAYSON.COM = {
  kdc = ip-172-31-16-68.ap-southeast-1.compute.internal
  admin_server = ip-xxx-xx-xx-xx.ap-southeast-1.compute.internal
 }
[domain_realm]
 .ap-southeast-1.compute.internal = FAYSON.COM
 ap-southeast-1.compute.internal = FAYSON.COM
    • /var/kerberos/krb5kdc/kadm5.acl配置修改
    $ sudo vim /var/kerberos/krb5kdc/kadm5.acl
*/admin@FAYSON.COM      *
    • /var/kerberos/krb5kdc/kdc.conf配置修改
    [ec2-user@ip-xxx-xx-xx-xx ~]$ sudo vim /var/kerberos/krb5kdc/kdc.conf 
[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88
[realms]
 FAYSON.COM = {
  #master_key_type = aes256-cts
  max_renewable_life= 7d 0h 0m 0s
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hma
c:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:norm
al des-cbc-crc:normal
 }
    • Kerberos数据库创建
    [ec2-user@ip-xxx-xx-xx-xx ~]$ sudo kdb5_util create -r FAYSON.COM -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'FAYSON.COM',
master key name 'K/M@FAYSON.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key: 
Re-enter KDC database master key to verify: 
[ec2-user@ip-xxx-xx-xx-xx ~]$
    • Kerberos的管理账号创建
    [ec2-user@ip-xxx-xx-xx-xx ~]$ sudo kadmin.local
Authenticating as principal root/admin@FAYSON.COM with password.
kadmin.local:  addprinc admin/admin@FAYSON.COM
WARNING: no policy specified for admin/admin@FAYSON.COM; defaulting to no policy
Enter password for principal "admin/admin@FAYSON.COM": 
Re-enter password for principal "admin/admin@FAYSON.COM": 
Principal "admin/admin@FAYSON.COM" created.
kadmin.local:  exit
[ec2-user@ip-xxx-xx-xx-xx ~]$
    • 启动krb5kdc和kadmin服务，Kerberos服务添加到自启动服务
    $ sudo systemctl enable krb5kdc
$ sudo systemctl enable kadmin
$ sudo systemctl start krb5kdc
$ sudo systemctl start kadmin
    • 管理员账号测试
    [ec2-user@ip-xxx-xx-xx-xx ~]$ kinit admin/admin@FAYSON.COM
Password for admin/admin@FAYSON.COM: 
[ec2-user@ip-xxx-xx-xx-xx ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: admin/admin@FAYSON.COM
Valid starting       Expires              Service principal
02/04/2018 01:35:10  02/05/2018 01:35:10  krbtgt/FAYSON.COM@FAYSON.COM
        renew until 02/11/2018 01:35:10
[ec2-user@ip-xxx-xx-xx-xx ~]$
    • 安装Kerberos客户端
    # sh ssh_do_all.sh node.list 'yum -y install krb5-libs krb5-workstation'
    • CM Server服务器上安装额外的包
    $ sudo yum -y install openldap-clients
    • 将KDC Server上的krb5.conf文件拷贝到所有Kerberos客户端
    # sh bk_cp.sh node.list /etc/krb5.conf /etc/
    • 给CM添加管理员账号
    [ec2-user@ip-xxx-xx-xx-xx ~]$ sudo kadmin.local
Authenticating as principal root/admin@FAYSON.COM with password.
kadmin.local:  addprinc cloudera-scm/admin@FAYSON.COM
WARNING: no policy specified for cloudera-scm/admin@FAYSON.COM; defaulting to no policy
Enter password for principal "cloudera-scm/admin@FAYSON.COM": 
Re-enter password for principal "cloudera-scm/admin@FAYSON.COM": 
Principal "cloudera-scm/admin@FAYSON.COM" created.
kadmin.local:  exit
[ec2-user@ip-xxx-xx-xx-xx ~]$










    • 创建一个fayson的principal
    [ec2-user@ip-xxx-xx-xx-xx ~]$ sudo kadmin.local
Authenticating as principal root/admin@FAYSON.COM with password.
kadmin.local:  addprinc fayson@FAYSON.COM
WARNING: no policy specified for fayson@FAYSON.COM; defaulting to no policy
Enter password for principal "fayson@FAYSON.COM": 
Re-enter password for principal "fayson@FAYSON.COM": 
Principal "fayson@FAYSON.COM" created.
kadmin.local:
    • fayson用户登录Kerberos
    [ec2-user@ip-xxx-xx-xx-xx ~]$ sudo kadmin.local
Authenticating as principal root/admin@FAYSON.COM with password.
kadmin.local:  addprinc fayson@FAYSON.COM
WARNING: no policy specified for fayson@FAYSON.COM; defaulting to no policy
Enter password for principal "fayson@FAYSON.COM": 
Re-enter password for principal "fayson@FAYSON.COM": 
Principal "fayson@FAYSON.COM" created.
kadmin.local:  exit
[ec2-user@ip-xxx-xx-xx-xx ~]$ kdestroy
[ec2-user@ip-xxx-xx-xx-xx ~]$ kinit fayson
    • 所有节点添加fayson用户
    # sh ssh_do_all.sh node.list "useradd fayson"
    • 运行MapReduce
    $ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 10 1
    • beeline连接hive
    [ec2-user@ip-xxx-xx-xx-xx ~]$ beeline 
Beeline version 1.1.0-cdh5.14.0 by Apache Hive
beeline> !connect jdbc:hive2://localhost:10000/;principal=hive/ip-xxx-xx-xx-xx.ap-southeast-1.compute.internal@FAYSON.COM
scan complete in 1ms
Connecting to jdbc:hive2://localhost:10000/;principal=hive/ip-xxx-xx-xx-xx.ap-southeast-1.compute.internal@FAYSON.COM
Connected to: Apache Hive (version 1.1.0-cdh5.14.0)
Driver: Hive JDBC (version 1.1.0-cdh5.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://localhost:10000/> show tables;
    • test表
    0: jdbc:hive2://localhost:10000/> create table test(id int, name string);
    • 插入数据
    0: jdbc:hive2://localhost:10000/> insert into test values(1, 'fayson');
    • 执行语句
    0: jdbc:hive2://localhost:10000/> select count(*) from test;
    • Kerberos用户身份运行MapReduce报错
    main : run as user is fayson
main : requested yarn user is fayson
Requested user fayson is not whitelisted and has id 501,which is below the minimum allowed 1000
Failing this attempt. Failing the application.
17/09/02 20:05:04 INFO mapreduce.Job: Counters: 0
Job Finished in 6.184 seconds
java.io.FileNotFoundException: File does not exist: hdfs://ip-xxx-xx-x-148:8020/user/fayson/QuasiMonteCarlo_1504382696029_1308422444/out/reduce-out
        at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1266)
        at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1258)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1258)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1820)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1844)
        at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
        at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

    • 因为Yarn限制了用户id小于10000的用户提交，所以修改Yarn的min.user.id来解决


    • kinit操作后，执行MR报“User fayson not found”
      • 因为在集群的节点上没有fayson这个用户，所以要在集群所有节点添加fayson用户

    大数据视频推荐：
    腾讯课堂
    CSDN
    AIops智能运维机器学习算法实战
    ELK入门精讲
    AIOps智能运维实战
    ELK7 stack开发运维
    大数据语音推荐：
    ELK7 stack开发运维
    企业级大数据技术应用
    大数据机器学习案例之推荐系统
    自然语言处理
    大数据基础
    人工智能：深度学习入门到精通

    相关文章

      网友评论

        本文标题：115.Kerberos的安装配置与启用

        本文链接：https://www.haomeiwen.com/subject/fcyxgrtx.html

        延伸阅读

        深度阅读

        • 您也可以注册成为美文阅读网的作者，发表您的原创作品、分享您的心情！

        栏目导航

        热点阅读

        Hadoop

        关于我们|服务条款|联系我们|115.Kerberos的安装配置与启用|投稿指南|网站地图|RSS订阅|排版工具|手机版

        提供经典美文摘抄,优美散文欣赏,现代诗歌精选,短篇小说,心情随笔,表白情书范文,故事会在线阅读欣赏

        Copyright © 2014-2023 Haomeiwen.com All Rights Reserved. 好美文阅读网 版权所有

        备案信息：桂公网安备 45052102000051号 · 桂ICP备13007215号-3

        本站所收录作品、热点评论等信息部分来源互联网，目的只是为了系统归纳学习和传递资讯

        所有作品版权归原创作者所有，与本站立场无关，如不慎侵犯了你的权益，请联系我们告知，我们将做删除处理！