通过ganesha-nfs将Ceph导出为NFS（Luminou

作者: Patrick_QiWei | 来源:发表于2017-11-29 09:57 被阅读34次

通过ganesha-nfs将Ceph导出为NFS（Luminou
ceph 将cephFS导出成NFS使用实例
部署Ceph分布式存储，迁移网站数据到ceph集群
nfs-ganesha将rgw导出为nfs文件接口
OpenStack组件 Glance, Cinder, Swif
ceph-objectstore-tool导出对象Teminat
多机使用dockerswarm部署多个容器并行openmpi
网络文件共享nfs服务
关于 ip 端口监测的一点思路
rgw实现nfs

1. 环境：

        1. CentOS Linux release  7.4.1708 (Core)（Basic-web安装）
        2. Ceph Luminous 12.2.2
        3. ceph-deploy 1.5.39

主机	别名
192.168.1.50	node1
192.168.1.51	node2
192.168.1.53	test node

关于ceph的安装可参考ceph luminous安装

2. 安装依赖

yum install gcc git cmake autoconf libtool bison flex doxygen openssl-devel gcc-c++ krb5-libs krb5-devel libuuid-devel nfs-utils -y

如果要生成FSAL_RGW模块，需要安装librgw-devel

yum install librgw-devel -y

3. 源码下载

If ceph version is 12.2.1，you can download ganesha-2.5.3 & ganesha-2.5.2
If ceph version is 12.2.2，you can download ganesha-2.5.4

wget https://github.com/nfs-ganesha/nfs-ganesha/releases/tag/V2.5.4/nfs-ganesha-2.5.4.tar.gz

登录https://github.com/nfs-ganesha/ntirpc/tree/fadcbdedb65998925482b26c88964b4213b9e1ac网址，下载zip包

image.png

4.编译

编译nfs-ganesha时要打开RGW模块,在编译选项中添加： -DUSE_FSAL_RGW=ON

# Both on node1 & node2
unzip ntirpc-fadcbdedb65998925482b26c88964b4213b9e1ac.zip
tar -zxvf nfs-ganesha-2.5.4.tar.gz
cd ntirpc-fadcbdedb65998925482b26c88964b4213b9e1ac
mv * /root/nfs-ganesha-2.5.4/src/libntirpc
cd /root/nfs-ganesha-2.5.4/src/
mkdir build
cd build
cmake -DUSE_FSAL_RGW=ON ../

image.png

Note：在ceph12.2.2版本中，在cmake中RGW的version会显示为1.1.6

image.png

编译完成后查看输出内容，一定要确保： -- USE_FSAL_RGW为ON
编译和安装，在build目录下

make
make install

输出内容：
[ 18%] Built target ntirpc
[ 19%] Built target log
[ 21%] Built target config_parsing
[ 23%] Built target cidr
[ 24%] Built target avltree
[ 24%] Built target hashtable
[ 28%] Built target sal
[ 29%] Built target rpcal
[ 29%] Built target nfs4callbacks
[ 52%] Built target nfsproto
[ 54%] Built target nfs_mnt_xdr
[ 57%] Built target nlm
[ 57%] Built target gos
[ 58%] Built target string_utils
[ 60%] Built target rquota
[ 69%] Built target 9p
[ 69%] Built target sm_notify.ganesha
[ 70%] Built target hash
[ 70%] Built target netgroup_cache
[ 73%] Built target support
[ 73%] Built target uid2grp
[ 75%] Built target fsalnull
[ 78%] Built target fsalmdcache
[ 78%] Built target fsalpseudo
[ 79%] Built target fsalproxy
[ 81%] Built target fsalrgw
[ 86%] Built target fsalgpfs
[ 86%] Built target fsal_os
[ 88%] Built target fsalvfs
[ 89%] Built target fsalmem
[ 90%] Built target idmap
[ 93%] Built target FsalCore
[ 96%] Built target MainServices
[100%] Built target ganesha.nfsd
Install the project...
-- Install configuration: "Debug"
-- Installing: /etc/ganesha/ganesha.conf
-- Installing: /usr/share/doc/ganesha/config_samples
-- Installing: /usr/share/doc/ganesha/config_samples/README
-- Installing: /usr/share/doc/ganesha/config_samples/ceph.conf
-- Installing: /usr/share/doc/ganesha/config_samples/config.txt
-- Installing: /usr/share/doc/ganesha/config_samples/ds.conf
-- Installing: /usr/share/doc/ganesha/config_samples/export.txt
-- Installing: /usr/share/doc/ganesha/config_samples/ganesha.conf.example
-- Installing: /usr/share/doc/ganesha/config_samples/gluster.conf
-- Installing: /usr/share/doc/ganesha/config_samples/gpfs.conf
-- Installing: /usr/share/doc/ganesha/config_samples/gpfs.ganesha.exports.conf
-- Installing: /usr/share/doc/ganesha/config_samples/gpfs.ganesha.log.conf
-- Installing: /usr/share/doc/ganesha/config_samples/gpfs.ganesha.main.conf
-- Installing: /usr/share/doc/ganesha/config_samples/gpfs.ganesha.nfsd.conf
-- Installing: /usr/share/doc/ganesha/config_samples/logging.txt
-- Installing: /usr/share/doc/ganesha/config_samples/logrotate_fsal_gluster
-- Installing: /usr/share/doc/ganesha/config_samples/logrotate_ganesha
-- Installing: /usr/share/doc/ganesha/config_samples/mem.conf
-- Installing: /usr/share/doc/ganesha/config_samples/rgw.conf
-- Installing: /usr/share/doc/ganesha/config_samples/rgw_bucket.conf
-- Installing: /usr/share/doc/ganesha/config_samples/vfs.conf
-- Installing: /usr/share/doc/ganesha/config_samples/xfs.conf
-- Installing: /usr/share/doc/ganesha/config_samples/zfs.conf
-- Installing: /var/run/ganesha
-- Installing: /usr/lib64/pkgconfig/libntirpc.pc
-- Installing: /usr/include/ntirpc
-- Installing: /usr/include/ntirpc/fpmath.h
-- Installing: /usr/include/ntirpc/getpeereid.h
-- Installing: /usr/include/ntirpc/intrinsic.h
-- Installing: /usr/include/ntirpc/libc_private.h
-- Installing: /usr/include/ntirpc/misc
-- Installing: /usr/include/ntirpc/misc/abstract_atomic.h
-- Installing: /usr/include/ntirpc/misc/bsd_epoll.h
-- Installing: /usr/include/ntirpc/misc/city.h
-- Installing: /usr/include/ntirpc/misc/citycrc.h
-- Installing: /usr/include/ntirpc/misc/event.h
-- Installing: /usr/include/ntirpc/misc/opr.h
-- Installing: /usr/include/ntirpc/misc/opr_queue.h
-- Installing: /usr/include/ntirpc/misc/os_epoll.h
-- Installing: /usr/include/ntirpc/misc/portable.h
-- Installing: /usr/include/ntirpc/misc/queue.h
-- Installing: /usr/include/ntirpc/misc/rbtree.h
-- Installing: /usr/include/ntirpc/misc/rbtree_x.h
-- Installing: /usr/include/ntirpc/misc/socket.h
-- Installing: /usr/include/ntirpc/misc/stdint.h
-- Installing: /usr/include/ntirpc/misc/stdio.h
-- Installing: /usr/include/ntirpc/misc/timespec.h
-- Installing: /usr/include/ntirpc/misc/wait_queue.h
-- Installing: /usr/include/ntirpc/misc/winpthreads.h
-- Installing: /usr/include/ntirpc/namespace.h
-- Installing: /usr/include/ntirpc/netconfig.h
-- Installing: /usr/include/ntirpc/reentrant.h
-- Installing: /usr/include/ntirpc/rpc
-- Installing: /usr/include/ntirpc/rpc/auth.h
-- Installing: /usr/include/ntirpc/rpc/auth_des.h
-- Installing: /usr/include/ntirpc/rpc/auth_gss.h
-- Installing: /usr/include/ntirpc/rpc/auth_inline.h
-- Installing: /usr/include/ntirpc/rpc/auth_kerb.h
-- Installing: /usr/include/ntirpc/rpc/auth_unix.h
-- Installing: /usr/include/ntirpc/rpc/clnt.h
-- Installing: /usr/include/ntirpc/rpc/clnt_soc.h
-- Installing: /usr/include/ntirpc/rpc/clnt_stat.h
-- Installing: /usr/include/ntirpc/rpc/des.h
-- Installing: /usr/include/ntirpc/rpc/des_crypt.h
-- Installing: /usr/include/ntirpc/rpc/gss_internal.h
-- Installing: /usr/include/ntirpc/rpc/nettype.h
-- Installing: /usr/include/ntirpc/rpc/pmap_clnt.h
-- Installing: /usr/include/ntirpc/rpc/pmap_prot.h
-- Installing: /usr/include/ntirpc/rpc/pmap_rmt.h
-- Installing: /usr/include/ntirpc/rpc/pool_queue.h
-- Installing: /usr/include/ntirpc/rpc/raw.h
-- Installing: /usr/include/ntirpc/rpc/rpc.h
-- Installing: /usr/include/ntirpc/rpc/rpc_cksum.h
-- Installing: /usr/include/ntirpc/rpc/rpc_com.h
-- Installing: /usr/include/ntirpc/rpc/rpc_msg.h
-- Installing: /usr/include/ntirpc/rpc/rpcb_clnt.h
-- Installing: /usr/include/ntirpc/rpc/rpcb_prot.h
-- Installing: /usr/include/ntirpc/rpc/rpcb_prot.x
-- Installing: /usr/include/ntirpc/rpc/rpcent.h
-- Installing: /usr/include/ntirpc/rpc/svc.h
-- Installing: /usr/include/ntirpc/rpc/svc_auth.h
-- Installing: /usr/include/ntirpc/rpc/svc_rqst.h
-- Installing: /usr/include/ntirpc/rpc/svc_soc.h
-- Installing: /usr/include/ntirpc/rpc/tirpc_compat.h
-- Installing: /usr/include/ntirpc/rpc/types.h
-- Installing: /usr/include/ntirpc/rpc/work_pool.h
-- Installing: /usr/include/ntirpc/rpc/xdr.h
-- Installing: /usr/include/ntirpc/rpc/xdr_inline.h
-- Installing: /usr/include/ntirpc/rpc/xdr_inrec.h
-- Installing: /usr/include/ntirpc/rpc/xdr_ioq.h
-- Installing: /usr/include/ntirpc/rpcsvc
-- Installing: /usr/include/ntirpc/rpcsvc/crypt.h
-- Installing: /usr/include/ntirpc/un-namespace.h
-- Installing: /usr/include/ntirpc/version.h
-- Installing: /usr/lib64/libntirpc.so.1.5.3
-- Installing: /usr/lib64/libntirpc.so.1.5
-- Installing: /usr/lib64/libntirpc.so
-- Installing: /usr/lib64/ganesha/libfsalnull.so.4.2.0
-- Installing: /usr/lib64/ganesha/libfsalnull.so.4
-- Installing: /usr/lib64/ganesha/libfsalnull.so
-- Installing: /usr/lib64/ganesha/libfsalproxy.so.4.2.0
-- Installing: /usr/lib64/ganesha/libfsalproxy.so.4
-- Installing: /usr/lib64/ganesha/libfsalproxy.so
-- Removed runtime path from "/usr/lib64/ganesha/libfsalproxy.so.4.2.0"
-- Installing: /usr/lib64/ganesha/libfsalrgw.so.4.2.0
-- Installing: /usr/lib64/ganesha/libfsalrgw.so.4
-- Installing: /usr/lib64/ganesha/libfsalrgw.so
-- Removed runtime path from "/usr/lib64/ganesha/libfsalrgw.so.4.2.0"
-- Installing: /usr/lib64/ganesha/libfsalgpfs.so.4.2.0
-- Installing: /usr/lib64/ganesha/libfsalgpfs.so.4
-- Installing: /usr/lib64/ganesha/libfsalgpfs.so
-- Removed runtime path from "/usr/lib64/ganesha/libfsalgpfs.so.4.2.0"
-- Installing: /usr/lib64/ganesha/libfsalvfs.so.4.2.0
-- Installing: /usr/lib64/ganesha/libfsalvfs.so.4
-- Installing: /usr/lib64/ganesha/libfsalvfs.so
-- Removed runtime path from "/usr/lib64/ganesha/libfsalvfs.so.4.2.0"
-- Installing: /usr/lib64/ganesha/libfsalmem.so.4.2.0
-- Installing: /usr/lib64/ganesha/libfsalmem.so.4
-- Installing: /usr/lib64/ganesha/libfsalmem.so
-- Removed runtime path from "/usr/lib64/ganesha/libfsalmem.so.4.2.0"
-- Installing: /usr/bin/ganesha.nfsd
-- Removed runtime path from "/usr/bin/ganesha.nfsd"

编辑配置文件

注意Path后面的路径需要加引号，rgw替换成创建S3用户生成的用户信息

vim /etc/ganesha/ganesha.conf
# 在ganesha.conf文件中输入如下内容
EXPORT
{
        Export_ID=2;
        Path = "/";
        Pseudo = /rgw;
        Access_Type = RW;
        Squash = No_root_squash;
        NFS_Protocols = 4;
        Transport_Protocols = TCP;
        FSAL {
                Name = RGW;
                User_Id = "testuser";
                Access_Key_Id ="1MWH3LWM1BS4ZF4HN5IH";
                Secret_Access_Key = "cuObxYgtl1lJgqNxOIpENycVqXfxLxZ8z5IXDM0O";
        }
}
RGW {
    ceph_conf = "/etc/ceph/ceph.conf";
}

其中，User_Id , Access_Key_Id, Secret_Access_Key 根据自己的环境进行修改。关于如何获得这些参数可参考Ceph RGW安装和创建

5.启动 Ganesha

ganesha.nfsd -f /etc/ganesha/ganesha.conf -L /var/log/nfs-ganesha.log -N NIV_DEBUG

6.Check Export

[root@node1 build]# showmount -e  
Export list for node1:  
/ (everyone)

librgw init failed (-5)的解决方法
按照上面的步骤部署好后，输入showmount -e回车，未出现上面的输出目录，查看log可见如下提示

07/12/2017 15:47:45 : epoch 5a28f221 : node2 : nfs-ganesha-23362[main] create_export :FSAL :CRIT :RGW module: librgw init failed (-5)
07/12/2017 15:47:45 : epoch 5a28f221 : node2 : nfs-ganesha-23362[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL RGW

解决方法就是关闭CephX，将/etc/ceph/ceph.conf内的三个 cephx改为none，然后重启ceph

vim /etc/ceph/ceph.conf
# cephx修改成none
auth_cluster_required = none
auth_service_required = none
auth_client_required = none

systemctl restart ceph.target                    # 在node1、node2上重启ceph
systemctl reset-failed ceph-mgr@node1.service    # 在node1上重启mgr
systemctl start ceph-mgr.target

7.挂载 NFS

mount -t nfs -o nfsvers=4,noauto,soft,proto=tcp <ganesha-host-name>:/ <mount-point>
[root@testNode mnt]# mount -t nfs -o nfsvers=4,noauto,soft,proto=tcp 192.168.1.50:/ /mnt/nfs-ganesha
[root@testNode mnt]# tree nfs-ganesha/
nfs-ganesha/
└── node1
    └── my-new-bucket

通过ganesha-nfs将Ceph导出为NFS（Luminou
1. 环境：关于ceph的安装可参考ceph luminous安装 2. 安装依赖如果要生成FSAL_RGW模...
ceph 将cephFS导出成NFS使用实例
在window上挂载Ceph FS
部署Ceph分布式存储，迁移网站数据到ceph集群
使用三台服务器部署Ceph分布式存储，实现Ceph文件系统共享，将网站数据从NFS迁移到Ceph存储实验拓扑如下...
nfs-ganesha将rgw导出为nfs文件接口
概述我们知道ceph是统一的分布式存储系统，其可以提供给应用三种访问数据的接口：对象(RGW)、块(RBD)和文...
OpenStack组件 Glance, Cinder, Swif
Glance模块作用：提供镜像模板支持本地存储，NFS，Swift和Ceph等。 Swift是将存储的文件打成一...
ceph-objectstore-tool导出对象Teminat
导出对象的方法： 1、 stop ceph-osd id=x 2、 ceph-objectstore-tool -...
多机使用dockerswarm部署多个容器并行openmpi
将镜像导出到node节点，并开启nfs 初始化docker swarm环境使用docker-compose文件创...
网络文件共享nfs服务
NFS服务 NFS服务介绍 NFS服务主要进程：配置防火墙 NFS配置文件每个条目指定目录导出到的哪些主机，及...
关于 ip 端口监测的一点思路
线上有ceph 对象存储集群以及ceph 块存储集群定位一个nas nfs 挂载时而可以时而不可以的问题首先...
rgw实现nfs
ceph一、功能介绍关于rgw实现nfs接口这个，刚接触的人可能并不清楚这个是个什么样的服务架构，rgw是ceph...