nfs-ganesha - Recovery

1. Server端Recovery过程

1. CREATE_SESSION(即mount)会触发nfs4_add_clid，将clientid存在backend
2. DESTROY_CLIENTID(即umount)会触发nfs4_rm_clid,将clientid从backend删掉
3. Server重启后，进入grace period，将backend信息读入内存，形成linklist，item内容是clid_entry。它记录着Server重启前active client list
4. Client试图恢复lock,然后发送RECLAIM_COMPLETE(参数rca_one_fs为FALSE)，全局变量reclaim_completes自增。
5. Server会定期检查reclaim_completes和clid_count是否一致，如果一致表示可以提前结束grace period

2. 相关函数:

• nfs_start_grace //这里设置全局变量nfs_in_grace,并调用nfs4_recovery_load_clids
• nfs_in_grace //判断是否在grace period
• nfs_try_lift_grace //试图提前结束grace period
• nfs_lift_grace_locked //结束grace period
• nfs4_recovery_reclaim_complete //给定一个clientid，遍历active client list，找到对应的，将其cl_reclaim_complete设为true
• nfs4_add_clid_entry //加入一个新的active client for recovery
• nfs4_cleanup_clid_entries //清空active client list

3. recovery backend相关函数

• nfs4_recovery_init //调用recovery_init接口,如fs_create_recov_dir，创建恢复的目录
• nfs_end_grace //调用end_grace接口，如fs_clean_old_recov_dir，把old相关目录删掉
• nfs4_recovery_load_clids //调用recovery_read_clids接口，如fs_read_recov_clids_takeover,从backend恢复active client list
• nfs4_add_clid //调用add_clid接口，如fs_add_clid，将clientid保存在backend中
• nfs4_rm_clid //调用rm_clid接口，如fs_rm_clid，将clientid从backend中删掉
• grace_mutex //保护上面这些变量的mutex

4. 全局变量

• reclaim_completes //统计有多少client 已经发送reclaim_complete
• nfs_in_grace //server启动时，记录着grace period的开始时间。grace period结束后，设置为0
• clid_count //Server重启前active clients数量
• clid_list //Server重启前active clients list

5. 数据结构

typedef struct clid_entry {
    struct glist_head cl_list; //用于插入到全局链表clid_list中
    struct glist_head cl_rfh_list; //delegation相关
    bool cl_reclaim_complete;  //已经send过reclaim_complete
    char cl_name[PATH_MAX];//clientid名字
} clid_entry_t;

6. 配置参数

• fsal_grace: 默认值是false,代表在grace period里的时候，是否处理lock

7. NFS4.1协议需要注意的地方

7.1 RECLAIM_COMPLETE

• When rca_one_fs is FALSE, a global RECLAIM_COMPLETE is being done. (一般情况是这个)
• When rca_one_fs is TRUE, a file system-specific RECLAIM_COMPLETE is being done.

7.2 open_claim_type4

enum open_claim_type4 {
    CLAIM_NULL = 0, //文件来自给定文件名
    CLAIM_PREVIOUS = 1, //to re-establish its locking state
    CLAIM_DELEGATE_CUR = 2,
    CLAIM_DELEGATE_PREV = 3,
    CLAIM_FH = 4, //文件来自current_obj
    CLAIM_DELEG_CUR_FH = 5,
    CLAIM_DELEG_PREV_FH = 6,
};

7.3 Reclaim of Open and Byte-Range Locks

• To reclaim existing opens, an OPEN operation is performed using a CLAIM_PREVIOUS
• To reclaim byte-range locks, a LOCK operation with the reclaim parameter set to true is used.