上篇文章已经简单了说了一下人脸识别的流程，那接下来的篇幅，我将详细的写出整个流程中用到的Library 和 具体的代码，让大家更直观的感受到整个人脸识别的检测过程

回顾一下人脸识别的过程

这里我将标注出用到的 Library

• 1.发现人脸 (ios-AVFoundation+MTCNN Android-MTCNN)
• 2.关键点检测 （MTCNN）
• 3.确定人脸姿态 （关键点计算，这里可以不用管，因为我用了9个方向的特征，比如说左转的姿态就用底库中左脸的特征点来比较，这里只做正脸检测）
• 4.特征抽取 （MobileFaceNet模型）
• 5.根据姿态对比相应的特征 （特征余弦相似度）

单张图片特征抽取的流程

image.png

1.发现人脸 && 关键点抽取

即 找到人脸的位置，这里iOS端多用了一个 用AVFoundation来检测人脸，为什么这么做呢？
因为MTCNN里面是3个网络，跑起来CPU很高，手机发热，耗电量也高，为了不一直去跑MTCNN，所以先用原生方法检测人脸，因为原生检测人脸效率更高，也不是特别发热，当检测到人脸后再丢给MTCNN检测，拿出关键点；
Android 因为系统版本差异太大，没有用原生，直接丢进MTCNN，检测出人脸框和关键点

• iOS代码 检测最大人脸，返回格式[关键点(array),人脸Rect(value)]

- (NSArray *)detectMaxFace:(UIImage *)image
{
    
    int w = image.size.width;
    int h = image.size.height;
    unsigned char* rgba = new unsigned char[w*h*4];
    {
        CGColorSpaceRef colorSpace = CGImageGetColorSpace(image.CGImage);
        CGContextRef contextRef = CGBitmapContextCreate(rgba, w, h, 8, w*4,
                                                        colorSpace,
                                                        kCGImageAlphaNoneSkipLast | kCGBitmapByteOrderDefault);
        
        CGContextDrawImage(contextRef, CGRectMake(0, 0, w, h), image.CGImage);
        CGContextRelease(contextRef);
    }

    ncnn::Mat ncnn_img;
    ncnn_img = ncnn::Mat::from_pixels(rgba, ncnn::Mat::PIXEL_RGBA2RGB, w, h);
    
    std::vector<Bbox> finalBbox;
    //    new_mtcnn->detect(ncnn_img, finalBbox);
    cv::Mat temp;
    UIImageToMat(image, temp);
    //    float cost;
    new_mtcnn->detectMaxFace(ncnn_img, finalBbox);
    int32_t num_face = static_cast<int32_t>(finalBbox.size());
    
    int out_size = 1+num_face*14;
    
    NSMutableArray *faceInfoArr = [NSMutableArray arrayWithCapacity:0];
    //
    int *faceInfo = new int[out_size];
    faceInfo[0] = num_face;
    for(int i=0;i<num_face;i++){
        NSMutableArray *points = [NSMutableArray arrayWithCapacity:0];
        
        CGRect rect = CGRectMake(finalBbox[i].x1, finalBbox[i].y1, finalBbox[i].x2 - finalBbox[i].x1, finalBbox[i].y2 - finalBbox[i].y1);
        
        for (int j =0;j<5;j++){
            CGPoint point = CGPointMake(finalBbox[i].ppoint[j], finalBbox[i].ppoint[j + 5]);
            [points addObject:[NSValue valueWithCGPoint:point]];
        }

        [faceInfoArr addObject:points];
        [faceInfoArr addObject:[NSValue valueWithCGRect:rect]];
    }
    
    delete [] rgba;
    delete [] faceInfo;
    finalBbox.clear();
    return faceInfoArr;
}

2.判断人脸姿态是否为正脸

因为我们要抽取特征的脸 一定要是正脸 所抽取的特征才有意义

image.png

上图为MTCNN抽取出来的关键点，同时在数组里的位置我也有标注出来,主要比的就是眼睛到鼻子的距离

- (BOOL)faceFront:(NSArray *)shape
{
    int mStartNumRightTemp = 0;
    int mStartNumLeftTemp = 0;
    
    CGPoint right_pupil;
    CGPoint left_pupil;
    left_pupil.x = [shape[0] CGPointValue].x ;  //左眼瞳孔坐标x
    left_pupil.y = [shape[0] CGPointValue].y ;  //左眼瞳孔坐标y
    
    right_pupil.x = [shape[1] CGPointValue].x  ;  //右眼瞳孔坐标x
    right_pupil.y = [shape[1] CGPointValue].y ;  //右眼瞳孔坐标y
    
    mStartNumRightTemp = abs(right_pupil.x - [shape[2] CGPointValue].x);  //到鼻子中间的距离
    mStartNumLeftTemp = abs([shape[2] CGPointValue].x  - left_pupil.x);
    
    float tempp1 = mStartNumRightTemp / (float)mStartNumLeftTemp;
    float tempp2 = mStartNumLeftTemp / (float)mStartNumRightTemp;

    if (tempp1 > 0.7 && tempp2>0.7)
        return YES;
    else
        return NO; 
}

3.人脸对齐 （openCV）

所谓人脸对齐，看下面的图就知道什么是人脸对齐了

image.png
左边如果是原图的话，对齐后的效果就是右边这样，旋转了一下图像
下面的方法包括了 旋转与抠出人脸图像，
输入整张图片，人脸的5个关键点，返回对齐后 并截取了脸部的图

- (Mat)getAlignMat:(Mat)bgrmat landmarks:(NSArray *)landmarks
{
    //left eye
    float left_eye_x = [landmarks[0] CGPointValue].x;
    float left_eye_y = [landmarks[0] CGPointValue].y;
    //right eye
    float right_eye_x = [landmarks[1] CGPointValue].x;
    float right_eye_y = [landmarks[1] CGPointValue].y;
    //nose
    float nose_x = [landmarks[2] CGPointValue].x;
    float nose_y = [landmarks[2] CGPointValue].y;
    //mouth left
    float mouth_left_x = [landmarks[3] CGPointValue].x;
    float mouth_left_y = [landmarks[3] CGPointValue].y;
    //mouth right
    float mouth_right_x = [landmarks[4] CGPointValue].x;
    float mouth_right_y = [landmarks[4] CGPointValue].y;
    //mouth center
    float mouth_center_x = (mouth_left_x + mouth_right_x) / 2;
    float mouth_center_y = (mouth_left_y + mouth_right_y) / 2;
    
    cv::Mat affineMat;
    std::vector<cv::Point2f> src_pts ;
    src_pts.push_back(cv::Point2f(left_eye_x, left_eye_y));
    src_pts.push_back(cv::Point2f(right_eye_x, right_eye_y));
    src_pts.push_back(cv::Point2f(mouth_center_x, mouth_center_y));
    
    
    cv::Point2f left_eye(38, 52);
    cv::Point2f right_eye(74, 52);
    cv::Point2f mouth_center(56, 92);
    cv::Size dsize(112, 112);
    
    std::vector<cv::Point2f> dst_pts;
    dst_pts.push_back(left_eye);
    dst_pts.push_back(right_eye);
    dst_pts.push_back(mouth_center);
    
    affineMat = cv::getAffineTransform(src_pts, dst_pts);
    
    cv::Mat alignedImg;
    cv::warpAffine(bgrmat, alignedImg, affineMat, dsize, cv::INTER_CUBIC, cv::BORDER_REPLICATE);
    
    return alignedImg;
    
}

4.抽取特征 （MobileFaceNet）

ncnn::Mat NCNNNet::getFaceFeatures(cv::Mat face)
    {
        ncnn::Mat in = ncnn::Mat::from_pixels(face.data, ncnn::Mat::PIXEL_BGR, face.cols, face.rows);
        const float mean_vals[3] = {127.5f, 127.5f, 127.5f};
        const float std_vals[3] = {0.0078125f, 0.0078125f, 0.0078125f};
        in.substract_mean_normalize(mean_vals, std_vals);
        
        ncnn::Extractor ex = features.create_extractor();
        ex.set_light_mode(true);
        ex.set_num_threads(4);

        ex.input(mobilefacenet_proto_id::BLOB_data, in);
        ncnn::Mat out;
        ex.extract(mobilefacenet_proto_id::BLOB_fc1_fc1_scale, out);

        
        return out;
    }

抽取出来的为128个float的数据，我们需要将ncnn:Mat 转换成你要的Array，然后再进行相似度计算,这里以iOS端为🌰

- (NSArray <NSNumber *>*)turnNCNNMat:(ncnn::Mat)feature
{
    NSMutableArray *features = [NSMutableArray arrayWithCapacity:0];
    for (int i = 0; i < feature.w; i++) {
        [features addObject:@((float)feature[i])];
    }
    return features;
}

同时为了提高运算精度，我们需要将抽取的特征数据归一化，这里得到的就是我们需要的人脸特征了

- (NSArray *)normalizeFeature:(NSArray <NSNumber *>*)feature
{
    NSMutableArray *fixFArr = [NSMutableArray arrayWithCapacity:0];
    if (!feature.count) return nil;
    float norm = 0;
    for (int i = 0; i < feature.count; i++) {
        norm += [feature[i] floatValue] * [feature[i] floatValue];
    }
    norm = sqrt(norm);
    if (norm == 0) return feature;
    for (int i = 0; i < feature.count; i++) {
        float newFeature = [feature[i] floatValue] / norm;
        [fixFArr addObject:@(newFeature)];
    }
    return fixFArr;
}

5.人脸相似度比对（余弦相似度）

通过以上步骤，我们已经可以抽取一张人脸的特征了，如果我们抽取两张，就可以得到两张人脸的信息，这时候，我们就可以进行对比的操作了

- (CGFloat)getFeatureDistanceWithFirstFeatures:(NSArray *)firstFeature second:(NSArray *)secondFeature
{
    if (!firstFeature.count || !secondFeature.count) return 0;
    float distance = 0;
    for (int i = 0; i < firstFeature.count && i < firstFeature.count; ++i) {
        distance += ([firstFeature[i] floatValue] - [secondFeature[i] floatValue]) * ([firstFeature[i] floatValue]- [secondFeature[i] floatValue]);
    }
    distance = sqrt(distance);
    CGFloat similarity = (1 - pow(distance / 2, 2)) * 100;
    return similarity;
}

PS：以上部分代码或者模型为网络上下载，如有侵权请及时通知我删除