CoreML介绍 part1

1.Machine Learning 应用

image.png

2.CoreML 结构图

image.png
Vision: 高性能的图像分析和计算机视觉处理框架

Apply high-performance image analysis and computer vision techniques to identify faces, detect features, and classify scenes in images and video.
e.g.

• 物体追踪
• 面部识别

CroeML 优点

优势

CoreML 支持的模型类型

image.png

应用举例

image.png

3.CoreML 模型获取

可以从Apple machine-learning 页面获得,也可以从三方机构获取模型然后用苹果的 python 工具转换为coreml的模型.
https://developer.apple.com/machine-learning

convert 3rd to mlmodel

4.CoreML workflow

1. 拖入 CoreML model
2. 勾选 对应的 Target MemberShip, 自动生成需要的模型实体代码
3. 同时可以在 Xcode 预览 模型的输入参数/输出参数

image.png

** 创建模型,传入值 得到输出结果 **

Demo1
FlowerClassifiler.mlmodel v3 type

let flowerModel = FlowerClassifier()

func caption(image: CVPixelBuffer) throws -> String {
 let prediction = try self.flowerModel.prediction(flowerImage: image)
return prediciton.flowerType
}

Demo1.png

** 拖入训练模型的时候记得勾选target **

Demo2
使用 GoogLeNetPlaces.mlmodel<NNC> 用于图片识别;Detects the scene of an image from 205 categories such as airport, bedroom, forest, coast etc.

代码实现部分

- (NSString *)predictImageScene:(UIImage *)image {
    GoogLeNetPlaces *model = [[GoogLeNetPlaces alloc] init];
    NSError *error;
    UIImage *scaledImage = [image scaleToSize:CGSizeMake(224, 224)];
    CVPixelBufferRef buffer = [image pixelBufferFromCGImage:scaledImage];
    GoogLeNetPlacesInput *input = [[GoogLeNetPlacesInput alloc] initWithSceneImage:buffer];
    GoogLeNetPlacesOutput *output = [model predictionFromFeatures:input error:&error];
    return output.sceneLabel;
}

相关 category
1.实现UIImage 和 CVPixelBuffer(Ref)的转换函数[Objective-C]

• (CVPixelBufferRef)pixelBufferFromCGImage:(UIImage *)originImage

- (CVPixelBufferRef)pixelBufferFromCGImage:(UIImage *)originImage {
    CGImageRef image = originImage.CGImage;
    NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
                             nil];

    CVPixelBufferRef pxbuffer = NULL;

    CGFloat frameWidth = CGImageGetWidth(image);
    CGFloat frameHeight = CGImageGetHeight(image);

    CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault,
                                          frameWidth,
                                          frameHeight,
                                          kCVPixelFormatType_32ARGB,
                                          (__bridge CFDictionaryRef) options,
                                          &pxbuffer);

    NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);

    CVPixelBufferLockBaseAddress(pxbuffer, 0);
    void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
    NSParameterAssert(pxdata != NULL);

    CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();

    CGContextRef context = CGBitmapContextCreate(pxdata,
                                                 frameWidth,
                                                 frameHeight,
                                                 8,
                                                 CVPixelBufferGetBytesPerRow(pxbuffer),
                                                 rgbColorSpace,
                                                 (CGBitmapInfo)kCGImageAlphaNoneSkipFirst);
    NSParameterAssert(context);
    CGContextConcatCTM(context, CGAffineTransformIdentity);
    CGContextDrawImage(context, CGRectMake(0,
                                           0,
                                           frameWidth,
                                           frameHeight),
                       image);
    CGColorSpaceRelease(rgbColorSpace);
    CGContextRelease(context);

    CVPixelBufferUnlockBaseAddress(pxbuffer, 0);

    return pxbuffer;
}

- (UIImage *)scaleToSize:(CGSize)size {
    UIGraphicsBeginImageContext(size);
    [self drawInRect:CGRectMake(0, 0, size.width, size.height)];
    UIImage* scaledImage = UIGraphicsGetImageFromCurrentImageContext();
    UIGraphicsEndImageContext();
    return scaledImage;
}

swift 辅助代码

fileprivate func CreatePixelBufferFromImage(_ image: UIImage) -> CVPixelBuffer?{
        let size = image.size
        var pxbuffer : CVPixelBuffer?
        let pixelBufferPool = createPixelBufferPool(224, 224, FourCharCode(kCVPixelFormatType_32BGRA), 2056) // Hard coded values for demo purposes.
        let status = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pixelBufferPool!, &pxbuffer)
        
        guard (status == kCVReturnSuccess) else{
            return nil
        }
        
        CVPixelBufferLockBaseAddress(pxbuffer!, CVPixelBufferLockFlags(rawValue: 0))
        let pxdata = CVPixelBufferGetBaseAddress(pxbuffer!)
        let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
        let context = CGContext(data: pxdata,
                                width: Int(size.width),
                                height: Int(size.height),
                                bitsPerComponent: 8,
                                bytesPerRow: CVPixelBufferGetBytesPerRow(pxbuffer!),
                                space: rgbColorSpace,
                                bitmapInfo: CGImageAlphaInfo.premultipliedFirst.rawValue)
        
        context?.translateBy(x: 0, y: image.size.height)
        context?.scaleBy(x: 1.0, y: -1.0)
        UIGraphicsPushContext(context!)
        image.draw(in: CGRect(x: 0, y: 0, width: size.width, height: size.height))
        UIGraphicsPopContext()
        CVPixelBufferUnlockBaseAddress(pxbuffer!, CVPixelBufferLockFlags(rawValue: 0))
        return pxbuffer
    }
    ```

fileprivate func createPixelBufferPool(_ width: Int32, _ height: Int32, _ pixelFormat: FourCharCode, _ maxBufferCount: Int32) -> CVPixelBufferPool? {
    var outputPool: CVPixelBufferPool? = nil
    let sourcePixelBufferOptions: NSDictionary = [kCVPixelBufferPixelFormatTypeKey: pixelFormat,
                                                  kCVPixelBufferWidthKey: width,
                                                  kCVPixelBufferHeightKey: height,
                                                  kCVPixelFormatOpenGLESCompatibility: true,
                                                  kCVPixelBufferIOSurfacePropertiesKey: NSDictionary()]
    
    let pixelBufferPoolOptions: NSDictionary = [kCVPixelBufferPoolMinimumBufferCountKey: maxBufferCount]
    CVPixelBufferPoolCreate(kCFAllocatorDefault, pixelBufferPoolOptions, sourcePixelBufferOptions, &outputPool)
    return outputPool
}

** 简单总结: **
CoreML  就是屌. 利用Xcode 导入训练的模型(mlmodel),勾选 target membership, 根据模型描述传入输入值,得到输出结果. 你可以不懂机器学习,但是调用函数传递参数得到输出值展示 UI 你总会吧.

------
下一部分跟着 sessions  继续补充



1. NLP 的使用 Wednesday 9:00 AM


-------

2. Vision 的使用 Wednesday 3:10 PM

-----

![静态图片分析](https://img.haomeiwen.com/i519579/529caa47e41acbcf.png)

连续追踪

![image.png](https://img.haomeiwen.com/i519579/d510d15fe0dde379.png)


支持图片类型

![image.png](https://img.haomeiwen.com/i519579/a900baa5dda65e71.png)

不需要你缩小 矫正方向

流: CVPixelBuffer CMSampleBuffer 获取文件是 nsurl


![image.png](https://img.haomeiwen.com/i519579/091ee7e0bd900b97.png)

![image.png](https://img.haomeiwen.com/i519579/cb8e5495a22a2e84.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

![image.png](https://img.haomeiwen.com/i519579/fd1e3d33f11265c8.png)


![image.png](https://img.haomeiwen.com/i519579/b122d086a3b088d5.png)




![image.png](https://img.haomeiwen.com/i519579/74e449566b506957.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)



![image.png](https://img.haomeiwen.com/i519579/de25a7d170999560.png)

![image.png](https://img.haomeiwen.com/i519579/54e4c0b169346280.png)


![image.png](https://img.haomeiwen.com/i519579/34e431951f423e2b.png)


![image.png](https://img.haomeiwen.com/i519579/4727f9b6e05d1216.png)

![image.png](https://img.haomeiwen.com/i519579/a3ec230704042def.png)



3. CoreML  深入使用    Thursday 9:00 AM
4. CoreML模型的转换