基于ReplayKit实现屏幕录制

作者: captain_Lu | 来源:发表于2021-06-28 20:31 被阅读0次

前言

近期项目中需要完成一个实现屏幕录制(包含画面、麦克风、app内声音)功能，并压缩上传服务器，因此对iOS系统的replaykit进行了初步的研究，现分享一下结果：

截屏2021-06-25 下午11.08.22.png

概述

基于目前项目的快速迭代要求，首先想到的是官方ReplayKit框架，初步调研发现ReplayKit框架最低要求是iOS9.0，且支持屏幕、麦克风、app声音的录制，满足技术可行性，因此决定直接采用ReplayKit实施。

ReplayKit介绍

ReplayKit在WWDC15的时候随iOS9.0推出。当时的目的是给游戏开发者录制玩游戏的视频，进行社交分享使用。除了录制和共享外，ReplayKit还包括一个功能齐全的用户界面，玩家可以用来编辑其视频剪辑。

Replaykit功能介绍视频 WWDC15

ReplayKit除了实现屏幕录制以外，还能够将录制的音视频流实时广播出去，对于iOS端，需要两个关键技术：屏幕内容采集和媒体流广播。前者需要系统提供相关权限，可以让开发者采集到app或者整个系统层面的屏幕上的内容，后者需要系统提供采集到实时的视频流和音频流，这样才能通过推流到服务器，实现媒体流的广播。

录制

iOS9.0

//头文件
#import <ReplayKit/ReplayKit.h>

//启动录制
- (void)startRecordingWithMicrophoneEnabled:(BOOL)microphoneEnabled handler:(nullable void (^)(NSError *_Nullable error))handler API_DEPRECATED("Use microphoneEnabled property", ios(9.0, 10.0)) API_UNAVAILABLE(macOS);

//停止录制
- (void)stopRecordingWithHandler:(nullable void (^)(RPPreviewViewController *_Nullable previewViewController, NSError *_Nullable error))handler;

通过stopRecordingWithHandler的api，回调previewViewController(预览页面),通过presentViewController推出预览页，可以：裁剪、分享、保存相册

[self presentViewController:previewViewController animated:YES completion:^{}];

预览页监听操作结果

#pragma mrak - RPPreviewViewControllerDelegate

- (void)previewController:(RPPreviewViewController *)previewController didFinishWithActivityTypes:(NSSet <NSString *> *)activityTypes
{
    if ([activityTypes containsObject:@"com.apple.UIKit.activity.SaveToCameraRoll"]) {
        dispatch_async(dispatch_get_main_queue(), ^{
            NSLog(@"保存成功");
        });
    }
    if ([activityTypes containsObject:@"com.apple.UIKit.activity.CopyToPasteboard"]) {
        dispatch_async(dispatch_get_main_queue(), ^{
            NSLog(@"复制成功");

        });
    }
}

- (void)previewControllerDidFinish:(RPPreviewViewController *)previewController
{
    [previewController dismissViewControllerAnimated:YES completion:^{
        
    }];
}

通过拦截RPPreViewController，打印录制视频的地址：videoUrl = file:///private/var/mobile/Library/ReplayKit/ReplaykitDemo_06-28-2021%2015-51-13_1.mp4 可以发现文件存在于系统的位置，所以无法直接获取

总结：

优点：
高度封装，操作简单，能够快速的实现屏幕录制功能。

缺点：

不能获取到视频录制时的数据，只能在停止录制视频的时候获取到苹果已经处理合成好的MP4文件
不能直接获取录制好的视频文件，需要先通过用户存储到相册，你才能通过相册去访问到该文件、
停止录制的时候需要弹出一个视频的预览窗口，你可以在这个窗口进行保存或者取消或者分享该视频文件、你还可以直接编辑该视频
由于上面的限制，你只能在用户存储录制的视频保存到相册你才能访问。想要上传该视频到服务器，你还需要把相册的那个视频先想办法copy到沙盒中，然后再开始上传服务器。
无法配置屏幕录制参数

iOS10.0

优化内容：

//新增启动录制
- (void)startRecordingWithHandler:(nullable void (^)(NSError *_Nullable error))handler API_AVAILABLE(ios(10.0), tvos(10.0), macos(11.0));

//通过microphoneEnabled 控制是否开启麦克风
@property (nonatomic, getter = isMicrophoneEnabled) BOOL microphoneEnabled API_UNAVAILABLE(tvOS);

//结束录制以及录制完成后跳转预览页做编辑操作同iOS9.0保持一致

总结：同iOS9.0

新增内容

iOS 10 系统在 iOS 9 系统的 ReplayKit保存录屏视频的基础上，增加了视频流实时直播功能（streaming live）,可以将广播出来的直播流进行分发和直播。具体实现是通过增加ReplayKit的扩展分别为Broadcast Upload Extension 和 Broadcast Setup UI Extension，
Broadcast Upload Extension 是处理捕捉到App屏幕录制的数据的
Broadcast Setup UI Extension一些关于屏幕捕捉的UI交互

步骤：

添加扩展插件file->new->target->Broadcast upload Extension
系统会生成两个target,两个对应的目录以及4个文件分别：

SampleHandler.h
SampleHandler.m
BroadcastSetupViewController.h
BroadcastSetupViewController.m

SampleHandler主要处理流数据RPSampleBufferTypeVideo、RPSampleBufferTypeAudioApp、RPSampleBufferTypeAudioMic，BroadcastSetupViewController作为启动进程间插入的交互页面，可以用于用户输入信息鉴权，或者自定义其他界面

启动备选界面

//启动备选界面
+ (void)loadBroadcastActivityViewControllerWithHandler:(void (^)(RPBroadcastActivityViewController *_Nullable broadcastActivityViewController, NSError *_Nullable error))handler;

[RPBroadcastActivityViewController loadBroadcastActivityViewControllerWithHandler:^(RPBroadcastActivityViewController * _Nullable broadcastActivityViewController, NSError * _Nullable error) {
    if (error) {
        NSLog(@"RPBroadcast err %@", [error localizedDescription]);
    }
    broadcastActivityViewController.delegate = self;
    [self presentViewController:broadcastActivityViewController animated:YES completion:nil];
}];

通过代理回调，启动录制进程

#pragma mark - Broadcasting

- (void)broadcastActivityViewController:(RPBroadcastActivityViewController *) broadcastActivityViewController
       didFinishWithBroadcastController:(RPBroadcastController *)broadcastController
                                  error:(NSError *)error {
    
    [broadcastActivityViewController dismissViewControllerAnimated:YES completion:nil];
                                                        
    self.broadcastController = broadcastController;
    self.broadcastController.delegate = self;
    if (error) {
        return;
    }

    //启动广播
    [broadcastController startBroadcastWithHandler:^(NSError * _Nullable error) {
        if (!error) {
            NSLog(@"-----start success----");
            // 这里可以添加camerPreview
        } else {
            NSLog(@"startBroadcast:%@",error.localizedDescription);
        }
    }];
}

UI交互配置

- (void)userDidFinishSetup {
    NSURL *broadcastURL = [NSURL URLWithString:@"http://apple.com/broadcast/streamID"];
    NSDictionary *setupInfo = @{ @"broadcastName" : @"example" };
    // Tell ReplayKit that the extension is finished setting up and can begin broadcasting
    [self.extensionContext completeRequestWithBroadcastURL:broadcastURL setupInfo:setupInfo];
}

- (void)userDidCancelSetup {
    [self.extensionContext cancelRequestWithError:[NSError errorWithDomain:@"YourAppDomain" code:-1 userInfo:nil]];
}

- (void)viewWillAppear:(BOOL)animated
{
    [self userDidFinishSetup];
}

数据流的接收与处理

- (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo {
    // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional. 
}
- (void)broadcastPaused {
    // User has requested to pause the broadcast. Samples will stop being delivered.
}
- (void)broadcastResumed {
    // User has requested to resume the broadcast. Samples delivery will resume.
}
- (void)broadcastFinished {
    // User has requested to finish the broadcast.
}
- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
    
    switch (sampleBufferType) {
        case RPSampleBufferTypeVideo:
            // Handle video sample buffer
            break;
        case RPSampleBufferTypeAudioApp:
            // Handle audio sample buffer for app audio
            break;
        case RPSampleBufferTypeAudioMic:
            // Handle audio sample buffer for mic audio
            break;
        default:
            break;
    }
}

processSampleBuffer方法就是最终采集到的音频、视频原始数据。其中音频未做混音，包括麦克音频pcm和app音频pcm，而视频输出为yuv数据。

总结：

优点：

除了录屏以外，新增直播特性，功能更加强大
能够拿到音视频原始流数据，满足一些需要做音视频特效的需求

缺点：

增加用户交互成本，需要拉起录制列表，然后用户点击选择对应的录制程序，操作成功相对高一些
集成难度相比于iOS9.0加大，处理原始数据难度比较大

iOS11.0

新增内容

新增api,跳过iOS10的中间列表sheet在点击选择的过程，但是还是只能录制app内的内容。

+ (void)loadBroadcastActivityViewControllerWithPreferredExtension:(NSString * _Nullable)preferredExtension handler:(nonnull void(^)(RPBroadcastActivityViewController * _Nullable broadcastActivityViewController, NSError * _Nullable error))handler API_AVAILABLE(ios(11.0)) API_UNAVAILABLE(tvOS);

处理的流程同iOS10的扩展插件

新增开启屏幕捕捉

开启捕捉回调sampleBuffer
- (void)startCaptureWithHandler:(nullable void (^)(CMSampleBufferRef sampleBuffer, RPSampleBufferType bufferType, NSError *_Nullable error))captureHandler completionHandler:(nullable void (^)(NSError *_Nullable error))completionHandler API_AVAILABLE(ios(11.0), tvos(11.0), macos(11.0));

可以直接调用接口捕捉到sampleBuffer,省去了iOS10的扩展插件环节,可以直接拿到想要的buffer裸数据，无需中间交互环节，完成满足最上面所说的项目要求

总结：

优点：

调用方法简单，易于集成
无中间用户交互环节，用户交互成本低
直接获取到音视频裸数据

缺点：
裸数据处理难度稍大

补充

音视频裸数据编码合成mp4写入本地沙盒

iOS端编码合成采用AVAssetWriter,配套AVAssetWriterInput使用

//writer
@property (nonatomic, strong) AVAssetWriter *assetWriter;
//视频输入
@property (nonatomic, strong) AVAssetWriterInput *assetWriterVideoInput;
//音频输入
@property (nonatomic, strong) AVAssetWriterInput *assetWriterAudioInput;
//app内音频输入
@property (nonatomic, strong) AVAssetWriterInput *assetWriterAppAudioInput;

//初始化
self.assetWriter = [AVAssetWriter assetWriterWithURL:[NSURL fileURLWithPath:videoOutPath] fileType:AVFileTypeMPEG4 error:&error];

2.视频编码配置

    //视频的配置
    NSDictionary *compressionProperties = @{
        AVVideoProfileLevelKey : AVVideoProfileLevelH264HighAutoLevel,
        AVVideoH264EntropyModeKey      : AVVideoH264EntropyModeCABAC,
        AVVideoAverageBitRateKey       : @(DEVICE_WIDTH * DEVICE_HEIGHT * 6.0),
        AVVideoMaxKeyFrameIntervalKey  : @15,
        AVVideoExpectedSourceFrameRateKey : @(15),
        AVVideoAllowFrameReorderingKey : @NO};
        
    NSNumber* width= [NSNumber numberWithFloat:DEVICE_WIDTH];
    NSNumber* height = [NSNumber numberWithFloat:DEVICE_HEIGHT];

    NSDictionary *videoSettings = @{
            AVVideoCompressionPropertiesKey :compressionProperties,
            AVVideoCodecKey :AVVideoCodecTypeH264,
            AVVideoWidthKey : width,
            AVVideoHeightKey: height
    };

    self.assetWriterVideoInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:videoSettings];

3.音频编码配置

    // 音频设置
    NSDictionary * audioCompressionSettings = @{                       AVEncoderBitRatePerChannelKey : @(28000),
        AVFormatIDKey : @(kAudioFormatMPEG4AAC),
        AVNumberOfChannelsKey : @(1),
        AVSampleRateKey : @(22050) };

    self.assetWriterAudioInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeAudio outputSettings:audioCompressionSettings];

4.input添加writer

    //视频
    [self.assetWriter addInput:self.assetWriterVideoInput];
    [self.assetWriterVideoInput setMediaTimeScale:60];
    [self.assetWriterVideoInput setExpectsMediaDataInRealTime:YES];
        
    [self.assetWriter setMovieTimeScale:60];
        
    //音频
    [self.assetWriter addInput:self.assetWriterAudioInput];
    self.assetWriterAudioInput.expectsMediaDataInRealTime = YES;
    
    //app内声音
    [self.assetWriter addInput:self.assetWriterAppAudioInput];
    self.assetWriterAppAudioInput.expectsMediaDataInRealTime = YES;

5.合并代码

    [[RPScreenRecorder sharedRecorder] startCaptureWithHandler:^(CMSampleBufferRef  _Nonnull sampleBuffer, RPSampleBufferType bufferType, NSError * _Nullable error) {
            
        if (CMSampleBufferDataIsReady(sampleBuffer)) {
                
            if (self.assetWriter.status == AVAssetWriterStatusUnknown && bufferType == RPSampleBufferTypeVideo) {
                    [self.assetWriter startWriting];
                    [self.assetWriter startSessionAtSourceTime:CMSampleBufferGetPresentationTimeStamp(sampleBuffer)];
                }

            if (self.assetWriter.status == AVAssetWriterStatusFailed) {
                    NSLog(@"An error occured.");
                    [self writeDidOccureError:self.assetWriter.error callBack:handler];
                    return;
                }
            
                            if (bufferType == RPSampleBufferTypeVideo) {
                    if (self.assetWriterVideoInput.isReadyForMoreMediaData) {
                        [self.assetWriterVideoInput appendSampleBuffer:sampleBuffer];
                    }
                }else if (bufferType == RPSampleBufferTypeAudioMic)
                {
                    if (self.assetWriterAudioInput.isReadyForMoreMediaData) {
                        [self.assetWriterAudioInput appendSampleBuffer:sampleBuffer];
                        [self sampleBuffer2PcmData:sampleBuffer];
                    }
                }else if (bufferType == RPSampleBufferTypeAudioApp)
                {
                    if (self.assetWriterAppAudioInput.isReadyForMoreMediaData) {
                        [self.assetWriterAppAudioInput appendSampleBuffer:sampleBuffer];
                    }
                } 
            
        } completionHandler:^(NSError * _Nullable error) {
            if (!error) {
                // Start recording
                NSLog(@"Recording started successfully.");
                
            }else{
                //show alert
            }
        }];

音频解码获取声音大小

关键的代码

/// buffer转pcm
/// @param audiobuffer
- (void)sampleBuffer2PcmData:(CMSampleBufferRef)audiobuffer
{
    CMSampleBufferRef ref = audiobuffer;
    if(ref==NULL){
        return;
    }
    
    //copy data to file
    //read next one
    AudioBufferList audioBufferList;
    NSMutableData *data=[[NSMutableData alloc] init];
    CMBlockBufferRef blockBuffer;
    
    CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(ref, NULL, &audioBufferList, sizeof(audioBufferList), NULL, NULL, 0, &blockBuffer);
 
    for( int y=0; y<audioBufferList.mNumberBuffers; y++ )
    {
        AudioBuffer audioBuffer = audioBufferList.mBuffers[y];
        Float32 *frame = (Float32*)audioBuffer.mData;
        [data appendBytes:frame length:audioBuffer.mDataByteSize];
   
    }
    
    [self volumeFromPcmData:data] ;
    CFRelease(blockBuffer);
    blockBuffer=NULL;
}

/// 通过pcmdata获取声音分贝
/// @param pcmData pcm
-(void)volumeFromPcmData:(NSData *)pcmData
{
    if (pcmData == nil)
    {
        if ([self.delegate respondsToSelector:@selector(screenRecord:micVolume:)]) {
            [self.delegate screenRecord:self micVolume:0];
        }
        return;
    }
    
    long long pcmAllLenght = 0;
    short butterByte[pcmData.length/2];
    memcpy(butterByte, pcmData.bytes, pcmData.length);//frame_size * sizeof(short)
    
    // 将 buffer 内容取出，进行平方和运算
    for (int i = 0; i < pcmData.length/2; I++)
    {
        pcmAllLenght += butterByte[i] * butterByte[I];
    }
    // 平方和除以数据总长度，得到音量大小。
    double mean = pcmAllLenght / (double)pcmData.length;
    double volume =10*log10(mean);//volume为分贝数大小
    
    /*
     *0-20 很静 几乎感觉不到
     20-40 安静
     40-60一般室内谈话
     60-70吵闹
     70-90很吵、神经细胞受到破坏
     90-100吵闹家具 听力受损
     */
    if ([self.delegate respondsToSelector:@selector(screenRecord:micVolume:)]) {
        [self.delegate screenRecord:self micVolume:volume];
    }
}

总结

通过以上各个系统版本的对比，最终项目采用了iOS11的startCaptureWithHandler接口实现屏幕录制数据采集，然后通过AVAssetWriter进行编码合成mp4文件以及通过音频裸数据提取声音，最终完成该需求。以上均为代码的片段，还需要集合业务考虑各种异常情况的处理，以及视频、音频的编码配置需要进一步研究，通过优化配置参数，能够进一步提升录制视频的体验，整个过程坑点有点进一步补充。

基于ReplayKit实现屏幕录制

前言

概述

ReplayKit介绍

录制

iOS9.0

iOS10.0

iOS11.0

补充

总结

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

iOS技术专题