美文网首页
MTK 单帧算法集成方法

MTK 单帧算法集成方法

作者: 程序员Android1 | 来源:发表于2021-06-25 09:21 被阅读0次

    和你一起终身学习,这里是程序员 Android

    经典好文推荐,通过阅读本文,您将收获以下知识点:

    算法概览
    一、算法集成前的准备
    二、 为算法选择feature
    三、 将算法对应的feature添加到scenario配置表
    四、挂载算法
    五、自定义metadata
    六、APP调用算法
    七、遇到的问题及解决方法
    八、结语

    算法概览

    为了给用户提供更好的成像效果,现在的手机都会接入一些第三方的图像处理算法。MTK平台的HAL3也在P2这一层提供接入的plugin。按图像处理算法需要的帧数和摄像头数量,大体可以分为三类:

    • 单帧算法:
      常见的单帧算法有:美颜算法(瘦脸、磨皮、大眼)、广角镜头畸变校正算法、附加表情算法、单摄背景虚化算法(伪双摄算法)等等,仅需单帧图像输入的算法都属于单帧算法。一般情况下,输入一帧图像,算法处理完输出一帧处理后的图像。

    • 多帧算法:
      常见的多帧算法有:MFNR(多帧降噪)、HDR(高动态范围)等等,需要连续多帧图像输入的算法都属于多帧算法。一般情况下,输入连续多帧图像,算法处理完输出一帧处理后的图像。

    • 双摄算法:
      最常见的双摄算法是双摄景深算法或者叫双摄背景虚化算法,除此之外,也有彩色+黑白用于增强夜拍效果的双摄算法。单帧算法和多帧算法仅需要获取一个摄像头的图像。而双摄算法需要获取主、辅两个摄像头的图像,并且一般还会要求主、辅摄像头同步。分别获取主、辅摄像头的两帧同步图像,处理后输出一帧主摄图像,用户也仅能看到主摄图像。

    根据这个大体上的分类,MTK HAL算法集成系列文章共三篇:

    本文是其中的第一篇。这个系列文章均基于Android 9.0,MT6763平台,HAL版本是HAL3。

    一、算法集成前的准备

    在开展集成工作之前,首先要对算法有一个基本的评估,并且对于集成也应有一定的要求。

    1. 1 算法要求及评估

    • 处理效果好,不能比竞品差,超过竞品更佳。(这条和camera调试的主观效果一样,主观性较强,往往一厢情愿,具体看项目要求吧)
    • 各个场景及压力测试下效果稳定。
    • 处理后照片无色差、锐度和饱和度无损失,或者损失在可接受范围。
    • 达到可接受的分辨率,最好可达到摄像头的最大分辨率。
    • 处理时间越快越好,不超过竞品时间、不超过项目和产品的目标时间。
    • 无内存泄露,占用内存少。
    • 提供必要的集成说明文档,包括算法类型、输入及输出图像要求、输入参数要求等等。

    注意:如果有条件,处理时间、内存占用、分辨率等等可量化的指标可要求算法提供方给出具体的参考数据,以便集成完后测试验证。

    1.2 算法集成要求

    • 编译时可根据项目控制是否集成算法。
    • 运行时可以用参数控制是否启用算法。
    • 集成算法库正常运行、压力测试下效果稳定、无内存泄露。

    1.3 算法集成的步骤

    (1). 根据算法选择feature类型,如果与MTK提供的feature不能对号入座,则需要添加自定义feature。
    (2). 将算法对应的feature类型添加到scenario配置表。
    (3). 根据算法选择plugin类型,编写CPP文件实现plugin,挂载算法。
    (4). 如果算法不能复用Android和MTK提供的metadata,则还需要为算法配置自定义的metadata以便APP控制是否启用算法。

    首先,我准备了一个libwatermark.so,它仅仅实现了一个添加水印的功能,用它来模拟第三方的单帧算法库。如果想了解添加水印的实现代码,可以参考我另外一篇文章:Android 实现图片加水印或logo。接下来,我们就按照集成步骤,逐步详细讲解。

    二、 为算法选择feature

    2.1 MTK提供的feature

    MTK在mtk_feature_type.h和customer_feature_type.h已经提供了一些feature。
    vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/mtk/mtk_feature_type.h:

        NO_FEATURE_NORMAL       = 0ULL,
        // MTK (bit 0-31)
        MTK_FEATURE_MFNR        = 1ULL << 0,
        MTK_FEATURE_HDR         = 1ULL << 1,
        MTK_FEATURE_REMOSAIC    = 1ULL << 2,
        MTK_FEATURE_ABF         = 1ULL << 3,
        MTK_FEATURE_NR          = 1ULL << 4,
        MTK_FEATURE_FB          = 1ULL << 5,
        MTK_FEATURE_CZ          = 1ULL << 6,
        MTK_FEATURE_DRE         = 1ULL << 7,
        MTK_FEATURE_DEPTH       = 1ULL << 8,
        MTK_FEATURE_BOKEH       = 1ULL << 9,
        MTK_FEATURE_VSDOF       = (MTK_FEATURE_DEPTH|MTK_FEATURE_BOKEH),
        MTK_FEATURE_FSC         = 1ULL << 10,
        MTK_FEATURE_3DNR        = 1ULL << 11,
        MTK_FEATURE_EIS         = 1ULL << 12,
        MTK_FEATURE_AINR        = 1ULL << 13,
        MTK_FEATURE_DUAL_YUV    = 1ULL << 14,
        MTK_FEATURE_DUAL_HWDEPTH  = 1ULL << 15,
        MTK_FEATURE_AIS         = 1ULL << 16,
        MTK_FEATURE_HFG         = 1ULL << 17,
        MTK_FEATURE_DCE         = 1ULL << 18,
    
    

    vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h:

        // ThirdParty (bit 32-63)
        TP_FEATURE_HDR          = 1ULL << 32,
        TP_FEATURE_MFNR         = 1ULL << 33,
        TP_FEATURE_EIS          = 1ULL << 34,
        TP_FEATURE_FB           = 1ULL << 35,
        TP_FEATURE_FILTER       = 1ULL << 36,
        TP_FEATURE_DEPTH        = 1ULL << 37,
        TP_FEATURE_BOKEH        = 1ULL << 38,
        TP_FEATURE_VSDOF        = (TP_FEATURE_DEPTH|TP_FEATURE_BOKEH),
        TP_FEATURE_FUSION       = 1ULL << 39,
        TP_FEATURE_HDR_DC       = 1ULL << 40,   // used by DualCam
        TP_FEATURE_DUAL_YUV     = 1ULL << 41,
        TP_FEATURE_DUAL_HWDEPTH = 1ULL << 42,
        TP_FEATURE_PUREBOKEH    = 1ULL << 43,
        TP_FEATURE_RAW_HDR      = 1ULL << 44,
        TP_FEATURE_RELIGHTING   = 1ULL << 45,
    
    

    MTK提供的这些feature可以满足绝大多数算法的集成,在可以对号入座的情况下,我们直接使用已有feature即可。如果不能够满足我们的要求,可以参考下节内容添加新的feature。

    2.2 添加自定义feature

    本来单帧算法对应的feature可以选择MTK提供的MTK_FEATURE_FB和TP_FEATURE_FB,但是为了讲解如何添加新feature,我们选择添加一个自定义feature:TP_FEATURE_WATERMARK。

    vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h b/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h
    old mode 100644
    new mode 100755
    index a41fd864f5..17bc35eea8
    --- a/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/customer/customer_feature_type.h
    @@ -59,6 +59,7 @@ enum eFeatureIndexCustomer {
         TP_FEATURE_PUREBOKEH    = 1ULL << 43,
         TP_FEATURE_RAW_HDR      = 1ULL << 44,
         TP_FEATURE_RELIGHTING   = 1ULL << 45,
    +    TP_FEATURE_WATERMARK    = 1ULL << 46,
         // TODO: reserve for customer feature index (bit 32-63)
     };
    
    

    vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/CaptureFeature_Common.cpp:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/CaptureFeature_Common.cpp b/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/CaptureFeature_Common.cpp
    old mode 100644
    new mode 100755
    index e32f80a609..47273b01c7
    --- a/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/CaptureFeature_Common.cpp
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/CaptureFeature_Common.cpp
    @@ -599,6 +599,7 @@ const char* FeatID2Name(FeatureID_T fid)
         case FID_FUSION_3RD_PARTY:      return "fusion_3rd_party";
         case FID_PUREBOKEH_3RD_PARTY:   return "purebokeh_3rd_party";
         case FID_RELIGHTING_3RD_PARTY:  return "relighting_3rd_party";
    +    case FID_WATERMARK_3RD_PARTY:   return "watermark_3rd_party";
    
         default:                        return "unknown";
         };
    
    

    vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp b/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp
    index 8bb794ba02..d4343aaccf 100755
    --- a/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp
    @@ -779,7 +779,8 @@ MBOOL YUVNode::onInit()
                 featId = FID_FB_3RD_PARTY;
             else if (rProperty.mFeatures & TP_FEATURE_RELIGHTING)
                 featId = FID_RELIGHTING_3RD_PARTY;
    -
    +        else if (rProperty.mFeatures & TP_FEATURE_WATERMARK)
    +            featId = FID_WATERMARK_3RD_PARTY;
    
             if (featId != NULL_FEATURE) {
                 MY_LOGD_IF(mLogLevel, "%s finds plugin:%s, priority:%d",
    
    

    vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/feature/featurePipe/ICaptureFeaturePipe.h:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/feature/featurePipe/ICaptureFeaturePipe.h b/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/feature/featurePipe/ICaptureFeaturePipe.h
    old mode 100644
    new mode 100755
    index 2f1ad8a665..ab47aae456
    --- a/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/feature/featurePipe/ICaptureFeaturePipe.h
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/feature/featurePipe/ICaptureFeaturePipe.h
    @@ -172,6 +172,7 @@ enum CaptureFeatureFeatureID {
         FID_FUSION_3RD_PARTY,
         FID_PUREBOKEH_3RD_PARTY,
         FID_RELIGHTING_3RD_PARTY,
    +    FID_WATERMARK_3RD_PARTY,
         NUM_OF_FEATURE,
         NULL_FEATURE = 0xFF,
     };
    
    

    vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/hwnode/p2/P2_CaptureProcessor.cpp:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/hwnode/p2/P2_CaptureProcessor.cpp b/vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/hwnode/p2/P2_CaptureProcessor.cpp
    old mode 100644
    new mode 100755
    index cc1dc549fd..00559cbc30
    --- a/vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/hwnode/p2/P2_CaptureProcessor.cpp
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/hwnode/p2/P2_CaptureProcessor.cpp
    @@ -428,6 +428,9 @@ MBOOL CaptureProcessor::onEnque(const sp<P2FrameRequest> &pP2Frame)
                     pCapRequest->addFeature(FID_HFG);
                 if (feature & MTK_FEATURE_DCE)
                     pCapRequest->addFeature(FID_DCE);
    +            if (feature & TP_FEATURE_WATERMARK)
    +                pCapRequest->addFeature(FID_WATERMARK_3RD_PARTY);
    +
             }
         }
    
    

    三、 将算法对应的feature添加到scenario配置表

    在我们打开camera进行预览和拍照的时候,MTK HAL3会执行vendor/mediatek/proprietary/hardware/mtkcam3/pipeline/policy/FeatureSettingPolicy.cpp的代码,会分别调用
    vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/scenario_mgr.cpp的
    get_streaming_scenario函数和get_capture_scenario函数。它们会读取一个scenario的feature配置表,遍历所有的feature,决定哪些feature会被执行。这个配置表中有许多的scenario,一个scenario可能对应多个feature。因此添加自定义feature后,还需将自定义的feature添加到配置表中。MTK feature 对应的配置表是 gMtkScenarioFeaturesMaps,customer feature 对应的配置表是 gCustomerScenarioFeaturesMaps。

    vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/customer_scenario_mgr.cpp:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/customer_scenario_mgr.cpp b/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/customer_scenario_mgr.cpp
    old mode 100644
    new mode 100755
    index f8d081e433..577f85797e
    --- a/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/customer_scenario_mgr.cpp
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/customer_scenario_mgr.cpp
    @@ -93,30 +93,30 @@ using namespace NSCam::v3::pipeline::policy::scenariomgr;
     // #define  <feature combination>              (key feature         | post-processing features | ...)
     //
     // single cam capture feature combination
    -#define TP_FEATURE_COMBINATION_SINGLE          (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB)
    -#define TP_FEATURE_COMBINATION_HDR             (TP_FEATURE_HDR      | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB)
    -#define TP_FEATURE_COMBINATION_AINR            (MTK_FEATURE_AINR    | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB)
    -#define TP_FEATURE_COMBINATION_MFNR            (MTK_FEATURE_MFNR    | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB)
    -#define TP_FEATURE_COMBINATION_REMOSAIC        (MTK_FEATURE_REMOSAIC| MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB)
    +#define TP_FEATURE_COMBINATION_SINGLE          (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
    +#define TP_FEATURE_COMBINATION_HDR             (TP_FEATURE_HDR      | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
    +#define TP_FEATURE_COMBINATION_AINR            (MTK_FEATURE_AINR    | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
    +#define TP_FEATURE_COMBINATION_MFNR            (MTK_FEATURE_MFNR    | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
    +#define TP_FEATURE_COMBINATION_REMOSAIC        (MTK_FEATURE_REMOSAIC| MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
     #define TP_FEATURE_COMBINATION_CSHOT           (NO_FEATURE_NORMAL   | MTK_FEATURE_CZ| MTK_FEATURE_HFG)
    -#define TP_FEATURE_COMBINATION_YUV_REPROCESS   (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| TP_FEATURE_FB)
    -#define TP_FEATURE_COMBINATION_RAW_REPROCESS   (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| TP_FEATURE_FB)
    +#define TP_FEATURE_COMBINATION_YUV_REPROCESS   (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
    +#define TP_FEATURE_COMBINATION_RAW_REPROCESS   (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
     #define TP_FEATURE_COMBINATION_PRO             (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE)
    -#define TP_FEATURE_COMBINATION_SUPER_NIGHT_RAW_REPROCESS (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| TP_FEATURE_FB)
    +#define TP_FEATURE_COMBINATION_SUPER_NIGHT_RAW_REPROCESS (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| TP_FEATURE_FB| TP_FEATURE_WATERMARK)
    
     // dual cam capture feature combination
     // the VSDOF means the combination of Bokeh feature and Depth feature
    -#define TP_FEATURE_COMBINATION_TP_VSDOF           (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_VSDOF)
    -#define TP_FEATURE_COMBINATION_TP_VSDOF_HDR       (TP_FEATURE_HDR_DC   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_VSDOF)
    -#define TP_FEATURE_COMBINATION_TP_VSDOF_MFNR      (MTK_FEATURE_MFNR    | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_VSDOF)
    -#define TP_FEATURE_COMBINATION_TP_FUSION          (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_FUSION)
    -#define TP_FEATURE_COMBINATION_TP_PUREBOKEH       (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_PUREBOKEH)
    +#define TP_FEATURE_COMBINATION_TP_VSDOF           (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_VSDOF| TP_FEATURE_WATERMARK)
    +#define TP_FEATURE_COMBINATION_TP_VSDOF_HDR       (TP_FEATURE_HDR_DC   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_VSDOF| TP_FEATURE_WATERMARK)
    +#define TP_FEATURE_COMBINATION_TP_VSDOF_MFNR      (MTK_FEATURE_MFNR    | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_VSDOF| TP_FEATURE_WATERMARK)
    +#define TP_FEATURE_COMBINATION_TP_FUSION          (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_FUSION| TP_FEATURE_WATERMARK)
    +#define TP_FEATURE_COMBINATION_TP_PUREBOKEH       (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE| TP_FEATURE_FB| TP_FEATURE_PUREBOKEH| TP_FEATURE_WATERMARK)
    
     // streaming feature combination (TODO: it should be refined by streaming scenario feature)
    -#define TP_FEATURE_COMBINATION_VIDEO_NORMAL       (MTK_FEATURE_FB|TP_FEATURE_FB)
    -#define TP_FEATURE_COMBINATION_VIDEO_DUAL_YUV     (MTK_FEATURE_FB|MTK_FEATURE_DUAL_YUV|TP_FEATURE_FB|TP_FEATURE_DUAL_YUV)
    -#define TP_FEATURE_COMBINATION_VIDEO_DUAL_HWDEPTH (MTK_FEATURE_FB|MTK_FEATURE_DUAL_HWDEPTH|TP_FEATURE_FB|TP_FEATURE_DUAL_HWDEPTH)
    -#define TP_FEATURE_COMBINATION_VIDEO_DUAL_HWVSDOF (MTK_FEATURE_FB|TP_FEATURE_FB)
    +#define TP_FEATURE_COMBINATION_VIDEO_NORMAL       (MTK_FEATURE_FB|TP_FEATURE_FB|TP_FEATURE_WATERMARK)
    +#define TP_FEATURE_COMBINATION_VIDEO_DUAL_YUV     (MTK_FEATURE_FB|MTK_FEATURE_DUAL_YUV|TP_FEATURE_FB|TP_FEATURE_DUAL_YUV|TP_FEATURE_WATERMARK)
    +#define TP_FEATURE_COMBINATION_VIDEO_DUAL_HWDEPTH (MTK_FEATURE_FB|MTK_FEATURE_DUAL_HWDEPTH|TP_FEATURE_FB|TP_FEATURE_DUAL_HWDEPTH|TP_FEATURE_WATERMARK)
    +#define TP_FEATURE_COMBINATION_VIDEO_DUAL_HWVSDOF (MTK_FEATURE_FB|TP_FEATURE_FB|TP_FEATURE_WATERMARK)
     // ======================================================================================================
     //
     /******************************************************************************
    
    

    vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/mtk/mtk_scenario_mgr.cpp:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/mtk/mtk_scenario_mgr.cpp b/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/mtk/mtk_scenario_mgr.cpp
    old mode 100644
    new mode 100755
    index 011f551354..f14ff8a6e2
    --- a/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/mtk/mtk_scenario_mgr.cpp
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/mtk/mtk_scenario_mgr.cpp
    @@ -89,29 +89,29 @@ using namespace NSCam::v3::pipeline::policy::scenariomgr;
     // #define  <feature combination>              (key feature         | post-processing features | ...)
     //
     // single cam capture feature combination
    -#define MTK_FEATURE_COMBINATION_SINGLE         (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB)
    -#define MTK_FEATURE_COMBINATION_HDR            (TP_FEATURE_HDR      | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB)
    -#define MTK_FEATURE_COMBINATION_AINR           (MTK_FEATURE_AINR    | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB)
    -#define MTK_FEATURE_COMBINATION_MFNR           (MTK_FEATURE_MFNR    | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB)
    -#define MTK_FEATURE_COMBINATION_REMOSAIC       (MTK_FEATURE_REMOSAIC| MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB)
    +#define MTK_FEATURE_COMBINATION_SINGLE         (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_HDR            (TP_FEATURE_HDR      | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_AINR           (MTK_FEATURE_AINR    | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_MFNR           (MTK_FEATURE_MFNR    | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_REMOSAIC       (MTK_FEATURE_REMOSAIC| MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_WATERMARK)
     #define MTK_FEATURE_COMBINATION_CSHOT          (NO_FEATURE_NORMAL   | MTK_FEATURE_CZ| MTK_FEATURE_HFG)
    -#define MTK_FEATURE_COMBINATION_YUV_REPROCESS  (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_FB)
    -#define MTK_FEATURE_COMBINATION_RAW_REPROCESS  (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_FB)
    -#define MTK_FEATURE_COMBINATION_SUPER_NIGHT_RAW_REPROCESS  (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_FB)
    +#define MTK_FEATURE_COMBINATION_YUV_REPROCESS  (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_FB| TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_RAW_REPROCESS  (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_FB| TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_SUPER_NIGHT_RAW_REPROCESS  (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_FB| TP_FEATURE_WATERMARK)
    
     // dual cam capture feature combination
     // the VSDOF means the combination of Bokeh feature and Depth feature
    -#define MTK_FEATURE_COMBINATION_TP_VSDOF          (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_VSDOF)
    -#define MTK_FEATURE_COMBINATION_TP_VSDOF_HDR      (TP_FEATURE_HDR_DC   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_VSDOF)
    -#define MTK_FEATURE_COMBINATION_TP_VSDOF_MFNR     (MTK_FEATURE_MFNR    | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_VSDOF)
    -#define MTK_FEATURE_COMBINATION_TP_FUSION         (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_FUSION)
    -#define MTK_FEATURE_COMBINATION_TP_PUREBOKEH      (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_PUREBOKEH)
    +#define MTK_FEATURE_COMBINATION_TP_VSDOF          (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_VSDOF| TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_TP_VSDOF_HDR      (TP_FEATURE_HDR_DC   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_VSDOF| TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_TP_VSDOF_MFNR     (MTK_FEATURE_MFNR    | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_VSDOF| TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_TP_FUSION         (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_FUSION| TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_TP_PUREBOKEH      (NO_FEATURE_NORMAL   | MTK_FEATURE_NR| MTK_FEATURE_ABF| MTK_FEATURE_CZ| MTK_FEATURE_DRE| MTK_FEATURE_HFG| MTK_FEATURE_DCE | MTK_FEATURE_FB| TP_FEATURE_PUREBOKEH| TP_FEATURE_WATERMARK)
    
     // streaming feature combination (TODO: it should be refined by streaming scenario feature)
    -#define MTK_FEATURE_COMBINATION_VIDEO_NORMAL     (MTK_FEATURE_FB|TP_FEATURE_FB)
    -#define MTK_FEATURE_COMBINATION_VIDEO_DUAL_YUV   (MTK_FEATURE_FB|MTK_FEATURE_DUAL_YUV|TP_FEATURE_FB|TP_FEATURE_DUAL_YUV)
    -#define MTK_FEATURE_COMBINATION_VIDEO_DUAL_HWDEPTH (MTK_FEATURE_FB|MTK_FEATURE_DUAL_HWDEPTH|TP_FEATURE_FB|TP_FEATURE_DUAL_HWDEPTH)
    -#define MTK_FEATURE_COMBINATION_VIDEO_DUAL_HWVSDOF (MTK_FEATURE_FB|TP_FEATURE_FB)
    +#define MTK_FEATURE_COMBINATION_VIDEO_NORMAL     (MTK_FEATURE_FB|TP_FEATURE_FB|TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_VIDEO_DUAL_YUV   (MTK_FEATURE_FB|MTK_FEATURE_DUAL_YUV|TP_FEATURE_FB|TP_FEATURE_DUAL_YUV|TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_VIDEO_DUAL_HWDEPTH (MTK_FEATURE_FB|MTK_FEATURE_DUAL_HWDEPTH|TP_FEATURE_FB|TP_FEATURE_DUAL_HWDEPTH|TP_FEATURE_WATERMARK)
    +#define MTK_FEATURE_COMBINATION_VIDEO_DUAL_HWVSDOF (MTK_FEATURE_FB|TP_FEATURE_FB|TP_FEATURE_WATERMARK)
     // ======================================================================================================
     //
     /******************************************************************************
    
    

    注意:
    MTK在Android Q(10.0)上优化了scenario配置表的客制化,Android Q及更高版本,scenario需要在:
    vendor/mediatek/proprietary/custom/[platform]/hal/camera/camera_custom_feature_table.cpp中配置,[platform]是诸如mt6580,mt6763之类的。

    将自定义feature添加到scenario配置表时,不可贪多,只要添加到合适的scenario就行,多了可能多个算法会有冲突。如果仅在简单场景,添加到MTK_FEATURE_COMBINATION_SINGLE和TP_FEATURE_COMBINATION_SINGLE就可以满足绝大多数需求。(2021-02-02更新)

    四、挂载算法

    4.1 为算法选择plugin

    MTK HAL3在vendor/mediatek/proprietary/hardware/mtkcam3/include/mtkcam3/3rdparty/plugin/PipelinePluginType.h 中将三方算法的挂载点大致分为以下几类:

    • BokehPlugin: Bokeh算法挂载点,双摄景深算法的虚化部分。
    • DepthPlugin: Depth算法挂载点,双摄景深算法的计算深度部分。
    • FusionPlugin: Depth和Bokeh放在1个算法中,即合并的双摄景深算法挂载点。
    • JoinPlugin: Streaming相关算法挂载点,预览算法都挂载在这里。
    • MultiFramePlugin: 多帧算法挂载点,包括YUV与RAW,例如MFNR/HDR
    • RawPlugin: RAW算法挂载点,例如remosaic
    • YuvPlugin: Yuv单帧算法挂载点,例如美颜、广角镜头畸变校正等

    对号入座,将要集成的算法选择相应的plugin。这里是单帧算法,所以预览我们选择JoinPlugin,拍照选择YuvPlugin。

    4.2 编写算法集成文件

    参照FBImpl.cpp和sample_streaming_fb.cpp中分别实现拍照和预览。目录结构如下:
    vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/tp_watermark/
    ├── Android.mk
    ├── include
    │ └── watermark.h
    ├── lib
    │ ├── arm64-v8a
    │ │ └── libwatermark.so
    │ └── armeabi-v7a
    │ └── libwatermark.so
    ├── res
    │ └── watermark.rgba
    ├── WatermarkCapture.cpp
    └── WatermarkPreview.cpp

    文件说明:

    • Android.mk中配置算法库、头文件、集成的源代码CPP文件编译成库libmtkcam.plugin.tp_watermark,供libmtkcam_3rdparty.customer依赖调用。
    • 集成的源代码CPP文件,WatermarkCapture.cpp用于拍照,WatermarkPreview.cpp用于预览。
    • libwatermark.so实现了添加水印的功能,libwatermark.so用来模拟需要接入的第三方算法库。watermark.h是头文件。
    • watermark.rgba是对应的水印文件。
    4.2.1 添加全局宏控

    为了能控制某个项目是否集成此算法,我们在device/mediateksample/k63v2_64_bsp/ProjectConfig.mk中添加一个宏,用于控制新接入算法的编译:

    QXT_WATERMARK_SUPPORT = yes
    
    

    当某个项目不需要新接入的算法时,将device/mediateksample/[platform]/ProjectConfig.mk的QXT_WA_SUPPORT的值设为 no 就可以了。

    4.2.2 mtkcam3/3rdparty/customer/tp_watermark/Android.mk
    ifeq ($(QXT_WATERMARK_SUPPORT),yes)
    LOCAL_PATH := $(call my-dir)
    
    include $(CLEAR_VARS)
    LOCAL_MODULE := libwatermark
    LOCAL_SRC_FILES_32 := lib/armeabi-v7a/libwatermark.so
    LOCAL_SRC_FILES_64 := lib/arm64-v8a/libwatermark.so
    LOCAL_MODULE_TAGS := optional
    LOCAL_MODULE_CLASS := SHARED_LIBRARIES
    LOCAL_MODULE_SUFFIX := .so
    LOCAL_PROPRIETARY_MODULE := true
    LOCAL_MULTILIB := both
    include $(BUILD_PREBUILT)
    ################################################################################
    
    include $(CLEAR_VARS)
    
    #-----------------------------------------------------------
    include $(TOP)/$(MTK_PATH_SOURCE)/hardware/mtkcam/mtkcam.mk
    
    #-----------------------------------------------------------
    LOCAL_SRC_FILES += WatermarkCapture.cpp
    LOCAL_SRC_FILES += WatermarkPreview.cpp
    
    #-----------------------------------------------------------
    LOCAL_C_INCLUDES += $(MTKCAM_C_INCLUDES)
    LOCAL_C_INCLUDES += $(TOP)/$(MTK_PATH_SOURCE)/hardware/mtkcam3/include
    LOCAL_C_INCLUDES += $(TOP)/$(MTK_PATH_SOURCE)/hardware/mtkcam/include
    #
    LOCAL_C_INCLUDES += system/media/camera/include
    LOCAL_C_INCLUDES += $(TOP)/external/libyuv/files/include/
    LOCAL_C_INCLUDES += $(TOP)/$(MTK_PATH_SOURCE)/hardware/mtkcam3/3rdparty/customer/tp_watermark/include
    
    #-----------------------------------------------------------
    LOCAL_CFLAGS += $(MTKCAM_CFLAGS)
    #
    
    #-----------------------------------------------------------
    LOCAL_STATIC_LIBRARIES +=
    #
    LOCAL_WHOLE_STATIC_LIBRARIES +=
    
    #-----------------------------------------------------------
    LOCAL_SHARED_LIBRARIES += liblog
    LOCAL_SHARED_LIBRARIES += libutils
    LOCAL_SHARED_LIBRARIES += libcutils
    LOCAL_SHARED_LIBRARIES += libmtkcam_modulehelper
    LOCAL_SHARED_LIBRARIES += libmtkcam_stdutils
    LOCAL_SHARED_LIBRARIES += libmtkcam_pipeline
    LOCAL_SHARED_LIBRARIES += libmtkcam_metadata
    LOCAL_SHARED_LIBRARIES += libmtkcam_metastore
    LOCAL_SHARED_LIBRARIES += libmtkcam_streamutils
    LOCAL_SHARED_LIBRARIES += libmtkcam_imgbuf
    LOCAL_SHARED_LIBRARIES += libyuv.vendor
    #-----------------------------------------------------------
    LOCAL_HEADER_LIBRARIES := libutils_headers liblog_headers libhardware_headers
    
    #-----------------------------------------------------------
    LOCAL_MODULE := libmtkcam.plugin.tp_watermark
    LOCAL_PROPRIETARY_MODULE := true
    LOCAL_MODULE_OWNER := mtk
    LOCAL_MODULE_TAGS := optional
    include $(MTK_STATIC_LIBRARY)
    ################################################################################
    
    include $(call all-makefiles-under,$(LOCAL_PATH))
    endif
    
    
    4.2.3 mtkcam3/3rdparty/customer/tp_watermark/WatermarkCapture.cpp

    主要函数介绍:

    • 在property函数中feature类型设置我们在第三步中添加的TP_FEATURE_WATERMARK,并设置名称、优先级等等属性。
    • 在negotiate函数中配置算法需要的输入、输出图像的格式、尺寸。
    • 在negotiate函数或者process函数中获取上层传下来的metadata参数,根据参数决定算法是否运行,或者将参数传给算法。
    • 在process函数中接入算法。

    注意:

    MTK原文:
    negotiate函数设置格式时,一个挂载点如果挂载多个同类型的plugin,则只有第一个 plugin 中的 negotiate 中的 input buffer 设定有效。
    在YUVNode 下挂载单帧 YUV plugin时,一定要确保 MTK 平台的SWNR plugin 的 negotiate 直接返回不OK,不做任何 accepted format 等的设定。否则,可能会出现因 SWNR plugin和三方plugin negotiate时设定的 accepted format 不一致而导致的三方 plugin 拿不到它想要的 format 的buffer。

    vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/mtk/swnr/SWNRImpl.cpp:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/mtk/swnr/SWNRImpl.cpp b/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/mtk/swnr/SWNRImpl.cpp
    old mode 100644
    new mode 100755
    index 0ae951cc83..c4819068f7
    --- a/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/mtk/swnr/SWNRImpl.cpp
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/mtk/swnr/SWNRImpl.cpp
    @@ -340,7 +340,7 @@ negotiate(Selection& sel)
         sel.mOMetadataApp.setRequired(false);
         sel.mOMetadataHal.setRequired(true);
    
    -    return OK;
    +    return -EINVAL;//OK;
     }
    
    

    vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/tp_watermark/WatermarkCapture.cpp:

    #define LOG_TAG "WatermarkCapture"
    //
    #include <mtkcam/utils/std/Log.h>
    //
    #include <stdlib.h>
    #include <utils/Errors.h>
    #include <utils/List.h>
    #include <utils/RefBase.h>
    #include <sstream>
    //
    #include <mtkcam/utils/metadata/client/mtk_metadata_tag.h>
    #include <mtkcam/utils/metadata/hal/mtk_platform_metadata_tag.h>
    //
    //
    #include <mtkcam/utils/imgbuf/IIonImageBufferHeap.h>
    //
    #include <mtkcam/drv/IHalSensor.h>
    #include <mtkcam/utils/std/Format.h>
    //
    #include <mtkcam3/pipeline/hwnode/NodeId.h>
    
    #include <mtkcam/utils/metastore/ITemplateRequest.h>
    #include <mtkcam/utils/metastore/IMetadataProvider.h>
    #include <mtkcam3/3rdparty/plugin/PipelinePlugin.h>
    #include <mtkcam3/3rdparty/plugin/PipelinePluginType.h>
    
    #include <stdlib.h>
    #include <watermark.h>
    #include <mtkcam/utils/std/Time.h>
    #include <time.h>
    #include <libyuv.h>
    //
    using namespace NSCam;
    using namespace android;
    using namespace std;
    using namespace NSCam::NSPipelinePlugin;
    /******************************************************************************
     *
     ******************************************************************************/
    #define MY_LOGV(fmt, arg...)        CAM_LOGV("(%d)[%s] " fmt, ::gettid(), __FUNCTION__, ##arg)
    #define MY_LOGD(fmt, arg...)        CAM_LOGD("(%d)[%s] " fmt, ::gettid(), __FUNCTION__, ##arg)
    #define MY_LOGI(fmt, arg...)        CAM_LOGI("(%d)[%s] " fmt, ::gettid(), __FUNCTION__, ##arg)
    #define MY_LOGW(fmt, arg...)        CAM_LOGW("(%d)[%s] " fmt, ::gettid(), __FUNCTION__, ##arg)
    #define MY_LOGE(fmt, arg...)        CAM_LOGE("(%d)[%s] " fmt, ::gettid(), __FUNCTION__, ##arg)
    //
    #define FUNCTION_IN                 MY_LOGD("%s +", __FUNCTION__)
    #define FUNCTION_OUT                MY_LOGD("%s -", __FUNCTION__)
    //systrace
    #if 1
    #ifndef ATRACE_TAG
    #define ATRACE_TAG                           ATRACE_TAG_CAMERA
    #endif
    #include <utils/Trace.h>
    
    #define WATERMARK_TRACE_CALL()                      ATRACE_CALL()
    #define WATERMARK_TRACE_NAME(name)                  ATRACE_NAME(name)
    #define WATERMARK_TRACE_BEGIN(name)                 ATRACE_BEGIN(name)
    #define WATERMARK_TRACE_END()                       ATRACE_END()
    #else
    #define WATERMARK_TRACE_CALL()
    #define WATERMARK_TRACE_NAME(name)
    #define WATERMARK_TRACE_BEGIN(name)
    #define WATERMARK_TRACE_END()
    #endif
    
    template <class T>
    inline bool
    tryGetMetadata(IMetadata const *pMetadata, MUINT32 tag, T& rVal)
    {
        if(pMetadata == nullptr) return MFALSE;
    
        IMetadata::IEntry entry = pMetadata->entryFor(tag);
        if(!entry.isEmpty())
        {
            rVal = entry.itemAt(0,Type2Type<T>());
            return true;
        }
        else
        {
    #define var(v) #v
    #define type(t) #t
            MY_LOGW("no metadata %s in %s", var(tag), type(pMetadata));
    #undef type
    #undef var
        }
        return false;
    }
    
    /******************************************************************************
    *
    ******************************************************************************/
    class WatermarkCapture : public YuvPlugin::IProvider {
    
    public:
        typedef YuvPlugin::Property Property;
        typedef YuvPlugin::Selection Selection;
        typedef YuvPlugin::Request::Ptr RequestPtr;
        typedef YuvPlugin::RequestCallback::Ptr RequestCallbackPtr;
    
    private:
        int mOpenid;
    
        MBOOL mEnable = 1;
        MBOOL mDump = 0;
        unsigned char *mSrcRGBA = nullptr;
        unsigned char *mWatermarkRGBA = nullptr;
        int mWatermarkWidth = 0;
        int mWatermarkHeight = 0;
    
    public:
        WatermarkCapture();
    
        ~WatermarkCapture();
    
        void init();
    
        void uninit();
    
        void abort(vector <RequestPtr> &pRequests);
    
        void set(MINT32 iOpenId, MINT32 iOpenId2);
    
        const Property &property();
    
        MERROR negotiate(Selection &sel);
    
        MERROR process(RequestPtr pRequest, RequestCallbackPtr pCallback);
    
    };
    
    WatermarkCapture::WatermarkCapture() : mOpenid(-1) {
        FUNCTION_IN;
        mEnable = property_get_bool("vendor.debug.camera.watermark.capture.enable", 1);
        mDump = property_get_bool("vendor.debug.camera.watermark.capture.dump", 0);
        FUNCTION_OUT;
    }
    
    WatermarkCapture::~WatermarkCapture() {
        FUNCTION_IN;
        FUNCTION_OUT;
    }
    
    void WatermarkCapture::init() {
        FUNCTION_IN;
        mWatermarkWidth = 180;
        mWatermarkHeight = 640;
        int watermarkSize = mWatermarkWidth * mWatermarkHeight * 4;
        mWatermarkRGBA = (unsigned char *) malloc(watermarkSize);
    
        FILE *fp;
        char path[256];
        snprintf(path, sizeof(path), "/vendor/res/images/watermark.rgba");
        if ((fp = fopen(path, "r")) == NULL) {
            MY_LOGE("Failed to open /vendor/res/images/watermark.rgba");
        }
        fread(mWatermarkRGBA, 1, watermarkSize, fp);
        fclose(fp);
        FUNCTION_OUT;
    }
    
    void WatermarkCapture::uninit() {
        FUNCTION_IN;
        free(mWatermarkRGBA);
        FUNCTION_OUT;
    }
    
    void WatermarkCapture::abort(vector <RequestPtr> &pRequests) {
        FUNCTION_IN;
        (void)pRequests;
        FUNCTION_OUT;
    }
    
    void WatermarkCapture::set(MINT32 iOpenId, MINT32 iOpenId2) {
        FUNCTION_IN;
        MY_LOGD("set openId:%d openId2:%d", iOpenId, iOpenId2);
        mOpenid = iOpenId;
        FUNCTION_OUT;
    }
    
    const WatermarkCapture::Property &WatermarkCapture::property() {
        FUNCTION_IN;
        static Property prop;
        static bool inited;
    
        if (!inited) {
            prop.mName = "TP_WATERMARK";
            prop.mFeatures = TP_FEATURE_WATERMARK;
            prop.mInPlace = MTRUE;
            prop.mFaceData = eFD_Current;
            prop.mPosition = 0;
            inited = true;
        }
        FUNCTION_OUT;
        return prop;
    }
    
    MERROR WatermarkCapture::negotiate(Selection &sel) {
        FUNCTION_IN;
        if (!mEnable) {
            MY_LOGD("Force off TP_WATERMARK");
            FUNCTION_OUT;
            return -EINVAL;
        }
    
        sel.mIBufferFull
                .setRequired(MTRUE)
                .addAcceptedFormat(eImgFmt_I420)
                .addAcceptedSize(eImgSize_Full);
    
        sel.mIMetadataDynamic.setRequired(MTRUE);
        sel.mIMetadataApp.setRequired(MTRUE);
        sel.mIMetadataHal.setRequired(MTRUE);
        sel.mOMetadataApp.setRequired(MTRUE);
        sel.mOMetadataHal.setRequired(MTRUE);
    
        FUNCTION_OUT;
        return OK;
    }
    
    MERROR WatermarkCapture::process(RequestPtr pRequest,
                                   RequestCallbackPtr pCallback = nullptr) {
        FUNCTION_IN;
        WATERMARK_TRACE_CALL();
    
        MBOOL needRun = MFALSE;
        if (pRequest->mIBufferFull != nullptr && pRequest->mOBufferFull != nullptr) {
            IImageBuffer *pIBufferFull = pRequest->mIBufferFull->acquire();
            IImageBuffer *pOBufferFull = pRequest->mOBufferFull->acquire();
    
            if (pRequest->mIMetadataDynamic != nullptr) {
                IMetadata *meta = pRequest->mIMetadataDynamic->acquire();
                if (meta != NULL)
                    MY_LOGD("[IN] Dynamic metadata count: %d", meta->count());
                else
                    MY_LOGD("[IN] Dynamic metadata empty");
            }
    
            int frameNo = 0, requestNo = 0;
            if (pRequest->mIMetadataHal != nullptr) {
                IMetadata *pIMetataHAL = pRequest->mIMetadataHal->acquire();
                if (pIMetataHAL != NULL) {
                    MY_LOGD("[IN] HAL metadata count: %d", pIMetataHAL->count());
                    if (!tryGetMetadata<int>(pIMetataHAL, MTK_PIPELINE_FRAME_NUMBER, frameNo)) {
                        frameNo = 0;
                    }
                    if (!tryGetMetadata<int>(pIMetataHAL, MTK_PIPELINE_REQUEST_NUMBER, requestNo)) {
                        requestNo = 0;
                    }
                    MY_LOGD("frameNo: %d, requestNo: %d", frameNo, requestNo);
                } else {
                    MY_LOGD("[IN] HAL metadata empty");
                }
            }
    
            if (pRequest->mIMetadataApp != nullptr) {
                IMetadata *pIMetadataApp = pRequest->mIMetadataApp->acquire();
                MINT32 mode = 0;
                if (!tryGetMetadata<MINT32>(pIMetadataApp, QXT_FEATURE_WATERMARK, mode)) {
                    mode = 0;
                }
                needRun = mode == 1 ? 1 : 0;
            }
            MY_LOGD("needRun: %d", needRun);
    
            int width = pIBufferFull->getImgSize().w;
            int height = pIBufferFull->getImgSize().h;
            MINT inFormat = pIBufferFull->getImgFormat();
    
            if (needRun && inFormat == NSCam::eImgFmt_I420) {
                uint32_t currentTime = (NSCam::Utils::TimeTool::getReadableTime()) % 1000;
                time_t timep;
                time (&timep);
                char currentDate[20];
                strftime(currentDate, sizeof(currentDate), "%Y%m%d_%H%M%S", localtime(&timep));
    
                //dump input I420
                if (mDump) {
                    char path[256];
                    snprintf(path, sizeof(path), "/data/vendor/camera_dump/capture_in_frame%d_%dx%d_%s_%d.i420",
                        frameNo, width, height, currentDate, currentTime);
                    pIBufferFull->saveToFile(path);
                }
    
                nsecs_t t1 = systemTime(CLOCK_MONOTONIC);
                if (mSrcRGBA == NULL) {
                    mSrcRGBA = (unsigned char *) malloc(width * height * 4);
                }
                //convert I420 to RGBA
                libyuv::I420ToABGR((unsigned char *) (pIBufferFull->getBufVA(0)), width,
                                   (unsigned char *) (pIBufferFull->getBufVA(1)), width >> 1,
                                   (unsigned char *) (pIBufferFull->getBufVA(2)), width >> 1,
                                   mSrcRGBA, width * 4,
                                   width, height);
                nsecs_t t2 = systemTime(CLOCK_MONOTONIC);
                MY_LOGD("Prepare src cost %02ld ms", ns2ms(t2 - t1));
    
                Watermark::add(mSrcRGBA, width, height, mWatermarkRGBA, mWatermarkWidth, mWatermarkHeight, (width - mWatermarkWidth) / 2, (height - mWatermarkHeight) / 2);
                nsecs_t t3 = systemTime(CLOCK_MONOTONIC);
                MY_LOGD("Add watermark cost %02ld ms", ns2ms(t3 - t2));
    
                //convert RGBA to I420
                libyuv::ABGRToI420(mSrcRGBA, width * 4,
                                   (unsigned char *) (pOBufferFull->getBufVA(0)), width,
                                   (unsigned char *) (pOBufferFull->getBufVA(1)), width >> 1,
                                   (unsigned char *) (pOBufferFull->getBufVA(2)), width >> 1,
                                   width, height);
                nsecs_t t4 = systemTime(CLOCK_MONOTONIC);
                MY_LOGD("Copy in to out cost %02ld ms", ns2ms(t4 - t3));
    
                //dump output I420
                if (mDump) {
                    char path[256];
                    snprintf(path, sizeof(path), "/data/vendor/camera_dump/capture_out_frame%d_%dx%d_%s_%d.i420",
                        frameNo, width, height, currentDate, currentTime);
                    pOBufferFull->saveToFile(path);
                }
                free(mSrcRGBA);
            } else {
                if (!needRun) {
                    MY_LOGE("No need run, skip add watermark for capture.");
                } else if (inFormat != NSCam::eImgFmt_YV12) {
                    MY_LOGE("Unsupported format, skip add watermark for capture.");
                } else {
                    MY_LOGE("Unknown exception, skip add watermark for capture.");
                }
    
                memcpy((unsigned char *) (pOBufferFull->getBufVA(0)),
                       (unsigned char *) (pIBufferFull->getBufVA(0)),
                       pIBufferFull->getBufSizeInBytes(0));
                memcpy((unsigned char *) (pOBufferFull->getBufVA(1)),
                       (unsigned char *) (pIBufferFull->getBufVA(1)),
                       pIBufferFull->getBufSizeInBytes(1));
                memcpy((unsigned char *) (pOBufferFull->getBufVA(2)),
                       (unsigned char *) (pIBufferFull->getBufVA(2)),
                       pIBufferFull->getBufSizeInBytes(2));
            }
    
            pRequest->mIBufferFull->release();
            pRequest->mOBufferFull->release();
    
            if (pRequest->mIMetadataDynamic != nullptr) {
                pRequest->mIMetadataDynamic->release();
            }
    
            if (pRequest->mIMetadataHal != nullptr) {
                pRequest->mIMetadataHal->release();
            }
    
            if (pRequest->mIMetadataApp != nullptr) {
                pRequest->mIMetadataApp->release();
            }
        }
    
        if (pCallback != nullptr) {
            MY_LOGD("callback request");
            pCallback->onCompleted(pRequest, 0);
        }
        FUNCTION_OUT;
        return OK;
    }
    
    REGISTER_PLUGIN_PROVIDER(Yuv, WatermarkCapture);
    
    
    4.2.4 mtkcam3/3rdparty/customer/tp_watermark/WatermarkPreview.cpp
    #include <mtkcam3/3rdparty/plugin/PipelinePluginType.h>
    #include <mtkcam/utils/metadata/hal/mtk_platform_metadata_tag.h>
    #include <mtkcam/utils/metadata/client/mtk_metadata_tag.h>
    
    #include <cutils/properties.h>
    
    #include <watermark.h>
    #include <mtkcam/utils/std/Time.h>
    #include <time.h>
    #include <libyuv.h>
    #include <dlfcn.h>
    
    using NSCam::NSPipelinePlugin::Interceptor;
    using NSCam::NSPipelinePlugin::PipelinePlugin;
    using NSCam::NSPipelinePlugin::PluginRegister;
    using NSCam::NSPipelinePlugin::Join;
    using NSCam::NSPipelinePlugin::JoinPlugin;
    
    using namespace NSCam::NSPipelinePlugin;
    using NSCam::MSize;
    
    using NSCam::MERROR;
    using NSCam::IImageBuffer;
    using NSCam::IMetadata;
    using NSCam::Type2Type;
    
    #ifdef LOG_TAG
    #undef LOG_TAG
    #endif // LOG_TAG
    #define LOG_TAG "WatermarkPreview"
    
    #include <log/log.h>
    #include <android/log.h>
    
    #define MY_LOGI(fmt, arg...)  ALOGI("[%s] " fmt, __FUNCTION__, ##arg)
    #define MY_LOGD(fmt, arg...)  ALOGD("[%s] " fmt, __FUNCTION__, ##arg)
    #define MY_LOGW(fmt, arg...)  ALOGW("[%s] " fmt, __FUNCTION__, ##arg)
    #define MY_LOGE(fmt, arg...)  ALOGE("[%s] " fmt, __FUNCTION__, ##arg)
    #define FUNCTION_IN   MY_LOGD("%s +", __FUNCTION__)
    #define FUNCTION_OUT  MY_LOGD("%s -", __FUNCTION__)
    
    template <class T>
    inline bool
    tryGetMetadata(IMetadata const *pMetadata, MUINT32 tag, T& rVal)
    {
        if(pMetadata == nullptr) return MFALSE;
    
        IMetadata::IEntry entry = pMetadata->entryFor(tag);
        if(!entry.isEmpty())
        {
            rVal = entry.itemAt(0,Type2Type<T>());
            return true;
        }
        else
        {
    #define var(v) #v
    #define type(t) #t
            MY_LOGW("no metadata %s in %s", var(tag), type(pMetadata));
    #undef type
    #undef var
        }
        return false;
    }
    
    class WatermarkPreview : public JoinPlugin::IProvider {
    public:
        typedef JoinPlugin::Property Property;
        typedef JoinPlugin::Selection Selection;
        typedef JoinPlugin::Request::Ptr RequestPtr;
        typedef JoinPlugin::RequestCallback::Ptr RequestCallbackPtr;
    
    private:
        bool mDisponly = false;
        bool mInplace = false;
        int mOpenID1 = 0;
        int mOpenID2 = 0;
    
        MBOOL mEnable = 1;
        MBOOL mDump = 0;
        unsigned char *mSrcRGBA = nullptr;
        unsigned char *mWatermarkRGBA = nullptr;
        int mWatermarkWidth = 0;
        int mWatermarkHeight = 0;
    
    public:
        WatermarkPreview();
    
        ~WatermarkPreview();
    
        void init();
    
        void uninit();
    
        void abort(std::vector <RequestPtr> &pRequests);
    
        void set(MINT32 openID1, MINT32 openID2);
    
        const Property &property();
    
        MERROR negotiate(Selection &sel);
    
        MERROR process(RequestPtr pRequest, RequestCallbackPtr pCallback);
    
    private:
        MERROR getConfigSetting(Selection &sel);
    
        MERROR getP1Setting(Selection &sel);
    
        MERROR getP2Setting(Selection &sel);
    };
    
    WatermarkPreview::WatermarkPreview() {
        FUNCTION_IN;
        mEnable = property_get_bool("vendor.debug.camera.watermark.preview.enable", 1);
        mDump = property_get_bool("vendor.debug.camera.watermark.preview.dump", 0);
        FUNCTION_OUT;
    }
    
    WatermarkPreview::~WatermarkPreview() {
        FUNCTION_IN;
        FUNCTION_OUT;
    }
    
    void WatermarkPreview::init() {
        FUNCTION_IN;
        mWatermarkWidth = 180;
        mWatermarkHeight = 640;
        int watermarkSize = mWatermarkWidth * mWatermarkHeight * 4;
        mWatermarkRGBA = (unsigned char *) malloc(watermarkSize);
    
        FILE *fp;
        char path[256];
        snprintf(path, sizeof(path), "/vendor/res/images/watermark.rgba");
        if ((fp = fopen(path, "r")) == NULL) {
            MY_LOGE("Failed to open /vendor/res/images/watermark.rgba");
        }
        fread(mWatermarkRGBA, 1, watermarkSize, fp);
        fclose(fp);
        FUNCTION_OUT;
    }
    
    void WatermarkPreview::uninit() {
        FUNCTION_IN;
        free(mSrcRGBA);
        free(mWatermarkRGBA);
        FUNCTION_OUT;
    }
    
    void WatermarkPreview::abort(std::vector <RequestPtr> &pRequests) {
        FUNCTION_IN;
        (void)pRequests;
        FUNCTION_OUT;
    }
    
    void WatermarkPreview::set(MINT32 openID1, MINT32 openID2) {
        FUNCTION_IN;
        MY_LOGD("set openID1:%d openID2:%d", openID1, openID2);
        mOpenID1 = openID1;
        mOpenID2 = openID2;
        FUNCTION_OUT;
    }
    
    const WatermarkPreview::Property &WatermarkPreview::property() {
        FUNCTION_IN;
        static Property prop;
        static bool inited;
    
        if (!inited) {
            prop.mName = "TP_WATERMARK";
            prop.mFeatures = TP_FEATURE_WATERMARK;
            //prop.mInPlace = MTRUE;
            //prop.mFaceData = eFD_Current;
            //prop.mPosition = 0;
            inited = true;
        }
        FUNCTION_OUT;
        return prop;
    }
    
    MERROR WatermarkPreview::negotiate(Selection &sel) {
        FUNCTION_IN;
        MERROR ret = OK;
    
        if (sel.mSelStage == eSelStage_CFG) {
            ret = getConfigSetting(sel);
        } else if (sel.mSelStage == eSelStage_P1) {
            ret = getP1Setting(sel);
        } else if (sel.mSelStage == eSelStage_P2) {
            ret = getP2Setting(sel);
        }
        FUNCTION_OUT;
        return ret;
    }
    
    MERROR WatermarkPreview::process(RequestPtr pRequest, RequestCallbackPtr pCallback) {
        FUNCTION_IN;
        (void) pCallback;
        MERROR ret = -EINVAL;
        MBOOL needRun = MFALSE;
        IImageBuffer *in = NULL, *out = NULL;
    
        if (pRequest->mIBufferMain1 != NULL && pRequest->mOBufferMain1 != NULL) {
            in = pRequest->mIBufferMain1->acquire();
            out = pRequest->mOBufferMain1->acquire();
    
            int frameNo = 0, requestNo = 0;
            if (pRequest->mIMetadataHal1 != nullptr) {
                IMetadata *pIMetataHAL1 = pRequest->mIMetadataHal1->acquire();
                if (pIMetataHAL1 != NULL) {
                    if (!tryGetMetadata<int>(pIMetataHAL1, MTK_PIPELINE_FRAME_NUMBER, frameNo)) {
                        frameNo = 0;
                    }
                    if (!tryGetMetadata<int>(pIMetataHAL1, MTK_PIPELINE_REQUEST_NUMBER, requestNo)) {
                        requestNo = 0;
                    }
                    pRequest->mIMetadataHal1->release();
                    MY_LOGD("frameNo: %d, requestNo: %d", frameNo, requestNo);
                } else {
                    MY_LOGD("HAL metadata empty");
                }
            }
    
            MY_LOGD("in[%d](%dx%d)=%p out[%d](%dx%d)=%p",
                    in->getPlaneCount(), in->getImgSize().w, in->getImgSize().h, in,
                    out->getPlaneCount(), out->getImgSize().w, out->getImgSize().h, out);
    
            if (pRequest->mIMetadataApp != nullptr) {
                IMetadata *pIMetadataApp = pRequest->mIMetadataApp->acquire();
                MINT32 mode = 0;
                if (!tryGetMetadata<MINT32>(pIMetadataApp, QXT_FEATURE_WATERMARK, mode)) {
                    mode = 0;
                }
                needRun = mode == 1 ? 1 : 0;
                pRequest->mIMetadataApp->release();
            }
            MY_LOGD("needRun: %d", needRun);
    
            int width = in->getImgSize().w;
            int height = in->getImgSize().h;
            MINT inFormat = in->getImgFormat();
    
            if (needRun && inFormat == NSCam::eImgFmt_YV12) {
                uint32_t currentTime = (NSCam::Utils::TimeTool::getReadableTime()) % 1000;
                time_t timep;
                time (&timep);
                char currentDate[20];
                strftime(currentDate, sizeof(currentDate), "%Y%m%d_%H%M%S", localtime(&timep));
    
                //dump input YV12
                if (mDump) {
                    char path[256];
                    snprintf(path, sizeof(path), "/data/vendor/camera_dump/preview_in_frame%d_%dx%d_%s_%d.yv12",
                        frameNo, width, height, currentDate, currentTime);
                    in->saveToFile(path);
                }
    
                nsecs_t t1 = systemTime(CLOCK_MONOTONIC);
                if (mSrcRGBA == NULL) {
                    mSrcRGBA = (unsigned char *) malloc(width * height * 4);
                }
                //convert YV12 to RGBA
                libyuv::I420ToABGR((unsigned char *)(in->getBufVA(0)), width,
                                   (unsigned char *)(in->getBufVA(2)), width >> 1,
                                   (unsigned char *)(in->getBufVA(1)), width >> 1,
                                   mSrcRGBA, width * 4,
                                   width, height);
                nsecs_t t2 = systemTime(CLOCK_MONOTONIC);
                MY_LOGD("Prepare src cost %02ld ms", ns2ms(t2 - t1));
    
                Watermark::add(mSrcRGBA, width, height, mWatermarkRGBA, mWatermarkWidth, mWatermarkHeight, (width - mWatermarkWidth) / 2, (height - mWatermarkHeight) / 2);
                nsecs_t t3 = systemTime(CLOCK_MONOTONIC);
                MY_LOGD("Add watermark cost %02ld ms", ns2ms(t3 - t2));
    
                //convert RGBA to YV12
                libyuv::ABGRToI420(mSrcRGBA, width * 4,
                                   (unsigned char *)(out->getBufVA(0)), width,
                                   (unsigned char *)(out->getBufVA(2)), width >> 1,
                                   (unsigned char *)(out->getBufVA(1)), width >> 1,
                                   width, height);
                nsecs_t t4 = systemTime(CLOCK_MONOTONIC);
                MY_LOGD("Copy in to out cost %02ld ms", ns2ms(t4 - t3));
    
                //dump output YV12
                if (mDump) {
                    char path[256];
                    snprintf(path, sizeof(path), "/data/vendor/camera_dump/preview_out_frame%d_%dx%d_%s_%d.yv12",
                        frameNo, width, height, currentDate, currentTime);
                    out->saveToFile(path);
                }
    
            } else {
                if (!needRun) {
                    MY_LOGE("No need run, skip add watermark for preview.");
                } else if (inFormat != NSCam::eImgFmt_YV12) {
                    MY_LOGE("Unsupported format, skip add watermark for preview.");
                } else {
                    MY_LOGE("Unknown exception, skip add watermark for preview.");
                }
                memcpy((unsigned char *) (out->getBufVA(0)),
                       (unsigned char *)(in->getBufVA(0)),
                       in->getBufSizeInBytes(0));
                memcpy((unsigned char *) (out->getBufVA(1)),
                       (unsigned char *)(in->getBufVA(1)),
                       in->getBufSizeInBytes(1));
                memcpy((unsigned char *) (out->getBufVA(2)),
                       (unsigned char *)(in->getBufVA(2)),
                       in->getBufSizeInBytes(2));
            }
    
            pRequest->mIBufferMain1->release();
            pRequest->mOBufferMain1->release();
            ret = OK;
        }
    
        FUNCTION_OUT;
        return ret;
    }
    
    MERROR WatermarkPreview::getConfigSetting(Selection &sel) {
        MY_LOGI("max out size(%dx%d)",
                sel.mCfgInfo.mMaxOutSize.w, sel.mCfgInfo.mMaxOutSize.h);
    
        mDisponly = property_get_bool("vendor.debug.tpi.s.fb.disponly", 0);
        mInplace = mDisponly || property_get_bool("vendor.debug.tpi.s.fb.inplace", 0);
    
        sel.mCfgOrder = 3;
        sel.mCfgJoinEntry = eJoinEntry_S_YUV;
        sel.mCfgInplace = mInplace;
        sel.mCfgEnableFD = MTRUE;
        sel.mCfgRun = mEnable;
        sel.mIBufferMain1.setRequired(MTRUE);
        if (!mDisponly && property_get_bool("vendor.debug.tpi.s.fb.nv21", 0)) {
            sel.mIBufferMain1.addAcceptedFormat(NSCam::eImgFmt_NV21);
        }
        if (!mDisponly && property_get_bool("vendor.debug.tpi.s.fb.size", 0)) {
            sel.mIBufferMain1.setSpecifiedSize(sel.mCfgInfo.mMaxOutSize);
        }
        sel.mOBufferMain1.setRequired(MTRUE);
        sel.mIBufferMain1.addAcceptedFormat(NSCam::eImgFmt_YV12);
        sel.mIBufferMain1.addAcceptedSize(eImgSize_Full);
    
        IMetadata *meta = sel.mIMetadataApp.getControl().get();
        MY_LOGD("sessionMeta=%p", meta);
    
        return OK;
    }
    
    MERROR WatermarkPreview::getP1Setting(Selection &sel) {
        (void) sel;
        return OK;
    }
    
    MERROR WatermarkPreview::getP2Setting(Selection &sel) {
        MBOOL run = MTRUE;
        sel.mP2Run = run;
        return OK;
    }
    
    REGISTER_PLUGIN_PROVIDER(Join, WatermarkPreview);
    
    
    4.2.5 mtkcam3/3rdparty/customer/Android.mk

    最终vendor.img需要的目标共享库是libmtkcam_3rdparty.customer.so。因此,我们还需要修改Android.mk,使模块libmtkcam_3rdparty.customer依赖libmtkcam.plugin.tp_watermark。vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/Android.mk:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/Android.mk b/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/Android.mk
    old mode 100644
    new mode 100755
    index ce060c39f9..ff5763d3c2
    --- a/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/Android.mk
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/Android.mk
    @@ -70,6 +70,13 @@ LOCAL_WHOLE_STATIC_LIBRARIES += libmtkcam.plugin.tp_purebokeh
     LOCAL_SHARED_LIBRARIES += libcam.iopipe
     LOCAL_SHARED_LIBRARIES += libmtkcam_modulehelper
     endif
    +
    +ifeq ($(QXT_WATERMARK_SUPPORT), yes)
    +LOCAL_SHARED_LIBRARIES += libwatermark
    +LOCAL_SHARED_LIBRARIES += libyuv.vendor
    +LOCAL_WHOLE_STATIC_LIBRARIES += libmtkcam.plugin.tp_watermark
    +endif
    +
     # for app super night ev decision (experimental for customer only)
     LOCAL_WHOLE_STATIC_LIBRARIES += libmtkcam.control.customersupernightevdecision
    
    
    4.2.6 预置水印文件
    diff --git a/device/mediateksample/k63v2_64_bsp/device.mk b/device/mediateksample/k63v2_64_bsp/device.mk
    index 2619000c72..048c33462e 100644
    --- a/device/mediateksample/k63v2_64_bsp/device.mk
    +++ b/device/mediateksample/k63v2_64_bsp/device.mk
    @@ -98,6 +98,9 @@ PRODUCT_COPY_FILES += vendor/mediatek/proprietary/custom/k63v2_64_bsp/factory/re
     PRODUCT_COPY_FILES += vendor/mediatek/proprietary/custom/k63v2_64_bsp/factory/res/images/lcd_test_01.png:$(TARGET_COPY_OUT_VENDOR)/res/images/lcd_test_01.png:mtk
     PRODUCT_COPY_FILES += vendor/mediatek/proprietary/custom/k63v2_64_bsp/factory/res/images/lcd_test_02.png:$(TARGET_COPY_OUT_VENDOR)/res/images/lcd_test_02.png:mtk
    
    +ifeq ($(QXT_WATERMARK_SUPPORT),yes)
    +PRODUCT_COPY_FILES += vendor/mediatek/proprietary/hardware/mtkcam3/3rdparty/customer/tp_watermark/res/watermark.rgba::$(TARGET_COPY_OUT_VENDOR)/res/images/watermark.rgba
    +endif
    
     # overlay has priorities. high <-> low.
    
    

    camera hal进程为mtk_camera_hal,它要读取/vendor/res/images/watermark.rgba,读取需要vendor_file SELinux权限。这里为mtk_camera_hal配置SELinux权限:

    diff --git a/device/mediatek/sepolicy/bsp/non_plat/mtk_hal_camera.te b/device/mediatek/sepolicy/bsp/non_plat/mtk_hal_camera.te
    index 8de5d0a437..7ebd9a03e5 100644
    --- a/device/mediatek/sepolicy/bsp/non_plat/mtk_hal_camera.te
    +++ b/device/mediatek/sepolicy/bsp/non_plat/mtk_hal_camera.te
    @@ -92,6 +92,7 @@ allow mtk_hal_camera sysfs_boot_mode:file { read open };
     # Purpose: NDD
     allow mtk_hal_camera vendor_data_file:dir create_dir_perms;
     allow mtk_hal_camera vendor_data_file:file create_file_perms;
    +allow mtk_hal_camera vendor_file:file { read getattr open };
    
    

    五、自定义metadata

    添加metadata是为了让APP层能够通过metadata传递相应的参数给HAL层。APP层是通过CaptureRequest.Builder.set(@NonNull Key<T> key, T value)来设置参数的。
    由于我们是自定义的feature,无法复用MTK提供的metadata,因此,我们需要自定义metadata。
    vendor/mediatek/proprietary/hardware/mtkcam/include/mtkcam/utils/metadata/client/mtk_metadata_tag.h:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam/include/mtkcam/utils/metadata/client/mtk_metadata_tag.h b/vendor/mediatek/proprietary/hardware/mtkcam/include/mtkcam/utils/metadata/client/mtk_metadata_tag.h
    index 22d4aa2bf2..b020352092 100755
    --- a/vendor/mediatek/proprietary/hardware/mtkcam/include/mtkcam/utils/metadata/client/mtk_metadata_tag.h
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam/include/mtkcam/utils/metadata/client/mtk_metadata_tag.h
    @@ -89,6 +89,7 @@ typedef enum mtk_camera_metadata_section {
         MTK_BGSERVICE_FEATURE           = 12,
         MTK_CONFIGURE_SETTING           = 13,
         MTK_FLASH_FEATURE               = 14,
    +    QXT_FEATURE                     = 15,
         MTK_VENDOR_SECTION_COUNT,
     } mtk_camera_metadata_section_t;
    
    @@ -146,6 +147,7 @@ typedef enum mtk_camera_metadata_section_start {
         MTK_CONFIGURE_SETTING_START                 = (MTK_CONFIGURE_SETTING + MTK_VENDOR_TAG_SECTION) << 16,
         MTK_FLASH_FEATURE_START                     = (MTK_FLASH_FEATURE + MTK_VENDOR_TAG_SECTION) << 16,
    
    +    QXT_FEATURE_START                           = (QXT_FEATURE + MTK_VENDOR_TAG_SECTION) << 16,
     } mtk_camera_metadata_section_start_t;
    
    @@ -599,6 +601,8 @@ typedef enum mtk_camera_metadata_tag {
         MTK_FLASH_FEATURE_CALIBRATION_RESULT,    // flash calibration result
         MTK_FLASH_FEATURE_END,
    
    +    QXT_FEATURE_WATERMARK = QXT_FEATURE_START,
    +    QXT_FEATURE_END,
     } mtk_camera_metadata_tag_t;
    
     /**
    
    

    vendor/mediatek/proprietary/hardware/mtkcam/include/mtkcam/utils/metadata/client/mtk_metadata_tag_info.inl:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam/include/mtkcam/utils/metadata/client/mtk_metadata_tag_info.inl b/vendor/mediatek/proprietary/hardware/mtkcam/include/mtkcam/utils/metadata/client/mtk_metadata_tag_info.inl
    index 15449c433d..1b4fc75a0e 100755
    --- a/vendor/mediatek/proprietary/hardware/mtkcam/include/mtkcam/utils/metadata/client/mtk_metadata_tag_info.inl
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam/include/mtkcam/utils/metadata/client/mtk_metadata_tag_info.inl
    @@ -91,6 +91,11 @@ _IMP_SECTION_INFO_(MTK_DISTORTION_CORRECTION_INFO,  "mtk.distortionCorrection")
     _IMP_SECTION_INFO_(MTK_IOPIPE_INFO,                 "mtk.iopipe.info")
     _IMP_SECTION_INFO_(MTK_HAL_INFO,                    "mtk.hal.info")
    
    +_IMP_SECTION_INFO_(QXT_FEATURE,      "com.qxt.camera")
    +
    +_IMP_TAG_INFO_( QXT_FEATURE_WATERMARK,
    +                MINT32,     "watermark")
    +
     /******************************************************************************
      *
      ******************************************************************************/
    
    

    vendor/mediatek/proprietary/hardware/mtkcam/utils/metadata/vendortag/VendorTagTable.h:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam/utils/metadata/vendortag/VendorTagTable.h b/vendor/mediatek/proprietary/hardware/mtkcam/utils/metadata/vendortag/VendorTagTable.h
    index 2481492f90..33e581adfd 100755
    --- a/vendor/mediatek/proprietary/hardware/mtkcam/utils/metadata/vendortag/VendorTagTable.h
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam/utils/metadata/vendortag/VendorTagTable.h
    @@ -377,6 +377,16 @@ static auto& _FlashFeature_()
     }
    
    +static auto& _QxtFeature_()
    +{
    +    static const std::map<uint32_t, VendorTag_t>
    +    sInst = {
    +        _TAG_(QXT_FEATURE_WATERMARK,
    +            "watermark",   TYPE_INT32),
    +     };
    +     //
    +     return sInst;
    +}
    
     /******************************************************************************
      *
    @@ -460,6 +470,10 @@ static auto& getGlobalSections()
                         MTK_FLASH_FEATURE_END,
                         _FlashFeature_() ),
    
    +        _SECTION_( "com.qxt.camera",
    +                    QXT_FEATURE_START,
    +                    QXT_FEATURE_END,
    +                    _QxtFeature_() ),
         };
    
         // append custom vendor tags sections to mtk sections
    
    

    vendor/mediatek/proprietary/hardware/mtkcam/utils/metastore/metadataprovider/constructStaticMetadata.cpp:

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam/utils/metastore/metadataprovider/constructStaticMetadata.cpp b/vendor/mediatek/proprietary/hardware/mtkcam/utils/metastore/metadataprovider/constructStaticMetadata.cpp
    index edd5b5f1b9..591b25b162 100755
    --- a/vendor/mediatek/proprietary/hardware/mtkcam/utils/metastore/metadataprovider/constructStaticMetadata.cpp
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam/utils/metastore/metadataprovider/constructStaticMetadata.cpp
    @@ -578,6 +578,19 @@ updateData(IMetadata &rMetadata)
             }
         }
     #endif
    +
    +#if 1
    +    {
    +        IMetadata::IEntry qxtAvailRequestEntry = rMetadata.entryFor(MTK_REQUEST_AVAILABLE_REQUEST_KEYS);
    +        qxtAvailRequestEntry.push_back(QXT_FEATURE_WATERMARK , Type2Type< MINT32 >());
    +        rMetadata.update(qxtAvailRequestEntry.tag(), qxtAvailRequestEntry);
    +
    +        IMetadata::IEntry qxtAvailSessionEntry = rMetadata.entryFor(MTK_REQUEST_AVAILABLE_SESSION_KEYS);
    +        qxtAvailSessionEntry.push_back(QXT_FEATURE_WATERMARK , Type2Type< MINT32 >());
    +        rMetadata.update(qxtAvailSessionEntry.tag(), qxtAvailSessionEntry);
    +    }
    +#endif
    +
         // update multi-cam feature mode to static metadata
         // vendor tag
         {
    
    

    前面这些步骤完成之后,集成工作就基本完成了。我们需要重新编译一下系统源码,为节约时间,也可以只编译vendor.img。趁着编译的时间,我们可以写一个demo来验证算法是否集成成功了。

    六、APP调用算法

    WatermarkActivity:

    public class WatermarkActivity extends BaseActivity {
        private static final String TAG = WatermarkActivity.class.getSimpleName();
    
        /*
         * 16:9 picture size: 3840x2160  preview size 1280x720
         * 4:3  picture size: 3264x2448  preview size 960x720
         * Now is 4:3
         */
        private static final int PREVIEW_WIDTH = 1280;
        private static final int PREVIEW_HEIGHT = 720;
        private static final int CAPTURE_WIDTH = 3264;
        private static final int CAPTURE_HEIGHT = 2448;
    
        private static final String IMAGE_PATH =
                Environment.getExternalStorageDirectory().getAbsolutePath()
                + File.separator + "DCIM" + File.separator + "Camera";
    
        private static final String CAMERA_ID = "0";
        private static final String KEY_WATERMARK = "com.qxt.camera.watermark";
    
        private static final String SP_NAME = "watermark";
        private static final String SP_STATE_KEY = "state";
    
        private AutoFitTextureView mTextureView;
        private ProgressBar mProgressBar;
        private Handler mMainHandler;
        private Handler mCameraHandler;
        private HandlerThread mCameraHandlerThread;
    
        private CameraManager mCameraManager;
        private CaptureRequest.Builder mPreviewBuilder;
        private CameraDevice mCameraDevice;
        private CameraCaptureSession mCameraCaptureSession;
    
        private MediaActionSound mCameraSound;
        private String mTakePictureTime;
        private SimpleDateFormat mDateFormat = new SimpleDateFormat(
                "yyyyMMdd_HHmmss", Locale.getDefault());
    
        private ImageReader mCaptureImageReader;
        private Surface mSurface;
        public CaptureRequest.Key<int[]> mVendorKey;
        private int mVendorKeyEnable;
        private SharedPreferences mSharedPref;
    
        @Override
        protected void onCreate(Bundle savedInstanceState) {
            super.onCreate(savedInstanceState);
            getWindow().setFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON,
                    WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
            setContentView(R.layout.activity_watermark);
            mProgressBar = findViewById(R.id.progressbar);
            mTextureView = findViewById(R.id.texture);
            mTextureView.setAspectRatio(PREVIEW_HEIGHT, PREVIEW_WIDTH);
    
            mCameraSound = new MediaActionSound();
            mCameraSound.load(MediaActionSound.SHUTTER_CLICK);
    
            mCameraManager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);
            mMainHandler = new Handler();
            initVendorTag();
            mSharedPref = getSharedPreferences(SP_NAME, Context.MODE_PRIVATE);
            mVendorKeyEnable = mSharedPref.getInt(SP_STATE_KEY, 0);
            getCameraCharacteristics(CAMERA_ID);
        }
    
        @Override
        public boolean onCreateOptionsMenu(Menu menu) {
            getMenuInflater().inflate(R.menu.menu_watermark, menu);
    
            Switch s = menu.findItem(R.id.action_watermark)
                    .getActionView().findViewById(R.id.switch_watermark);
            s.setChecked(mVendorKeyEnable > 0);
            s.setOnCheckedChangeListener(new CompoundButton.OnCheckedChangeListener() {
    
                @Override
                public void onCheckedChanged(CompoundButton btn, boolean isChecked) {
                    if (isChecked) {
                        mVendorKeyEnable = 1;
                    } else {
                        mVendorKeyEnable = 0;
                    }
                    mSharedPref.edit().putInt(SP_STATE_KEY, mVendorKeyEnable).commit();
                    if (mPreviewBuilder != null && mCameraCaptureSession != null) {
                        try {
                            mCameraCaptureSession.stopRepeating();
                            setVendorTag(mPreviewBuilder);
                            mCameraCaptureSession.setRepeatingRequest(mPreviewBuilder.build(),
                                    mSessionCaptureCallback, mCameraHandler);
                        } catch (CameraAccessException e) {
                            e.printStackTrace();
                        }
                    }
                    LogUtils.d(TAG, "[onCheckedChanged] isChecked=" + isChecked
                            + ", mWideAngleEnable=" + mVendorKeyEnable);
                }
            });
            return true;
        }
    
        @Override
        protected void onResume() {
            super.onResume();
            initLooper();
    
            if (mTextureView.isAvailable()) {
                openCamera();
            } else {
                mTextureView.setSurfaceTextureListener(mSurfaceTextureListener);
            }
    
        }
    
        @Override
        protected void onPause() {
            super.onPause();
            closeCamera();
            stopLooper();
        }
    
        @Override
        protected void onDestroy() {
            super.onDestroy();
        }
    
        public void onClick(View view) {
            if (view != null && view.getId() == R.id.btn_capture) {
                takePicture();
            }
        }
    
        private void initLooper() {
            mCameraHandlerThread = new HandlerThread("WideAngleCamera");
            mCameraHandlerThread.start();
            mCameraHandler = new Handler(mCameraHandlerThread.getLooper());
        }
    
        private void stopLooper() {
            try {
                mCameraHandlerThread.quit();
                mCameraHandlerThread.join();
                mCameraHandlerThread = null;
                mCameraHandler = null;
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    
        @SuppressLint("MissingPermission")
        private void openCamera() {
            try {
                mCameraManager.openCamera(CAMERA_ID, new CameraDevice.StateCallback() {
                    @Override
                    public void onOpened(@NonNull CameraDevice camera) {
                        mCameraDevice = camera;
                        createCameraPreviewSession();
                    }
    
                    @Override
                    public void onDisconnected(@NonNull CameraDevice camera) {
                        LogUtils.d(TAG, "onDisconnected");
                        camera.close();
                        mCameraDevice = null;
                    }
    
                    @Override
                    public void onError(@NonNull CameraDevice camera, int error) {
                        LogUtils.d(TAG, "onError error=" + error);
                        camera.close();
                        mCameraDevice = null;
                    }
                }, mCameraHandler);
            } catch (CameraAccessException e) {
                e.printStackTrace();
            }
        }
    
        private void closeCamera() {
            try {
                if (null != mCameraCaptureSession) {
                    mCameraCaptureSession.close();
                    mCameraCaptureSession = null;
                }
                if (null != mCameraDevice) {
                    mCameraDevice.close();
                    mCameraDevice = null;
                }
                if (null != mCaptureImageReader) {
                    mCaptureImageReader.close();
                    mCaptureImageReader = null;
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    
        private void createCameraPreviewSession() {
            if (isFinishing() || isDestroyed() || mCameraDevice == null) {
                return;
            }
    
            try {
                mCaptureImageReader = ImageReader.newInstance(CAPTURE_WIDTH,
                        CAPTURE_HEIGHT, ImageFormat.YUV_420_888, 2);
                mCaptureImageReader.setOnImageAvailableListener(
                        mCaptureOnImageAvailableListener, mCameraHandler);
    
                mPreviewBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
                setVendorTag(mPreviewBuilder);
                mPreviewBuilder.addTarget(mSurface);
                mCameraDevice.createCaptureSession(Arrays.asList(mSurface,
                        mCaptureImageReader.getSurface()),
                        new CameraCaptureSession.StateCallback() {
                            @Override
                            public void onConfigured(@NonNull CameraCaptureSession session) {
                                if (isFinishing() || isDestroyed() || mCameraDevice == null) {
                                    return;
                                }
                                try {
                                    mCameraCaptureSession = session;
                                    mPreviewBuilder.set(CaptureRequest.CONTROL_AF_MODE,
                                            CaptureRequest.CONTROL_AF_MODE_AUTO);
                                    mPreviewBuilder.set(CaptureRequest.CONTROL_AF_MODE,
                                            CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE);
                                    mCameraCaptureSession.setRepeatingRequest(mPreviewBuilder.build(),
                                            mSessionCaptureCallback, mCameraHandler);
                                } catch (CameraAccessException e) {
                                    e.printStackTrace();
                                }
                            }
    
                            @Override
                            public void onConfigureFailed(@NonNull CameraCaptureSession session) {
    
                            }
                        }, mCameraHandler);
            } catch (CameraAccessException e) {
                e.printStackTrace();
            }
        }
    
        private void takePicture() {
            try {
                mTakePictureTime = mDateFormat.format(System.currentTimeMillis());
                final CaptureRequest.Builder captureBuilder =
                        mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE);
                setVendorTag(captureBuilder);
                Surface surface = mCaptureImageReader.getSurface();
                captureBuilder.addTarget(surface);
                mCameraCaptureSession.capture(captureBuilder.build(),
                        new CameraCaptureSession.CaptureCallback() {
                    @Override
                    public void onCaptureCompleted(@NonNull CameraCaptureSession session,
                                                   @NonNull CaptureRequest request,
                                                   @NonNull TotalCaptureResult result) {
                        super.onCaptureCompleted(session, request, result);
                    }
                }, mCameraHandler);
                if (mCameraSound != null) {
                    mCameraSound.play(MediaActionSound.SHUTTER_CLICK);
                }
                mProgressBar.setVisibility(View.VISIBLE);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    
        private void notifyPictureTaken() {
            mProgressBar.setVisibility(View.GONE);
            Toast toast = Toast.makeText(WatermarkActivity.this,
                    getString(R.string.image_saved, IMAGE_PATH), Toast.LENGTH_SHORT);
            toast.setGravity(Gravity.CENTER, 0, 0);
            toast.show();
        }
    
        @SuppressWarnings("unused")
        private void getCameraCharacteristics(String cameraId) {
            try {
                CameraCharacteristics cs = mCameraManager.getCameraCharacteristics(cameraId);
                StreamConfigurationMap map = cs.get(
                        CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
                if (map != null) {
                    //获取图像输出的尺寸
                    Size[] pictureSize = map.getOutputSizes(ImageFormat.JPEG);
                    Size[] previewSize = map.getOutputSizes(SurfaceTexture.class);
                    StringBuilder pictureBuilder = new StringBuilder("picture sizes: ");
                    for (Size size : pictureSize) {
                        pictureBuilder.append(size);
                        pictureBuilder.append(", ");
                    }
                    LogUtils.d(TAG, pictureBuilder.toString());
    
                    StringBuilder previewBuilder = new StringBuilder("preview sizes: ");
                    for (Size size : previewSize) {
                        previewBuilder.append(size);
                        previewBuilder.append(", ");
                    }
                    LogUtils.d(TAG, previewBuilder.toString());
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    
        TextureView.SurfaceTextureListener mSurfaceTextureListener
                = new TextureView.SurfaceTextureListener() {
            @Override
            public void onSurfaceTextureAvailable(SurfaceTexture surfaceTexture, int w, int h) {
                LogUtils.d(TAG, "onSurfaceAvaliable, width:" + w + ", height:" + h);
                surfaceTexture.setDefaultBufferSize(PREVIEW_WIDTH, PREVIEW_HEIGHT);
                mSurface = new Surface(surfaceTexture);
                openCamera();
            }
    
            @Override
            public void onSurfaceTextureSizeChanged(SurfaceTexture surfaceTexture, int w, int h) {
                LogUtils.d(TAG, "onSurfaceTextureSizeChanged, width:" + w + ", height:" + h);
            }
    
            @Override
            public boolean onSurfaceTextureDestroyed(SurfaceTexture surface) {
                LogUtils.d(TAG, "onSurfaceTextureDestroyed");
                mSurface = null;
                return false;
            }
    
            @Override
            public void onSurfaceTextureUpdated(SurfaceTexture surface) {
    
            }
        };
    
        private final CameraCaptureSession.CaptureCallback mSessionCaptureCallback
                = new CameraCaptureSession.CaptureCallback() {
            @Override
            public void onCaptureCompleted(@NonNull CameraCaptureSession session,
                                           @NonNull CaptureRequest request,
                                           @NonNull TotalCaptureResult result) {
                super.onCaptureCompleted(session, request, result);
                mCameraCaptureSession = session;
            }
        };
    
        private final ImageReader.OnImageAvailableListener mCaptureOnImageAvailableListener
                = new ImageReader.OnImageAvailableListener() {
            @Override
            public void onImageAvailable(final ImageReader reader) {
                LogUtils.d(TAG, "capture onImageAvailable");
                Image image = reader.acquireLatestImage();
                if (image == null) return;
                ImageUtils.saveImage(WatermarkActivity.this, image, IMAGE_PATH,
                        "WIDE_" + mTakePictureTime, ImageUtils.ROTATE_90);
                image.close();
                LogUtils.d(TAG, "saved");
                mMainHandler.post(new Runnable() {
                    @Override
                    public void run() {
                        notifyPictureTaken();
                    }
                });
            }
        };
    
        private void initVendorTag() {
            try {
                CameraCharacteristics c = mCameraManager.getCameraCharacteristics(CAMERA_ID);
                mVendorKey = CameraUtils.getSessionKey(c, KEY_WATERMARK);
            } catch (CameraAccessException e) {
                e.printStackTrace();
            }
        }
    
        private void setVendorTag(CaptureRequest.Builder builder) {
            if (mVendorKey != null) {
                builder.set(mVendorKey, new int[]{mVendorKeyEnable});
                LogUtils.d(TAG, "[setVendorTag] set watermark to " + mVendorKeyEnable);
            }
        }
    }
    
    

    CameraUtils:

    public class CameraUtils {
        private static final String TAG = CameraUtils.class.getSimpleName();
    
        @RequiresApi(api = Build.VERSION_CODES.P)
        public static CaptureRequest.Key<int[]> getSessionKey(
                CameraCharacteristics cs, String key) {
            if (cs == null) {
                LogUtils.i(TAG, "[getSessionKey] CameraCharacteristics is null");
                return null;
            }
            CaptureRequest.Key<int[]> targetKey = null;
            List<CaptureRequest.Key<?>> sessionKeys = cs.getAvailableSessionKeys();
            if (sessionKeys == null) {
                LogUtils.i(TAG, "[getSessionKey] No keys!");
                return null;
            }
            for (CaptureRequest.Key<?> sessionKey : sessionKeys) {
                if (sessionKey.getName().equals(key)) {
                    LogUtils.i(TAG, "[getSessionKey] key :" + key);
                    targetKey = (CaptureRequest.Key<int[]>) sessionKey;
                    break;
                }
            }
            return targetKey;
        }
    }
    
    

    为样机刷入系统整包或者vendor.img,开机后,安装demo验证。我们来拍一张电脑显示器看看效果:
    预览:

    image

    拍照:

    image

    七、遇到的问题及解决方法

    问题1:
    如果process函数中buffer的acquire和release没有成对出现,也就是buffer没正常release,那么就会出现连续拍多张之后,算法未被调用的情况。
    问题1解决方法:
    YUVNode.cpp中加入一个保险的代码,万一集成代码中忘记release,在YUVNode中release。

    diff --git a/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp b/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp
    index 8bb794ba02..d4343aaccf 100755
    --- a/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp
    +++ b/vendor/mediatek/proprietary/hardware/mtkcam3/feature/core/featurePipe/capture/nodes/YUVNode.cpp
    @@ -1050,9 +1051,11 @@ MBOOL YUVNode::onRequestProcess(RequestPtr& pRequest)
    
         auto pPlgRequest = mPlugin->createRequest();
    
    -    pPlgRequest->mIBufferFull  = (iBufferFullHandle == NULL) ? PluginHelper::CreateBuffer(pNodeReq, TID_MAN_FULL_YUV, INPUT) : iBufferFullHandle;
    +    //pPlgRequest->mIBufferFull  = (iBufferFullHandle == NULL) ? PluginHelper::CreateBuffer(pNodeReq, TID_MAN_FULL_YUV, INPUT) : iBufferFullHandle;
    +    pPlgRequest->mIBufferFull  = (iBufferFullHandle == NULL) ? PluginHelper::CreateBuffer(pNodeReq, TID_MAN_FULL_YUV, INPUT) : std::move(iBufferFullHandle);
         pPlgRequest->mIBufferClean = PluginHelper::CreateBuffer(pNodeReq, TID_MAN_FULL_PURE_YUV, INPUT);
    -    pPlgRequest->mOBufferFull  = (oBufferFullHandle == NULL) ? PluginHelper::CreateBuffer(pNodeReq, TID_MAN_FULL_YUV, OUTPUT) : oBufferFullHandle;
    +    //pPlgRequest->mOBufferFull  = (oBufferFullHandle == NULL) ? PluginHelper::CreateBuffer(pNodeReq, TID_MAN_FULL_YUV, OUTPUT) : oBufferFullHandle;
    +    pPlgRequest->mOBufferFull  = (oBufferFullHandle == NULL) ? PluginHelper::CreateBuffer(pNodeReq, TID_MAN_FULL_YUV, OUTPUT) : std::move(oBufferFullHandle);
    
         pPlgRequest->mIMetadataDynamic = PluginHelper::CreateMetadata(pNodeReq, MID_MAN_IN_P1_DYNAMIC);
         pPlgRequest->mIMetadataApp = PluginHelper::CreateMetadata(pNodeReq, MID_MAN_IN_APP);
    
    

    问题2:
    算法需要RGB数据,HAL层是YUV数据,使用openGL和各类RGB转换公式进行YUV和RGB互转后,最终照片有色差。
    问题2解决方法:
    使用libyuv进行转换,libyuv转换效率非常高,经测试,libyuv比公式法和opencv都要快,并且没有色差。android源码本身已集成libyuv,使用起来也非常方便。
    Android.mk:

    LOCAL_C_INCLUDES += $(TOP)/external/libyuv/files/include/
    LOCAL_SHARED_LIBRARIES += libyuv.vendor
    
    

    如不清楚libyuv的使用,请参考本人的另外一篇文章:YUV420转RGBA之使用libyuv

    八、结语

    现公司是家ODM公司,2018年起公司开始做AI算法,我被招来做android平台上的算法集成工作。然而我虽有几年App和Frameworks的工作经验,之前却一直是以java作为主要开发语言,C/C++语言的开发工作甚至是大学时代的事情,而部门当中会android开发的也仅有我一人。在这种情形下,重新学C/C++,并摸索着开始集成算法,真的非常具有挑战性。

    当时我们公司与某公司有合作,我们刚开始寄希望于该公司可以提供技术支援,以帮助我们完成算法集成的工作。然而,该公司不但有手机OS,也有自家的AI算法部门,在技术支援上对方不愿意配合。

    在一段时间的学习和实践之后,我也终于基本掌握了算法集成的方法和步骤。很感激部门leader L老师能够给予我如此多的耐心和支持。回过头来看,算法集成并不复杂,甚至没有任何技术原理上的东西,只是一些繁琐的步骤。于是,在项目不忙的情况下,写下本文。希望以此记录自己的心路历程,也期待可以帮助到有需要的人。本文的主要内容最初于2019年就写好了,但是由于某些原因,时至今日才整理发出来,不管怎样,但愿好文不怕晚。

    原文链接:https://www.jianshu.com/p/bf385ff1dafe

    至此,本篇已结束。转载网络的文章,小编觉得很优秀,欢迎点击阅读原文,支持原创作者,如有侵权,恳请联系小编删除,欢迎您的建议与指正。同时期待您的关注,感谢您的阅读,谢谢!

    相关文章

      网友评论

          本文标题:MTK 单帧算法集成方法

          本文链接:https://www.haomeiwen.com/subject/qkmnyltx.html