opencv +数字识别

现在很多场景需要使用的数字识别，比如银行卡识别，以及车牌识别等，在AI领域有很多图像识别算法，大多是居于opencv 或者谷歌开源的tesseract 识别.

由于公司业务需要，需要开发一个客户端程序，同时需要在xp这种老古董的机子上运行，故研究了如下几个数字识别方案：

ocr 识别的不同选择方案

• tesseract
  • 放弃：谷歌的开源tesseract ocr识别目前最新版本不支持xp系统
• 云端ocr 识别接口(不适用)
  • 费用比较贵：
  • 场景不同，我们的需求是可能毫秒级别就需要调用一次ocr 识别
• opencv
• 概念:OpenCV是一个基于BSD许可（开源）发行的跨平台计算机视觉库，可以运行在Linux、Windows、Android和Mac OS操作系统上。它轻量级而且高效——由一系列 C 函数和少量 C++ 类构成，同时提供了Python、Ruby、MATLAB等语言的接口，实现了图像处理和计算机视觉方面的很多通用算法。

以上几种ocr 识别比较，最后选择了opencv 的方式进行ocr 数字识别,下面讲解通过ocr识别的基本流程和算法.

opencv 数字识别流程及算法解析

要通过opencv 进行数字识别离不开训练库的支持，需要对目标图片进行大量的训练，才能做到精准的识别出目标数字;下面我会分别讲解图片训练的过程及识别的过程.

opencv 识别算法原理

1. 比如下面一张图片，需要从中识别出正确的数字，需要对图片进行灰度、二值化、腐蚀、膨胀、寻找数字轮廓、切割等一系列操作.

原图

image

灰度化图

image

二值化图

image

寻找轮廓

image

识别后的结果图

image

以上就是简单的图片进行灰度化、二值化、寻找数字轮廓得到的识别结果(==这是基于我之前训练过的数字模型下得到的识别结果==)
有些图片比较赋值，比如存在背景斜杠等的图片则需要一定的腐蚀或者膨胀等处理，才能寻找到正确的数字轮廓.

上面的说到我这里使用的是opencv 图像处理库进行的ocr 识别，那我这里简单介绍下C# 怎么使用opencv 图像处理看;

为了在xp上能够运行 我这里通过nuget 包引用了 OpenCvSharp-AnyCPU 第三方库,它使用的是opencv 2410 版本,你们如果不考虑xp系统的情况下开源使用最新的版本，最新版本支持了更多的识别算法.

右击你的个人项目，选择“管理Nuget程序包”。在包管理器页面中，点击“浏览”选项，然后在搜索框中键入“OpenCvSharp-AnyCPU”。选择最顶端的正确项目，并在右侧详情页中点击“安装”，等待安装完成即可。

以上的核心代码如下:

      private void runSimpleOCR(string pathName)
       {
            //构造opcvOcr 库，这里的是我单独对opencv 库进行的一次封装,加载训练库模板
            var opencvOcr = new OpencvOcr($"{path}Template\\Traindata.xml", opencvOcrConfig: new OCR.Model.OpencvOcrConfig()
            {
                ErodeLevel = 2.5,
                ThresholdType = OpenCvSharp.ThresholdType.Binary,
                ZoomLevel = 2,
            });

            var img = new Bitmap(this.txbFilaName.Text);

            var mat = img.ToMat();
            
            //核心识别方法
            var str = opencvOcr.GetText(mat, isDebug: true);
            this.labContent.Content = str;
        }

opencvOcr 的核心代码如下

        #region Constructor

        const double Thresh = 80;
        const double ThresholdMaxVal = 255;
        const int _minHeight = 35;
        bool _isDebug = false;
        CvKNearest _cvKNearest = null;
        OpencvOcrConfig _config = new OpencvOcrConfig() { ZoomLevel = 2, ErodeLevel = 3 };
        #endregion

        /// <summary>
        /// 构造函数
        /// </summary>
        /// <param name="path">训练库完整路径</param>
        /// <param name="opencvOcrConfig">OCR相关配置信息</param>
        public OpencvOcr(string path, OpencvOcrConfig opencvOcrConfig = null)
        {
            if (string.IsNullOrEmpty(path))
                throw new ArgumentNullException("path is not null");

            if (opencvOcrConfig != null)
                _config = opencvOcrConfig;

            this.LoadKnearest(path);
        }
        
        /// <summary>
        /// 加载Knn 训练库模型
        /// </summary>
        /// <param name="dataPathFile"></param>
        /// <returns></returns>
        private CvKNearest LoadKnearest(string dataPathFile)
        {
            if (_cvKNearest == null)
            {

                using (var fs = new FileStorage(dataPathFile, FileStorageMode.Read))
                {
                    var samples = fs["samples"].ReadMat();
                    var responses = fs["responses"].ReadMat();
                    this._cvKNearest = new CvKNearest();
                    this._cvKNearest.Train(samples, responses);
                }
            }
            return _cvKNearest;
        }

        /// <summary>
        /// OCR 识别,仅仅只能识别单行数字 
        /// </summary>
        /// <param name="kNearest">训练库</param>
        /// <param name="path">要识别的图片路径</param>
        public override string GetText(Mat src, bool isDebug = false)
        {
            this._isDebug = isDebug;

            #region 图片处理
            var respMat = MatProcessing(src, isDebug);
            if (respMat == null)
                return "";
            #endregion

            #region 查找轮廓
            var sortRect = FindContours(respMat.FindContoursMat);
            #endregion

            return GetText(sortRect, respMat.ResourcMat, respMat.RoiResultMat);
        }
        
         /// <summary>
        /// 查找轮廓
        /// </summary>
        /// <param name="src"></param>
        /// <returns></returns>
        private List<Rect> FindContours(Mat src)
        {
            try
            {
                #region 查找轮廓
                Point[][] contours;
                HierarchyIndex[] hierarchyIndexes;
                Cv2.FindContours(
                    src,
                    out contours,
                    out hierarchyIndexes,
                    mode: OpenCvSharp.ContourRetrieval.External,
                    method: OpenCvSharp.ContourChain.ApproxSimple);

                if (contours.Length == 0)
                    throw new NotSupportedException("Couldn't find any object in the image.");
                #endregion

                #region 单行排序（目前仅仅支持单行文字,多行文字顺序可能不对，按照x坐标进行排序）
                var sortRect = GetSortRect(contours, hierarchyIndexes);
                sortRect = sortRect.OrderBy(item => item.X).ToList();
                #endregion

                return sortRect;
            }
            catch { }

            return null;
        }
        
        /// <summary>
        /// 获得切割后的数量列表
        /// </summary>
        /// <param name="contours"></param>
        /// <param name="hierarchyIndex"></param>
        /// <returns></returns>
        private List<Rect> GetSortRect(Point[][] contours, HierarchyIndex[] hierarchyIndex)
        {
            var sortRect = new List<Rect>();

            var _contourIndex = 0;
            while ((_contourIndex >= 0))
            {
                var contour = contours[_contourIndex];
                var boundingRect = Cv2.BoundingRect(contour); //Find bounding rect for each contour

                sortRect.Add(boundingRect);
                _contourIndex = hierarchyIndex[_contourIndex].Next;
            }
            return sortRect;
        }


        /// <summary>
        /// 是否放大
        /// </summary>
        /// <param name="src"></param>
        /// <returns></returns>
        private bool IsZoom(Mat src)
        {
            if (src.Height <= _minHeight)
                return true;

            return false;
        }
        

        private List<EnumMatAlgorithmType> GetAlgoritmList(Mat src)
        {
            var result = new List<EnumMatAlgorithmType>();
            var algorithm = this._config.Algorithm;

            #region 自定义的算法
            try
            {
                if (algorithm.Contains("|"))
                {
                    result = algorithm.Split('|').ToList()
                        .Select(item => (EnumMatAlgorithmType)Convert.ToInt32(item))
                        .ToList();

                    if (!IsZoom(src))
                        result.Remove(EnumMatAlgorithmType.Zoom);

                    return result;
                }
            }
            catch { }

            #endregion

            #region 默认算法
            if (IsZoom(src))
            {
                result.Add(EnumMatAlgorithmType.Zoom);
            }
            if (this._config.ThresholdType == ThresholdType.Binary)
            {
                //result.Add(EnumMatAlgorithmType.Blur);

                result.Add(EnumMatAlgorithmType.Gray);
                result.Add(EnumMatAlgorithmType.Thresh);
                if (this._config.DilateLevel > 0)
                    result.Add(EnumMatAlgorithmType.Dilate);

                result.Add(EnumMatAlgorithmType.Erode);
                return result;
            }
            //result.Add(EnumMatAlgorithmType.Blur);

            result.Add(EnumMatAlgorithmType.Gray);
            result.Add(EnumMatAlgorithmType.Thresh);
            if (this._config.DilateLevel > 0)
                result.Add(EnumMatAlgorithmType.Dilate);

            result.Add(EnumMatAlgorithmType.Erode);
            return result;
            #endregion
        }


        /// <summary>
        /// 对查找的轮廓数据进行训练模型匹配，这里使用的是KNN 匹配算法
        /// </summary>
        private string GetText(List<Rect> sortRect, Mat source, Mat roiSource)
        {
            var response = "";
            try
            {
                if ((sortRect?.Count ?? 0) <= 0)
                    return response;

                var contourIndex = 0;
                using (var dst = new Mat(source.Rows, source.Cols, MatType.CV_8UC3, Scalar.All(0)))
                {
                    sortRect.ForEach(boundingRect =>
                    {
                        try
                        {
                            #region 绘制矩形
                            if (this._isDebug)
                            {
                                Cv2.Rectangle(source, new Point(boundingRect.X, boundingRect.Y),
                                new Point(boundingRect.X + boundingRect.Width, boundingRect.Y + boundingRect.Height),
                                new Scalar(0, 0, 255), 1);

                                Cv2.Rectangle(roiSource, new Point(boundingRect.X, boundingRect.Y),
                                   new Point(boundingRect.X + boundingRect.Width, boundingRect.Y + boundingRect.Height),
                                   new Scalar(0, 0, 255), 1);
                            }
                            #endregion

                            #region 单个ROI
                            var roi = roiSource.GetROI(boundingRect); //Crop the image
                            roi = roi.Compress();
                            var result = roi.ConvertFloat();
                            #endregion

                            #region KNN 匹配
                            var results = new Mat();
                            var neighborResponses = new Mat();
                            var dists = new Mat();
                            var detectedClass = (int)this._cvKNearest.FindNearest(result, 1, results, neighborResponses, dists);
                            var resultText = detectedClass.ToString(CultureInfo.InvariantCulture);
                            #endregion

                            #region 匹配
                            var isDraw = false;
                            if (detectedClass >= 0)
                            {
                                response += detectedClass.ToString();
                                isDraw = true;
                            }
                            if (detectedClass == -1 && !response.Contains("."))
                            {
                                response += ".";
                                resultText = ".";
                                isDraw = true;
                            }
                            #endregion

                            #region 绘制及输出切割信息库
                            try
                            {
                                //if (this._isDebug)
                                //{
                                Write(contourIndex, detectedClass, roi);
                                //}
                            }
                            catch { }

                            if (this._isDebug && isDraw)
                            {
                                Cv2.PutText(dst, resultText, new Point(boundingRect.X, boundingRect.Y + boundingRect.Height), 0, 1, new Scalar(0, 255, 0), 2);
                            }
                            #endregion

                            result?.Dispose();
                            results?.Dispose();
                            neighborResponses?.Dispose();
                            dists?.Dispose();
                            contourIndex++;
                        }
                        catch (Exception ex)
                        {
                            TextHelper.Error("GetText ex", ex);
                        }
                    });

                    #region 调试模式显示过程
                    source.IsDebugShow("Segmented Source", this._isDebug);
                    dst.IsDebugShow("Detected", this._isDebug);
                    dst.IsDebugWaitKey(this._isDebug);
                    dst.IsDebugImWrite("dest.jpg", this._isDebug);
                    #endregion
                }
            }
            catch
            {
                throw;
            }
            finally
            {
                source?.Dispose();
                roiSource?.Dispose();
            }
            return response;
        }
        
        /// <summary>
        /// 图片处理算法
        /// </summary>
        /// <param name="src"></param>
        /// <param name="isDebug"></param>
        /// <returns></returns>
        public ImageProcessModel MatProcessing(Mat src, bool isDebug = false)
        {
            src.IsDebugShow("原图", isDebug);

            var list = GetAlgoritmList(src);
            var resultMat = new Mat();
            src.CopyTo(resultMat);
            var isZoom = IsZoom(src);
            list?.ForEach(item =>
            {
                switch (item)
                {
                    case EnumMatAlgorithmType.Dilate:
                        resultMat = resultMat.ToDilate(Convert.ToInt32(this._config.DilateLevel));
                        resultMat.IsDebugShow(EnumMatAlgorithmType.Dilate.GetDescription(), isDebug);
                        break;
                    case EnumMatAlgorithmType.Erode:
                        var eroderLevel = isZoom ? this._config.ErodeLevel * this._config.ZoomLevel : this._config.ErodeLevel;
                        resultMat = resultMat.ToErode(eroderLevel);
                        resultMat.IsDebugShow(EnumMatAlgorithmType.Erode.GetDescription(), isDebug);
                        break;
                    case EnumMatAlgorithmType.Gray:
                        resultMat = resultMat.ToGrey();
                        resultMat.IsDebugShow(EnumMatAlgorithmType.Gray.GetDescription(), isDebug);
                        break;
                    case EnumMatAlgorithmType.Thresh:
                        var thresholdValue = this._config.ThresholdValue <= 0 ? resultMat.GetMeanThreshold() : this._config.ThresholdValue;
                        resultMat = resultMat.ToThreshold(thresholdValue, thresholdType: this._config.ThresholdType);
                        resultMat.IsDebugShow(EnumMatAlgorithmType.Thresh.GetDescription(), isDebug);
                        break;
                    case EnumMatAlgorithmType.Zoom:
                        resultMat = resultMat.ToZoom(this._config.ZoomLevel);
                        src = resultMat;
                        resultMat.IsDebugShow(EnumMatAlgorithmType.Zoom.GetDescription(), isDebug);
                        break;
                    case EnumMatAlgorithmType.Blur:
                        resultMat = resultMat.ToBlur();
                        src = resultMat;
                        resultMat.IsDebugShow(EnumMatAlgorithmType.Blur.GetDescription(), isDebug);
                        break;
                }
            });

            var oldThreshImage = new Mat();
            resultMat.CopyTo(oldThreshImage);

            return new ImageProcessModel()
            {
                ResourcMat = src,
                FindContoursMat = oldThreshImage,
                RoiResultMat = resultMat
            };
        }

opencv 图片处理开放出去的配置对象实体如下：

 public class OpencvOcrConfig
    {
        /// <summary>
        /// 放大程度级别 默认2
        /// </summary>
        public double ZoomLevel { set; get; }

        /// <summary>
        /// 腐蚀级别 默认2.5
        /// </summary>
        public double ErodeLevel { set; get; }

        /// <summary>
        /// 膨胀
        /// </summary>
        public double DilateLevel { set; get; }

        /// <summary>
        /// 阀值
        /// </summary>
        public double ThresholdValue { set; get; }

        /// <summary>
        /// 图片处理算法,用逗号隔开
        /// </summary>
        public string Algorithm { set; get; }

        /// <summary>
        /// 二值化方式
        /// </summary>
        public ThresholdType ThresholdType { set; get; } = ThresholdType.BinaryInv;

        /// <summary>
        /// 通道模式
        /// </summary>
        public OcrChannelTypeEnums ChannelType { set; get; } = OcrChannelTypeEnums.BlackBox;

    }

opencv 图片处理算法扩展方法如下：

 public static partial class OpenCvExtensions
    {
        private const int Thresh = 200;
        private const int ThresholdMaxVal = 255;

        /// <summary>
        /// Bitmap Convert Mat
        /// </summary>
        /// <param name="bitmap"></param>
        /// <returns></returns>
        public static Mat ToMat(this System.Drawing.Bitmap bitmap)
        {
            return OpenCvSharp.Extensions.BitmapConverter.ToMat(bitmap);
        }

        /// <summary>
        /// Bitmap Convert Mat
        /// </summary>
        /// <param name="bitmap"></param>
        /// <returns></returns>
        public static System.Drawing.Bitmap ToBitmap(this Mat mat)
        {
            return OpenCvSharp.Extensions.BitmapConverter.ToBitmap(mat);
        }


        public static bool MatIsEqual(this Mat mat1, Mat mat2)
        {
            try
            {
                if (mat1.Empty() && mat2.Empty())
                {
                    return true;
                }
                if (mat1.Cols != mat2.Cols || mat1.Rows != mat2.Rows || mat1.Dims() != mat2.Dims() ||
                    mat1.Channels() != mat2.Channels())
                {
                    return false;
                }
                if (mat1.Size() != mat2.Size() || mat1.Type() != mat2.Type())
                {
                    return false;
                }
                var nrOfElements1 = mat1.Total() * mat1.ElemSize();
                if (nrOfElements1 != mat2.Total() * mat2.ElemSize())
                    return false;

                return MatPixelEqual(mat1, mat2);
            }
            catch (Exception ex)
            {
                TextHelper.Error("MatIsEqual 异常", ex);
                return true;
            }
        }

        /// <summary>
        /// 灰度
        /// </summary>
        /// <param name="mat"></param>
        /// <returns></returns>
        public static Mat ToGrey(this Mat mat)
        {
            try
            {
                Mat grey = new Mat();
                Cv2.CvtColor(mat, grey, OpenCvSharp.ColorConversion.BgraToGray);
                return grey;
            }
            catch
            {
                return mat;
            }
        }

        /// <summary>
        /// 二值化
        /// </summary>
        /// <param name="data"></param>
        /// <returns></returns>
        public static Mat ToThreshold(this Mat data, double threshValue = 0, ThresholdType thresholdType = ThresholdType.BinaryInv)
        {
            Mat threshold = new Mat();

            if (threshValue == 0)
                threshValue = Thresh;
            Cv2.Threshold(data, threshold, threshValue, ThresholdMaxVal, thresholdType);
            if (threshold.IsBinaryInv())
            {
                Cv2.Threshold(threshold, threshold, threshValue, ThresholdMaxVal, ThresholdType.BinaryInv);
            }


            //Mat threshold = new Mat();

            //if (threshValue == 0)
            //    threshValue = Thresh;
            //Cv2.AdaptiveThreshold(data, threshold, ThresholdMaxVal,AdaptiveThresholdType.MeanC, thresholdType,3,0);
            //if (threshold.IsBinaryInv())
            //{
            //    Cv2.AdaptiveThreshold(threshold, threshold, ThresholdMaxVal, AdaptiveThresholdType.MeanC, ThresholdType.BinaryInv,3, 0);
            //}
            //Cv2.AdaptiveThreshold()
            // Threshold to find contour
            //var threshold = data.Threshold(80, 255, ThresholdType.BinaryInv);
            //Cv2.Threshold(data, threshold, Thresh, ThresholdMaxVal, ThresholdType.BinaryInv); // Threshold to find contour

            //Cv2.AdaptiveThreshold(data, threshold, 255, AdaptiveThresholdType.MeanC, ThresholdType.BinaryInv, 11, 2);

            //Cv2.Threshold(data, data, Thresh, ThresholdMaxVal, OpenCvSharp.ThresholdType.BinaryInv); // Threshold to find contour
            //Cv2.AdaptiveThreshold(data, threshold, ThresholdMaxVal, AdaptiveThresholdType.GaussianC, OpenCvSharp.ThresholdType.Binary, 3, 0); // Threshold to find contour
            //Cv2.AdaptiveThreshold(data, threshold, 255, AdaptiveThresholdType.MeanC, ThresholdType.Binary, 3, 0);
            //CvInvoke.AdaptiveThreshold(data, data, 255, Emgu.CV.CvEnum.AdaptiveThresholdType.GaussianC, Emgu.CV.CvEnum.ThresholdType.Binary, 3, 0);
            return threshold;
            //var mat = data.Threshold(100, 255,ThresholdType.Binary);
            //return mat;
        }

        /// <summary>
        /// 是否调试显示
        /// </summary>
        /// <param name="src"></param>
        /// <param name="name"></param>
        /// <param name="isDebug"></param>
        public static void IsDebugShow(this Mat src, string name, bool isDebug = false)
        {
            if (!isDebug)
                return;

            Cv2.ImShow(name, src);
        }

        public static void IsDebugWaitKey(this Mat src, bool isDebug = false)
        {
            if (!isDebug)
                return;

            Cv2.WaitKey();
        }

        public static void IsDebugImWrite(this Mat src, string path, bool isDebug = false)
        {
            if (!isDebug)
                return;

            try
            {
                Cv2.ImWrite(path, src);
            }
            catch { }
        }

        /// <summary>
        /// Mat 转成另外一种存储矩阵方式
        /// </summary>
        /// <param name="roi"></param>
        /// <returns></returns>
        public static Mat ConvertFloat(this Mat roi)
        {
            var resizedImage = new Mat();
            var resizedImageFloat = new Mat();
            Cv2.Resize(roi, resizedImage, new Size(10, 10)); //resize to 10X10
            resizedImage.ConvertTo(resizedImageFloat, MatType.CV_32FC1); //convert to float
            var result = resizedImageFloat.Reshape(1, 1);
            return result;
        }

        /// <summary>
        /// 腐蚀
        /// </summary>
        /// <param name="mat"></param>
        /// <returns></returns>
        public static Mat ToErode(this Mat mat, double level)
        {

            #region level 2.5时默认的,自动会判断是否需要腐蚀
            if (level < 1)
            {
                return mat;
            }
            if (level == 2.5)
            {
                if (!mat.IsErode())
                    return mat;
            }
            #endregion

            var erode = new Mat();

            var copyMat = new Mat();
            mat.CopyTo(copyMat);

            Cv2.Erode(mat, erode, Cv2.GetStructuringElement(StructuringElementShape.Ellipse, new Size(level, level)));
            return erode;
        }

        /// <summary>
        /// 膨胀
        /// </summary>
        /// <param name="mat"></param>
        /// <returns></returns>
        public static Mat ToDilate(this Mat mat, int level)
        {
            if (level <= 0)
                return mat;
            var dilate = new Mat();
            Cv2.Dilate(mat, dilate, Cv2.GetStructuringElement(StructuringElementShape.Ellipse, new Size(level, level)));
            return dilate;
            //return mat;
        }

        /// <summary>
        /// mat 转Roi
        /// </summary>
        /// <param name="image"></param>
        /// <param name="boundingRect"></param>
        /// <returns></returns>
        public static Mat GetROI(this Mat image, Rect boundingRect)
        {
            try
            {
                return new Mat(image, boundingRect); //Crop the image
            }
            catch
            {

            }
            return null;
        }

        /// <summary>
        /// 获取平均阀值
        /// </summary>
        /// <param name="mat"></param>
        /// <returns></returns>
        public static int GetMeanThreshold(this Mat mat)
        {
            var width = mat.Width;
            var height = mat.Height;

            var m = mat.Reshape(1, width * height);
            return (int)m.Sum() / (width * height);
        }

        /// <summary>
        /// 获得二值化阀值
        /// </summary>
        /// <param name="bitmap"></param>
        /// <returns></returns>
        public static int GetMeanThreshold(this System.Drawing.Bitmap bitmap)
        {
            using (var mat = bitmap.ToMat())
            using (var grap = mat.ToGrey())
            {
                return grap.GetMeanThreshold();
            }
        }

        public static bool IsErode(this System.Drawing.Bitmap bitmap)
        {
            using (var mat = bitmap.ToMat())
            using (var grap = mat.ToGrey())
            {

                var thresholdValue = grap.GetMeanThreshold();
                using (var threshold = grap.ToThreshold(thresholdValue, ThresholdType.BinaryInv))
                {
                    return threshold.IsErode();
                }
            }
        }

        /// <summary>
        /// 放大
        /// </summary>
        /// <param name="img"></param>
        /// <param name="times"></param>
        /// <returns></returns>
        public static Mat ToZoom(this Mat img, double times)
        {
            if (times <= 0)
                return img;
            var width = img.Width * times;
            var height = img.Height * times;

            img = img.Resize(new Size(width, height), 0, 0, Interpolation.NearestNeighbor);
            return img;
        }

        /// <summary>
        /// 均值滤波
        /// </summary>
        /// <param name="img"></param>
        /// <returns></returns>
        public static Mat ToBlur(this Mat img)
        {
            return img.Blur(new Size(3, 3));
        }

        public static Mat Compress(this Mat img)
        {
            var width = 28.0 * img.Width / img.Height;

            var fWidth = width / img.Width;
            var fHeight = 28.0 / img.Height;

            img = img.Resize(new Size(width, 28), fWidth, fHeight, Interpolation.NearestNeighbor);
            return img;
        }

        public static bool MatPixelEqual(this Mat src, Mat are)
        {
            var width = src.Width;
            var height = src.Height;
            var sum = width * height;

            for (int row = 0; row < height; row++)
            {
                for (int col = 0; col < width; col++)
                {
                    byte p = src.At<byte>(row, col); //获对应矩阵坐标的取像素
                    byte pAre = are.At<byte>(row, col);
                    if (p != pAre)
                        return false;
                }
            }
            return true;
        }

        public static int GetSumPixelCount(this Mat threshold)
        {
            var width = threshold.Width;
            var height = threshold.Height;
            var sum = width * height;

            var value = 0;
            for (int row = 0; row < height; row++)
            {
                for (int col = 0; col < width; col++)
                {
                    byte p = threshold.At<byte>(row, col); //获对应矩阵坐标的取像素
                    value++;
                }
            }
            return value;
        }

        public static int GetPixelCount(this Mat threshold, System.Drawing.Color color)
        {
            var width = threshold.Width;
            var height = threshold.Height;
            var sum = width * height;

            var value = 0;
            for (int row = 0; row < height; row++)
            {
                for (int col = 0; col < width; col++)
                {
                    byte p = threshold.At<byte>(row, col); //获对应矩阵坐标的取像素
                    if (Convert.ToInt32(p) == color.R)
                    {
                        value++;
                    }
                }
            }
            return value;
        }

        /// <summary>
        /// 是否需要二值化反转
        /// </summary>
        /// <param name="threshold"></param>
        /// <returns></returns>
        public static bool IsBinaryInv(this Mat threshold)
        {
            var width = threshold.Width;
            var height = threshold.Height;
            var sum = Convert.ToDouble(width * height);

            var black = GetPixelCount(threshold, System.Drawing.Color.Black);

            return (Convert.ToDouble(black) / sum) < 0.5;
        }

        /// <summary>
        /// 是否需要腐蚀
        /// </summary>
        /// <param name="mat"></param>
        /// <returns></returns>
        public static bool IsErode(this Mat mat)
        {
            var percent = mat.GetPercent();
            return percent >= 0.20;
        }

        /// <summary>
        /// 获得白色像素占比
        /// </summary>
        /// <param name="threshold"></param>
        /// <returns></returns>
        public static double GetPercent(this Mat threshold)
        {
            var width = threshold.Width;
            var height = threshold.Height;
            var sum = Convert.ToDouble(width * height);

            var white = GetPixelCount(threshold, System.Drawing.Color.White);
            return (Convert.ToDouble(white) / sum);
        }

        /// <summary>
        /// 根据模板查找目标图片的在原图标中的开始位置坐标
        /// </summary>
        /// <param name="source"></param>
        /// <param name="template"></param>
        /// <param name="matchTemplateMethod"></param>
        /// <returns></returns>
        public static Point FindTemplate(this Mat source, Mat template, MatchTemplateMethod matchTemplateMethod = MatchTemplateMethod.SqDiffNormed)
        {
            if (source == null)
                return new OpenCvSharp.CPlusPlus.Point();

            var result = new Mat();
            Cv2.MatchTemplate(source, template, result, matchTemplateMethod);

            Cv2.MinMaxLoc(result, out OpenCvSharp.CPlusPlus.Point minVal, out OpenCvSharp.CPlusPlus.Point maxVal);

            var topLeft = new OpenCvSharp.CPlusPlus.Point();
            if (matchTemplateMethod == MatchTemplateMethod.SqDiff || matchTemplateMethod == MatchTemplateMethod.SqDiffNormed)
            {
                topLeft = minVal;
            }
            else
            {
                topLeft = maxVal;
            }
            return topLeft;
        }
    }

以上代码中开源对图片进行轮廓切割，同时会生成切割后的图片代码如下

#region 绘制及输出切割信息库
    try
    {

        Write(contourIndex, detectedClass, roi);

    }
    catch { }
#endregion

private void Write(int contourIndex, int detectedClass, Mat roi)
{
    Task.Factory.StartNew(() =>
    {
        try
        {
            var templatePath = $"{AppDomain.CurrentDomain.BaseDirectory}template";
            FileHelper.CreateDirectory(templatePath);
            var templatePathFile = $"{templatePath}/{contourIndex}_{detectedClass.ToString()}.png";
            Cv2.ImWrite(templatePathFile, roi);
            if (!roi.IsDisposed)
            {
                roi.Dispose();
            }
        }
        catch {}
   });
}

切割后的图片如下：

image

这里我已经对数字进行切割好了，接下来就是需要对0-9 这些数字进行分类(建立文件夹进行数字归类)，如下：

image

图中的每一个分类都是我事先切割好的数字图片,图中有-1 和-2 这两个特殊分类，-1 里面我是放的是“.”好的分类，用于训练“.”的图片，这样就可以识别出小数点的数字支持.
-2 这个分类主要是其他一些无关紧要的图片，也就是不是数字和点的都归为这一类中.

现在训练库分类已经建立好了，接下来我们需要对这些分类数字进行归一化处理，生成训练模型. 代码如下：

        private void Button_Click_1(object sender, RoutedEventArgs e)
        {
            var opencvOcr = new OpencvOcr($"{path}Template\\Traindata.xml", opencvOcrConfig: null);
            opencvOcr.Save($"{path}Template\\NumberWrite", outputPath: $"{path}Template\\Traindata.xml");
            MessageBox.Show("生成训练库成功");
            //var img = new Bitmap(this.txbFilaName.Text);

            //var str = opencvOcr.GetText(img.ToMat(), isDebug: true);
            //this.labContent.Content = str;
        }
        
        /// <summary>
        /// 保存训练模型
        /// </summary>
        /// <param name="dataPath"></param>
        /// <param name="trainExt"></param>
        /// <param name="dataPathFile"></param>
        public void Save(string dataPath, string trainExt = "*.png", string outputPath = "")
        {
            if (string.IsNullOrEmpty(outputPath))
                throw new ArgumentNullException("save dataPath is not null");

            var trainingImages = this.ReadTrainingImages(dataPath, trainExt);
            var samples = GetSamples(trainingImages);
            var response = GetResponse(trainingImages);

            //写入到训练库中
            using (var fs = new FileStorage(outputPath, FileStorageMode.WriteText))
            {
                fs.Write("samples", samples);
                fs.Write("responses", response);
            }
        }

        /// <summary>
        /// 根据目录加载文件
        /// </summary>
        /// <param name="path"></param>
        /// <param name="ext"></param>
        /// <returns></returns>
        private IList<ImageInfo> ReadTrainingImages(string path, string ext)
        {
            var images = new List<ImageInfo>();
            var imageId = 1;
            foreach (var dir in new DirectoryInfo(path).GetDirectories())
            {
                var groupId = int.Parse(dir.Name);
                foreach (var imageFile in dir.GetFiles(ext))
                {
                    var srcMat = new Mat(imageFile.FullName, OpenCvSharp.LoadMode.GrayScale);
                    var image = srcMat.ConvertFloat();
                    if (image == null)
                    {
                        continue;
                    }

                    images.Add(new ImageInfo
                    {
                        Image = image,
                        ImageId = imageId++,
                        ImageGroupId = groupId
                    });
                }
            }
            return images;
        }
        
        /// <summary>
        /// Mat 转成另外一种存储矩阵方式
        /// </summary>
        /// <param name="roi"></param>
        /// <returns></returns>
        public static Mat ConvertFloat(this Mat roi)
        {
            var resizedImage = new Mat();
            var resizedImageFloat = new Mat();
            Cv2.Resize(roi, resizedImage, new Size(10, 10)); //resize to 10X10
            resizedImage.ConvertTo(resizedImageFloat, MatType.CV_32FC1); //convert to float
            var result = resizedImageFloat.Reshape(1, 1);
            return result;
        }
        
        /// <summary>
        /// 获取Samples
        /// </summary>
        /// <param name="trainingImages"></param>
        /// <returns></returns>
        private Mat GetSamples(IList<ImageInfo> trainingImages)
        {
            var samples = new Mat();
            foreach (var trainingImage in trainingImages)
            {
                samples.PushBack(trainingImage.Image);
            }
            return samples;
        }
        
        private Mat GetResponse(IList<ImageInfo> trainingImages)
        {
            var labels = trainingImages.Select(x => x.ImageGroupId).ToArray();
            var responses = new Mat(labels.Length, 1, MatType.CV_32SC1, labels);
            var tmp = responses.Reshape(1, 1); //make continuous
            var responseFloat = new Mat();
            tmp.ConvertTo(responseFloat, MatType.CV_32FC1); // Convert  to float

            return responses;
        }

到这里ocr 训练模型以及建立好了，会在目录中生成一个Traindata.xml 的训练模型库,我们来打开这个训练模型库文件探索它的神秘的容颜.

image
image

到这里opencv + 数字识别分享已经完成，它的神秘面纱也就到此结束了
到这里opencv + 数字识别分享已经完成，它的神秘面纱也就到此结束了
欢迎各位大佬关注公众号

dotNET 博士