Logo Recognition System

• Download Logo Recognition System source code - 518.4 KB

Introduction

In this article, I will present my program (Logo Recognition System) and show you the mechanism of the program by giving a step by step explanation of the code.

Note: To be able to understand the article, you should have general knowledge of the basics of deep learning, such as the components of an artificial neural network, such as the layer or the activation function you should also have advanced C# knowledge.

Before we get started, we need to install the required libraries. The following libraries are required to get our program working:

• Tensorflow.net and Tensorflow.Keras

  Using the Paket-Manager-Console, enter the following commands:

  1. NuGet\Install-Package TensorFlow.NET -Version 0.100.2
  2. NuGet\Install-Package TensorFlow.Keras -Version 0.10.2
  3. NuGet\Install-Package SciSharp.TensorFlow.Redist -Version 2.10.0
  4. NuGet\Install-Package SciSharp.TensorFlow.Redist-Windows-GPU -Version 2.10.0
• Emgu cv

  Using the Paket-Manager-Console, enter the following commands:

  1. NuGet\Install-Package EmguCV -Version 3.1.0.1
  2. NuGet\Install-Package Emgu.CV.runtime.windows -Version 4.6.0.5131
  3. NuGet\Install-Package Emgu.CV.Bitmap -Version 4.6.0.5131
• ScottPlot.WinForms

  Using the Paket-Manager-Console, enter the following command:

  1. NuGet\Install-Package ScottPlot.WinForms -Version 4.1.60
• Newtonsoft.Json

  Using the Paket-Manager-Console, enter the following command:

  1. NuGet\Install-Package Newtonsoft.Json -Version 13.0.2

Note: After we installed the required libraries, we need also to set the target build platform from any CPU to 64 bit as the following:

Project->Properties->Build->General->Platform Target-> x64

Part 1: Create Our Own Convolution Neural Network Model

Since the convolution layer is specialized in image recognition and since we mainly want to classify images, we will use the convolution layer in our network and to speed up the learning process, we will also use the "maxpooling" layer and last but not least and to eliminate overfitting and to avoid a neural network being overly dependent on a particular object property, we will use the "droupout"  layer in our network. As the final step in our cnn, we will use the "Dense" Layer which is a common classifier for neural networks. In this layer, every node is connected to every node in the previous layer. As activation function, we will use the Relu function in the hidden layers to speed up the training and the softmax function in the output layer because we will have more than two outputs.

Step by Step Network Structure Implementation

1. The network input is a 32 x 32 image, which means that the input size should be 32 x 32 with 3 RGB channels
2. x2 convolutional layers with 32 filter and with filter size as 3x3
   Note 1: To avoid reducing layer output size, we can set the padding property to "same".
3. maxpooling layer with size as 2x2
4. droupout layer with 0.25 rate
   Note 2: As a parameter, we can give this layer a value between 0 and 1, for example 0.25 means that when the network is trained, a predetermined number (25%) of neurons in a certain layer of the network will be switched off ("dropout") and become not processed for the next calculation step.
5. We repeat the 2, 3, 4 steps two times and in each time, we double the number of the filters count in the convolutional layer
6. Dense layer with 256 neuron
7. Dense layer with 128 neuron
8. As output, we will also use the dense layer with softmax as activation function:

public class CNN
{
    ILayer Conv2D_1_;
    ILayer Conv2D_2_;
    ILayer MaxPooling2D_1_;
    ILayer Dropout_1_;
    ILayer Conv2D_3_;
    ILayer Conv2D_4_;
    ILayer MaxPooling2D_2_;
    ILayer Dropout_2_;
    ILayer Conv2D_5_;
    ILayer Conv2D_6_;
    ILayer MaxPooling2D_3_;
    ILayer Dropout_3_;
    ILayer Flatten;
    ILayer Dense_1_;
    ILayer Dense_2_;
    ILayer Dense_3_;

    int outputCount;

    public CNN(int outputCount) {

        var layers = new LayersApi();

        Conv2D_1_ = layers.Conv2D(32, (3, 3), activation: "relu", padding: "same");

        Conv2D_2_ = layers.Conv2D(32, (3, 3), activation: "relu", padding: "same");

        Conv2D_3_ = layers.Conv2D(64, (3, 3), activation: "relu", padding: "same");

        Conv2D_4_ = layers.Conv2D(64, (3, 3), activation: "relu", padding: "same");

        Conv2D_5_ = layers.Conv2D(128, (3, 3), activation: "relu", padding: "same");

        Conv2D_6_ = layers.Conv2D(128, (3, 3), activation: "relu", padding: "same");

        MaxPooling2D_1_ = layers.MaxPooling2D(pool_size: (2, 2));

        MaxPooling2D_2_ = layers.MaxPooling2D(pool_size: (2, 2));

        MaxPooling2D_3_ = layers.MaxPooling2D(pool_size: (2, 2));

        Dropout_1_ = layers.Dropout(0.25f);

        Dropout_2_ = layers.Dropout(0.25f);

        Dropout_3_ = layers.Dropout(0.25f);

        Flatten = layers.Flatten();

        Dense_1_ = layers.Dense(256, activation: "relu");

        Dense_2_ = layers.Dense(128, activation: "relu");

        Dense_3_ = layers.Dense(outputCount, activation: "softmax");
    }

    public Model Build(string name)
    {
        var inputs = keras.Input(shape: (32, 32, 3), name: "img_"+name);

        var  x = Conv2D_1_.Apply(inputs);
            x = Conv2D_2_.Apply(x);
            x = MaxPooling2D_1_.Apply(x);
            x = Dropout_1_.Apply(x);
            x = Conv2D_3_.Apply(x);
            x = Conv2D_4_.Apply(x);
            x = MaxPooling2D_2_.Apply(x);
            x = Dropout_2_.Apply(x);
            x = Conv2D_5_.Apply(x);
            x = Conv2D_6_.Apply(x);
            x = MaxPooling2D_3_.Apply(x);
            x = Dropout_3_.Apply(x);
            x = Flatten.Apply(x);
            x = Dense_1_.Apply(x);
            x = Dense_2_.Apply(x);
       
        return keras.Model(inputs, Dense_3_.Apply(x), name: name);
    }
}

Part 2: The Program

To fully understand how to use the program, we need to know how it works in general so the program contains three parts; the first one where we prepare our own data set, the second part which represents the training process where we train our own model and the last part where we test our Model on realistic Objects.

1. Prepare Our Own Data Set

   For this part, I have developed a Snipping Image Tool by which we can do two things:

   • The first thing is to Select an Object Manually where in this process, the Extracted Object will be exported as either a new Output and here a new Label in the Ideal Set Combo-box will be added or the Extracted Object can be set to existing Label in the Ideal Set Combobox, and that will be done as in the following steps in the order:

     C#

     // Doses the Selected Object Already exists in the Data Set
     DialogResult res = CustomMsgbox.InputBox.ShowDialog
           ("represent the  Image a new Dataset  ?", "Question",
                                  CustomMsgbox.InputBox.Icon.Question,
                                  CustomMsgbox.InputBox.Buttons.YesNo);
     // if the Selected Object New or doesn´t exists in the Data Set,
     // we have to enter the new Ideal set Value
     if (res == System.Windows.Forms.DialogResult.Yes)
     {
         res = CustomMsgbox.InputBox.ShowDialog
         ("Please Enter the Ideal set value !!", "Ideal set",
         CustomMsgbox.InputBox.Icon.Question,
         CustomMsgbox.InputBox.Buttons.OkCancel,
         CustomMsgbox.InputBox.Type.TextBox,
         new string[] { "Ideal set new Value" });
         
         if (res == System.Windows.Forms.DialogResult.OK)
         {
             Bitmap image = ImageHelper.CutImage(rect, new Bitmap
                            (this.docu_pb.Image));
             
             // Filter Selected Object Colors on the basis of Program Settings
             
             switch (Central_Static_Value.Settings.Filter)
             {
                 case "Colored": image = ImageHelper.BGRFiler(image); break;
                 case "Gray": image = ImageHelper.GrayFilter(image); break;
                 case "BlackWeight": image = 
                       ImageHelper.BlackWhiteFilter(image); break;
                 default: image = ImageHelper.BGRFiler(image); break;
             }
             
             // get Ideal Set Value from the Message Box
             string InputBox_ResultValue = CustomMsgbox.InputBox.ResultValue;
             
             // save the Ideal set Value as new Output
             int match = this.AssignIdentity(InputBox_ResultValue);
             
             //give the Image a Name to show in to Train Images dataGridView,
             //Note that this Name does not play any role 
             //in the Actual training or Predict
             
             string imageName = "image" + 
             Central_Static_Value.Train_Model.
             to_Train_Images_dataGridView.Rows.Count.ToString();
             
             // add the Image in the to Train Images dataGridView
             
             Central_Static_Value.Train_Model.
                     to_Train_Images_dataGridView.Rows.Add
             (imageName, InputBox_ResultValue, 
                         ImageHelper.ResizeImage(32, 32, image));
             
             // Create a New Dataset Instance                                
             TrainingSet trainingSet = 
             new TrainingSet(imageName, match, 
                 ImageHelper.ResizeImage(32, 32, image), true);
             
             // add the Dataset in the TrainingSetList                                
             Central_Static_Value.Train_Model.TrainingSetList.Add(trainingSet);
             
             // refresh the Ideal set Combobox  
             refrech_ideal_set_comboBox();
         }
     }
     // if the Selected Object already exists in the Data Set, 
     // we have to choose one of the Available Ideal Set Values
     
     else
     {
         res = CustomMsgbox.InputBox.ShowDialog
         ("Please choose the Ideal set value !!", "Ideal set",
         CustomMsgbox.InputBox.Icon.Question,
         CustomMsgbox.InputBox.Buttons.OkCancel,
         CustomMsgbox.InputBox.Type.ComboBox,
         Central_Static_Value.Train_Model.neuron2identity.Select
                              (x => x.Value).ToArray());
         
         if (res == DialogResult.OK)
         {
             // get Ideal Set Value from the Message Box
             string result = CustomMsgbox.InputBox.ResultValue;
             
             //Cut the selected Area of the given Image
             Bitmap image = ImageHelper.CutImage
                            (rect, new Bitmap(this.docu_pb.Image));
             
             //give the Image a Name to show in to Train Images dataGridView,
             //Note that this Name does not play any role in 
             //the Actual training or Predict
             
             string imageName = "image" + 
             Central_Static_Value.Train_Model.
                    to_Train_Images_dataGridView.Rows.Count.ToString();
             
             // save the Ideal set Value as new Output
             int match = AssignIdentity(result);
             
             // Filter Selected Object Colors on the basis of Program Settings
             switch (Central_Static_Value.Settings.Filter)
             {
                 case "Colored": image = ImageHelper.BGRFiler(image); break;
                 case "Gray": image = ImageHelper.GrayFilter(image); break;
                 case "BlackWeight": image = 
                       ImageHelper.BlackWhiteFilter(image); break;
                 default: image = ImageHelper.BGRFiler(image); break;
             }
             
             // add the Image in the to Train Images dataGridView
             Central_Static_Value.Train_Model.to_Train_Images_
                        dataGridView.Rows.Add
                       (imageName, result, ImageHelper.ResizeImage
                       (32, 32, image));
                       
             // Create a New Dataset Instance
             TrainingSet trainingSet = 
             new TrainingSet(imageName, match, 
                 ImageHelper.ResizeImage(32, 32, image), false);
             
             // add the Dataset in the TrainingSetList 
             Central_Static_Value.Train_Model.TrainingSetList.Add(trainingSet);
         }
     }

     1. After we use the Mouse, select a new Object. The program will ask us whether the selected object already exists in the dataset or not.

     2. If the Selected Object is New or doesn't exist in the Data Set  we have to enter the new Ideal set Value. Other case we have to choose one of the Available Ideal Set Values.

     3. After we entered the Ideal Set Value, we need to save it into a Dictionary where the Key in the Dictionary represents the equivalent Current Output Count and the Value in the Dictionary represents the actual Ideal Set Value. Otherwise, we need to refresh the Output count Variable by incrementing it by one. All that, of course, if the Ideal set is New, refresh TrainingsetList by adding the New Dataset and the code:

     4. The second thing is to Select Objects automatically and that will be done using the Selective Search Algorithm) where in this process, the Extracted Object can be just set to existing Label in the Ideal Set Combo-box in the Snip Image Form so after we get one of them, using selective search automatic selected, object we take Ideal Set Value from the ideal_set_cb and save it as new Output.

        The code:

        C#

        //get the Image Name ,to show it into Train Images dataGridView,
        //Note that this Name does not play any role 
        //in the Actual training or Predict
        string imageName = Image_name_tb.Text;
        
        // get Ideal Set Value from the ideal_set_cb
        string selecteditem = ideal_set_cb.SelectedItem.ToString();
        
        // save the Ideal set Value as new Output
        int match = AssignIdentity(selecteditem);
        
        // add the Image in the to Train Images dataGridView
        Central_Static_Value.Train_Model.to_Train_Images_dataGridView.Rows.Add
        (imageName, selecteditem, ImageHelper.ResizeImage
        (32, 32, new Bitmap(sniped_img_pictureBox.Image)));
        
        // Create a New Dataset Instance
        TrainingSet trainingSet = new TrainingSet
        (imageName, match, ImageHelper.ResizeImage
        (32, 32, new Bitmap(sniped_img_pictureBox.Image)), false);
        
        // add the Dataset in the TrainingSetList
        Central_Static_Value.Train_Model.TrainingSetList.Add(trainingSet);

2. Second Part "Training Process"

   After the Dataset is prepared, the Training Process can be started as in the following steps:

   C#

   // Convert Trainigset Images List into Int 4 Dimensional ND Array 
   NDArray x_train = ImageHelper.ImagesToNDArray(this.TrainingSetList, 32, 32);
                
   // Convert Trainigset Idealsets or Matches in Int 2 Dimensional ND Array
   NDArray y_train = ImageHelper.IdelaValuesToNDArray
                     (this.TrainingSetList, this.outputCount);
   
   //Convert the Input Matrix Type Into float
   x_train = x_train.astype(np.float32);
   
   x_train /= 255;
   
   Model = new CNN(outputCount).Build("LogoDetector");
   
   var opt = new Tensorflow.Keras.Optimizers.Adam(learning_rate: 0.001f);
   //   Model.summary();
   
   Model.compile(optimizer: opt,
       loss: keras.losses.CategoricalCrossentropy(),
       metrics: new[] { "acc" });
       for (int i = 0; i < (int)Epoches_numericUpDown.Value; i++)
       {
           Model.fit(x_train, y_train);
    }

   1. We already know that our network accepts a Size of (32x32) as Input so firstly and during the Prepare Dataset Process, we have to resize our image to (32x32) after that in the Training Process, we have to convert our images into Int ND Array so when we Convert our Image, we get an 1x32x32x3 Array, the first 1 represents the Image Index and the last 3 mean that we have 3 colors in each Pixel and now, let's imagine we have more than one image to train, so we need a ND Array like (Variable-Images-Count x 32 x 32 x 3) and the code:

      C#

       /// <summary>
       /// Convert List of Images into four dimensional ND Array where the
       /// first dímension represent the Image Index and the Second  the
       /// Image width and the third the Imgae Height and the Last
       /// the Image Colors
       /// </summary>
       /// <param name="bitmaps">to be Converted Image List</param>
       /// <param name="width">Images Width</param>
       /// <param name="height">Images Height</param>
       /// <returns>4 Dimensional ND Array</returns>
       public static NDArray ImagesToNDArray(List<TrainingSet> bitmaps, 
              int width, int height)
       {
           int[,,,] imagespixels = new int[bitmaps.Count, width, height, 3];
           
           for (int bi = 0; bi < bitmaps.Count; bi++)
           {
               Bitmap bitmap = bitmaps[bi].Image;
               
               int[,,] pixels = getPixels(bitmap);
               
               for (int x = 0; x < width; x++)
               {
                   for (int y = 0; y < height; y++)
                   {
                       for (int z = 0; z < 3; z++)
                       {
                           imagespixels[bi, x, y, z] = pixels[x, y, z];
                       }
                   }
               }
           }
           return np.array(imagespixels);
      }

   2. After we convert our Image into ND Array, we need also to convert our Ideal Dataset Values or the Matches Values into 2d Nd Array where the First Dimension represent the Equivalent Image Index and the second Dimension represent the Match value or the Ideal Output and the code:

      C#

        /// <summary>
        ///the Idealset or the right Predict represented in 1 
        ///for identical and 0 for not identical
        ///will here converted into two dimensional Array,
        ///where the first dimension represent the Image Index
        ///and the second the output or the Idealset
        ///</summary>
        /// <param name="images">the Ideal Sets of the Images List</param>
        /// <param name="outputCount">Count of the CNN Outputs</param>
        /// <returns>2 Dimensional ND Array</returns>
        
        public static NDArray IdelaValuesToNDArray
               (List<TrainingSet> images,int outputCount)
        {
        
            int[,] idealvalues = new int[images.Count, outputCount];
            
            for (int i = 0; i < images.Count; i++)
            {
                for (int j = 0; j < outputCount; j++)
                {
                
                    if (j == images[i].Match)
                        idealvalues[i, j] = 1;
                        
                    else idealvalues[i, j] = 0;
                }
            }
            
            return np.array(idealvalues);
      }

   3. To improve our training result, we divide our data set or our 4D dimensional ND Array by 255, the result will be a matrix with values between 0 and 1.

   4. To speed up our learning process and save memory requirements, we will use the adam optimizer.

   5. After that, the learning process can be started, and the final code:

3. Part 3, "Testing Process"

   • The program needs firstly a selective search algorithm, this algorithm will select for us all object in the Image and prepare them for the next step namely to Predict what for object it is:

   • The Predict Process: In this step, the Program will tell us what for object it is in the Image for that the Program will convert the image into 3 dimensional ND Array, the first dimension represent the width and the second the height and the last the RGB or the Colors. After that, we will give the array to our CNN to process it and at last to know what for object it is.

Note

In case we try to Predict images from outside the range of the dataset on which the model has been trained, there is a possibility that false predictions will be generated. Here, I would like to point out that the model must be trained on all images that have been set through the selective search algorithm, because only in this case will the program return correct predictions.

In the following video, I will show you how to use the program:

History

• 19^th February, 2023: Initial version
• 28^th March, 2023: Article updated