Butterfly detector using CNN

Using CNN transfer learning to detect 11 different species of southern mediterranean butterflies.

<p dir="auto"><img src="https://images.hive.blog/768x0/https://files.peakd.com/file/peakd-hive/macrodrigues/23wWyFs5wbWiXjMsCuKCBVEeD2qWbogcaWBRCBRmihSNkwYsHdjsjaHrox6G4qMx1VrCJ.JPG" alt="capa butterfly detector.JPG" srcset="https://images.hive.blog/768x0/https://files.peakd.com/file/peakd-hive/macrodrigues/23wWyFs5wbWiXjMsCuKCBVEeD2qWbogcaWBRCBRmihSNkwYsHdjsjaHrox6G4qMx1VrCJ.JPG 1x, https://images.hive.blog/1536x0/https://files.peakd.com/file/peakd-hive/macrodrigues/23wWyFs5wbWiXjMsCuKCBVEeD2qWbogcaWBRCBRmihSNkwYsHdjsjaHrox6G4qMx1VrCJ.JPG 2x" /> <h2>Data Collection <p dir="auto"><span>The images used to train the deep learning model were taken from several websites, nevertheless <a href="https://www.gbif.org/" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://www.gbif.org/ seemed to be the elected choice for most of the pictures. <p dir="auto">In total, the training data has 200 images from each butterfly. And the species are the following: <p dir="auto"><img src="https://images.hive.blog/768x0/https://files.peakd.com/file/peakd-hive/macrodrigues/23vi6K5Mo9heyCvMMRAJPYyGAvZ6qBTJNdy1XH25qJcNw1eXUM5r2DN9tHLLQz64cKjSp.png" alt="image.png" srcset="https://images.hive.blog/768x0/https://files.peakd.com/file/peakd-hive/macrodrigues/23vi6K5Mo9heyCvMMRAJPYyGAvZ6qBTJNdy1XH25qJcNw1eXUM5r2DN9tHLLQz64cKjSp.png 1x, https://images.hive.blog/1536x0/https://files.peakd.com/file/peakd-hive/macrodrigues/23vi6K5Mo9heyCvMMRAJPYyGAvZ6qBTJNdy1XH25qJcNw1eXUM5r2DN9tHLLQz64cKjSp.png 2x" /> <p dir="auto">In order to reduce the computational cost and rapidly copy the images to the cloud, I used Microsoft PowerToys to reduce the image size beforehand. <h2>In-place Data Augmentation <p dir="auto">This type of data augmentation is implemented by Keras' <code>ImageDataGenerator, when implemented it sees new variations of the data at each and every epoch. <pre><code>imgs_gen = ImageDataGenerator(rotation_range=20, width_shift_range=0.3, height_shift_range=0.3, rescale=1/255, shear_range=0.3, zoom_range=0.3, horizontal_flip=True, validation_split=0.3, fill_mode='nearest') train_gen = imgs_gen.flow_from_directory(f"{PATH}/raw_data/train", class_mode = 'categorical', target_size=(150, 150), subset='training', batch_size=16) validation_gen = imgs_gen.flow_from_directory(f"{PATH}/raw_data/train", target_size=(150, 150), batch_size = 16, subset='validation', class_mode = 'categorical') <h2>Transfer Learning <p dir="auto">I have tried different models for the multiclass image classification: <ul> <li>VGG16 <li>VGG19 <li>Inceptionv3 <li>Xception <p dir="auto">Both <code>Inceptionv3 and <code>Xception are very fast models and less time consuming, however until now <code>VGG16 and <code>VGG19 gave the best results. All these models and others not mentioned are still being tested in order to optimize the accuracy. Since I'm using a free version of Google Colab, I'm still limited in terms of GPU use, and the optimization process takes longer. See below the model's compilation for VGG16: <pre><code>def compile_model(): """Uses VGG16 for transfer learning and applies additional layers. Then it compiles the model for a multiclass classification problem.""" base_model = VGG16(weights="imagenet", include_top=False, input_shape=(150, 150, 3)) flattening_layer = layers.Flatten() dense_layer_1 = layers.Dense(50, activation = 'relu') dense_layer_2 = layers.Dense(20, activation = 'relu') prediction_layer = layers.Dense(12, activation = 'softmax') model = models.Sequential([ base_model, flattening_layer, dense_layer_1, dense_layer_2, prediction_layer ]) <h2>Results <p dir="auto">After training the model and saving it, I tested the overall accuracy on several test images. <pre><code>def check_accuracy(pick_model='model_test'): """" Check accuracy for each one of the test elements. And ouputs the final yield""" list_test_elements = os.listdir(PATH_TEST) butterflies = list(train_gen.class_indices.keys()) model = models.load_model(f"{PATH}/butterfly_detector/models/{pick_model}") count = 0 for i in list_test_elements: img_file = f"{PATH_TEST}/{i}" img = image.load_img(img_file, target_size=(150, 150)) img_arr = image.img_to_array(img) img_arr = np.expand_dims(img_arr, axis=0) pred = list(model.predict(img_arr)[0]) preds = dict(zip(butterflies, pred)) new_preds = preds.copy() new_vals = [] new_keys = [] for key, value in preds.items(): if value != 0.0: new_vals.append(value) new_keys.append(key) new_preds = dict(zip(new_keys, new_vals)) print('TITLE:', i.split('_')[0]) print('PREDICTIONS:', new_preds) print('BEST PREDICTION:', max(preds, key=preds.get)) if max(preds, key=preds.get) == i.split('_')[0]: count += 1 print('RESULT', 'Good prediction!') print('\n') else: print('RESULT', 'Wrong Prediction!') print('\n') print(f"This model is {round((count)*100/len(list_test_elements), 2)}% efective.") <p dir="auto">This is the output: <pre><code>TITLE: Colias Croceus PREDICTIONS: {'Colias Croceus': 1.0} BEST PREDICTION: Colias Croceus RESULT Good prediction! TITLE: Maniola Jurtina PREDICTIONS: {'Maniola Jurtina': 1.0} BEST PREDICTION: Maniola Jurtina RESULT Good prediction! TITLE: Iphiclides Feisthamelii PREDICTIONS: {'Iphiclides Feisthamelii': 1.0} BEST PREDICTION: Iphiclides Feisthamelii RESULT Good prediction! TITLE: Aricia Cramera PREDICTIONS: {'Aricia Cramera': 1.0} BEST PREDICTION: Aricia Cramera RESULT Good prediction! TITLE: Coenonympha Pamphilus PREDICTIONS: {'Coenonympha Pamphilus': 1.0} BEST PREDICTION: Coenonympha Pamphilus RESULT Good prediction! TITLE: Maniola Jurtina PREDICTIONS: {'Coenonympha Pamphilus': 1.0} BEST PREDICTION: Coenonympha Pamphilus RESULT Wrong Prediction! TITLE: Lysandra Bellargus PREDICTIONS: {'Lysandra Bellargus': 1.0} BEST PREDICTION: Lysandra Bellargus RESULT Good prediction! TITLE: Issoria Lathonia PREDICTIONS: {'Issoria Lathonia': 1.0} BEST PREDICTION: Issoria Lathonia RESULT Good prediction! TITLE: Issoria Lathonia PREDICTIONS: {'Issoria Lathonia': 1.0} BEST PREDICTION: Issoria Lathonia RESULT Good prediction! TITLE: Anthocharis Cardamines PREDICTIONS: {'Anthocharis Cardamines': 1.0} BEST PREDICTION: Anthocharis Cardamines RESULT Good prediction! TITLE: Aricia Cramera PREDICTIONS: {'Vanessa Atalanta': 1.0} BEST PREDICTION: Vanessa Atalanta RESULT Wrong Prediction! TITLE: Anthocharis Cardamines PREDICTIONS: {'Anthocharis Cardamines': 1.0} BEST PREDICTION: Anthocharis Cardamines RESULT Good prediction! TITLE: Gonepteryx Cleopatra PREDICTIONS: {'Gonepteryx Cleopatra': 1.0} BEST PREDICTION: Gonepteryx Cleopatra RESULT Good prediction! TITLE: Coenonympha Pamphilus PREDICTIONS: {'Coenonympha Pamphilus': 1.0} BEST PREDICTION: Coenonympha Pamphilus RESULT Good prediction! TITLE: Vanessa Atalanta PREDICTIONS: {'Vanessa Atalanta': 1.0} BEST PREDICTION: Vanessa Atalanta RESULT Good prediction! TITLE: Lysandra Bellargus PREDICTIONS: {'Issoria Lathonia': 1.0} BEST PREDICTION: Issoria Lathonia RESULT Wrong Prediction! TITLE: Melanargia Ines PREDICTIONS: {'Melanargia Ines': 1.0} BEST PREDICTION: Melanargia Ines RESULT Good prediction! TITLE: Melanargia Ines PREDICTIONS: {'Melanargia Ines': 1.0} BEST PREDICTION: Melanargia Ines RESULT Good prediction! TITLE: Colias Croceus PREDICTIONS: {'Colias Croceus': 1.0} BEST PREDICTION: Colias Croceus RESULT Good prediction! TITLE: Vanessa Atalanta PREDICTIONS: {'Vanessa Atalanta': 1.0} BEST PREDICTION: Vanessa Atalanta RESULT Good prediction! TITLE: Gonepteryx Cleopatra PREDICTIONS: {'Gonepteryx Cleopatra': 1.0} BEST PREDICTION: Gonepteryx Cleopatra RESULT Good prediction! TITLE: Iphiclides Feisthamelii PREDICTIONS: {'Melanargia Ines': 1.0} BEST PREDICTION: Melanargia Ines RESULT Wrong Prediction! This model is 81.82% efective. <h2>The app <p dir="auto">For the front end I decided to make a GUI using <code>Kivy. First I tested the script using the computer's webcam, later on I'm planning to implement it on the smartphone, so it can be able to detect butterflies in the field. <p dir="auto"><img src="https://images.hive.blog/768x0/https://files.peakd.com/file/peakd-hive/macrodrigues/23xVChiuCEZMCMkCA26tVDCLsbVxhxaZibBdEMoswejFJFwEZJLfrLs71MhYiyFMz53Dv.png" alt="image.png" srcset="https://images.hive.blog/768x0/https://files.peakd.com/file/peakd-hive/macrodrigues/23xVChiuCEZMCMkCA26tVDCLsbVxhxaZibBdEMoswejFJFwEZJLfrLs71MhYiyFMz53Dv.png 1x, https://images.hive.blog/1536x0/https://files.peakd.com/file/peakd-hive/macrodrigues/23xVChiuCEZMCMkCA26tVDCLsbVxhxaZibBdEMoswejFJFwEZJLfrLs71MhYiyFMz53Dv.png 2x" /> <p dir="auto">For the App's full code and the rest of the project script, please feel free to take a look into my GitHub: <p dir="auto"><center><span>🚀 <a href="https://github.com/macrodrigues" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://github.com/macrodrigues 🚀