1. Image Collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "A notebook to show image collection process. And label using [Label Studio](https://labelstud.io). To use for TensorFlow Object Detection API (TFOD)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 1. Import Packages"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "# cv2 is for opencv, which is used for image operations\n",
    "# such as reading and processing images\n",
    "import cv2\n",
    "import os\n",
    "import time\n",
    "import shutil"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 2. Define Specifications for Images to Collect"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "# defining the class names for the dataset, \n",
    "CLASS_NAMES = ['coin']\n",
    "# define the number of images to be collected for each class\n",
    "IMAGES_PER_CLASS = 50"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 3. Setup Folders "
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "IMAGE_DIR = os.path.join('Tensorflow', 'workspace', 'images', 'collectedimages')\n",
    "\n",
    "# create the folder to save the images\n",
    "if not os.path.exists(IMAGE_DIR): img\n",
    "    os.makedirs(IMAGE_DIR)\n",
    "    print(f\"{IMAGE_DIR} created\")\n",
    "\n",
    "# create a folder for each class of images\n",
    "# for label in CLASS_NAMES:\n",
    "#     path = os.path.join(IMAGE_DIR, label)\n",
    "#     if not os.path.exists(path):\n",
    "#         os.makedirs(path)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Tensorflow\\workspace\\images\\collectedimages created\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "IMAGE_DIR"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'Tensorflow\\\\workspace\\\\images\\\\collectedimages'"
      ]
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 3. Collecting images"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "You can choose to collect images through your own smartphone, or collect images using a quick method mentioned in the next section. "
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "- Take only around 40-50 images of your coins of choice just for testing. Since labeling could take a very long time. \n",
    "- Generally, to make a model with good performance, you would need at least 100 images for each category (aka class), but usually the more the better. Malaysia coins also have some different patterns (such as different years) so it is actually very difficult to predict correctly.\n",
    "- Therefore, in the first demo of this project, only 29 images are used for prototyping.\n",
    "<br><br><img src=\"images/IMG_3566_KEWPYXB.jpg\" width=200></img><br><br>\n",
    "- After taking photos of the coins with your phone, convert them to JPG format online for easy processing if necessary.\n",
    "\n",
    "NOTE: If you are using iPhone, you will face problem with their photos because they are in HEIC format. I recommend using [this website](https://freetoolonline.com/heic-to-jpg.html) to convert all of them into JPG format. Or you can run the two code cells below to convert them programmatically. But you **must** download and install ImageMagick from [this website first](https://imagemagick.org/script/download.php) in order for it to work."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "!pip install Wand"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Requirement already satisfied: Wand in c:\\users\\user\\anaconda3\\envs\\tfod\\lib\\site-packages (0.6.6)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from wand.image import Image\n",
    "import os\n",
    "\n",
    "# ! change these two directories to the folders you're using\n",
    "HEIC_DIR = r\"T:\\New Download Folder\\coin images - phone (original)\\heic\"\n",
    "# ! don't forget to create a new folder to store the newly converted JPG images\n",
    "# and then change this path to the new folder\n",
    "NEW_JPG_DIR = r\"T:\\New Download Folder\\coin images - phone (original)\\jpg\"\n",
    "\n",
    "# ! make sure only HEIC images are in the folder, no other files or folders\n",
    "heic_images = os.listdir(HEIC_DIR)\n",
    "total_images = len(heic_images)\n",
    "\n",
    "for i, img_name in enumerate(heic_images):\n",
    "    print(f\"[INFO] Converting {img_name} ({i+1}/{total_images})\")\n",
    "    SourceFile = os.path.join(HEIC_DIR, img_name)\n",
    "    TargetFile = os.path.join(NEW_JPG_DIR, img_name.replace(\".HEIC\",\".JPG\"))\n",
    "\n",
    "    img = Image(filename=SourceFile)\n",
    "    img.format = 'jpg'\n",
    "    img.save(filename=TargetFile)\n",
    "    img.close()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Then the following cell will resize the images to smaller size if necessary, especially since iPhone images have very high resolution and consumes a large amount of memory."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "import imutils\n",
    "from imutils.paths import list_images\n",
    "\n",
    "# ! Change these paths to suit your needs\n",
    "ORIG_IMG_DIR = r\"T:\\New Download Folder\\coin images - phone (original)\\jpg\"\n",
    "NEW_IMG_DIR = r\"T:\\New Download Folder\\resized_images\"\n",
    "\n",
    "if not os.path.exists(NEW_IMG_DIR):\n",
    "    os.makedirs(NEW_IMG_DIR)\n",
    "\n",
    "image_paths = list(list_images(ORIG_IMG_DIR))\n",
    "print(f\"[INFO] Resizing {len(image_paths)} when necessary ...\")\n",
    "\n",
    "for img_path in image_paths:\n",
    "    img = cv2.imread(img_path)\n",
    "    width, height = img.shape[:2]\n",
    "    if height > 500:\n",
    "        # resize to height of 500 while preserving aspect ratio\n",
    "        img = imutils.resize(img, height=500)\n",
    "        new_img_path = os.path.join(NEW_IMG_DIR, os.path.basename(img_path))\n",
    "        # save it to new directory\n",
    "        cv2.imwrite(new_img_path, img)\n",
    "    else:\n",
    "        # if not resizing, just copy it to new directory\n",
    "        shutil.copy2(img_path, NEW_IMG_DIR)\n",
    "print(\"[INFO] done resizing\")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[INFO] Resizing 29 when necessary ...\n",
      "[INFO] done resizing\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Or collect from Google Search using a quick method"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "1. Install the [Fatkun Batch Download Image extension](https://chrome.google.com/webstore/detail/fatkun-batch-download-ima/nnjjahlikiabnchcpehcpkdeckfgnohf?hl=en) from Google Chrome.\n",
    "2. Open up a tab and search for the images you want, e.g. cloth face mask\n",
    "3. Then use the extension to download all the images that are found in the Google Search tab. Beware that the more you scroll down, the more images will be downloaded."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Check images"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "This code cell below check and verify that the images are usable for loading and training.\n",
    "\n",
    "If you found any errors with any of the images, it's better to just remove them to avoid unnecessary errors."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "from skimage import io\n",
    "from imutils import paths\n",
    "import cv2\n",
    "\n",
    "# NOTE: Change this to your desired path\n",
    "IMAGE_PATH_TO_CHECK = r\"T:\\New Download Folder\\Object Detection Data\\coin-dataset-phone\\images\"\n",
    "image_paths = list(paths.list_images(IMAGE_PATH_TO_CHECK))\n",
    "print(f'[INFO] Checking {len(image_paths)} images ...')\n",
    "\n",
    "error_img_paths = []\n",
    "cnt = 0\n",
    "for img_path in image_paths:\n",
    "    try:\n",
    "        # To check if the images are viable for training\n",
    "        _ = io.imread(img_path)\n",
    "        img = cv2.imread(img_path)\n",
    "    except Exception as e:\n",
    "        print(e)\n",
    "        print(img_path)\n",
    "        error_img_paths.append(img_path)\n",
    "        cnt += 1\n",
    "if not cnt:\n",
    "    print(\"[INFO] All images ok\")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[INFO] Checking 29 images ...\n",
      "[INFO] All images ok\n"
     ]
    }
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 4. Image Labelling"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "1. Open Anaconda prompt and activate your environment with `conda activate tfod`\n",
    "2. Then run Label Studio by entering `label-studio`\n",
    "3. Enter \"n\" if an error about JSON Support appeared\n",
    "4. Create a local account to be used, please do not forget the email and the password\n",
    "5. Click \"Create\" button at top right to create a new project\n",
    "6. Enter a project name such as \"Coin Detection - Phone\" and click the \"Save\" button\n",
    "7. Click the \"Go to import\"\n",
    "8. Go to your phone images folder and drag all the images into Label Studio and click \"Import\"\n",
    "9. Click \"Label All Tasks\" and you will be prompted to set up labelling configuration\n",
    "10. Click \"Go to setup\" > \"Browse Templates\" > \"Computer Vision\" > \"Object Detection with Bounding Boxes\"\n",
    "11. Delete both the existing labels (\"Airplane\" and \"Car\") by clicking the red \"X\" beside them\n",
    "12. Add your label names by typing into the textbox under \"Add label names\" (see example in image below), and click \"Add\" > \"Save\"\n",
    "\n",
    "![add-labelnames](images/add-labelnames.png)\n",
    "\n",
    "13. Now click \"Label all tasks\" and start labeling the bounding boxes! \n",
    "14. Select the **class** such as '5' and draw the box on the coin. See example and best practices below.\n",
    "15. After labelling each image, click the \"Submit\" at the right and continue until the Label Studio says you have finished labeling.\n",
    "16. Finally, click the \"Export\" at the top right and make sure to select \"Pascal VOC XML\" before clicking \"Export\". Then you are done labeling!\n",
    "\n",
    "<img src=\"images/label-example.png\" width=500></img>\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Best practices on how to label images properly:\n",
    "1. Label every object of interest in every image\n",
    "2. Label the entirety of an object **tightly** and not cutting off parts of the object\n",
    "3. Label occluded objects as if they were fully visible\n",
    "4. Create specific label names that are not too general\n",
    "\n",
    "Refer to the [link here](https://blog.roboflow.com/tips-for-how-to-label-images/) for more details.\n",
    "\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Copy the images to the desired path"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finally, use the code below to copy the files exported from Label Studio, and move them to the desired directory, `IMAGE_DIR`."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "IMAGE_DIR"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'Tensorflow\\\\workspace\\\\images\\\\collectedimages'"
      ]
     },
     "metadata": {},
     "execution_count": 6
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "# ! change this path to your dataset path\n",
    "DATA_PATH = r\"T:\\New Download Folder\\Object Detection Data\\coin-dataset-phone\"\n",
    "\n",
    "if os.path.exists(IMAGE_DIR):\n",
    "    # remove the directory before copying\n",
    "    shutil.rmtree(IMAGE_DIR)\n",
    "print('[INFO] Copying ...')\n",
    "shutil.copytree(DATA_PATH, IMAGE_DIR)\n",
    "print('[INFO] Copied successfully')"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[INFO] Copying ...\n",
      "[INFO] Copied successfully\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 6. Split the dataset and move them into train, valid, and test folders"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We will split the dataset into training, testing datasets of ratio 80% : 15 %."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "TEST_SIZE = 0.15\n",
    "TRAIN_SIZE = round(1 - TEST_SIZE, 2)\n",
    "\n",
    "# change this random seed if the split of images is good\n",
    "# i.e. the test set should at least contain one image of multiple coins together\n",
    "RANDOM_SEED = 41"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from imutils import paths\n",
    "\n",
    "# get the image paths and sort them\n",
    "image_paths = sorted(paths.list_images(os.path.join(IMAGE_DIR, \"images\")))\n",
    "print(f\"Total images = {len(image_paths)}\")\n",
    "\n",
    "# directory to annotation folder, only change this path when necessary\n",
    "ANNOTATION_DIR = os.path.join(IMAGE_DIR, \"Annotations\")\n",
    "# get the label paths and sort them to align with image paths\n",
    "label_paths = sorted(os.listdir(ANNOTATION_DIR))\n",
    "# append the directory at the front of the folder for full paths\n",
    "label_paths = [os.path.join(ANNOTATION_DIR, i) for i in label_paths]\n",
    "\n",
    "# split the dataset into ratio of train:test of 85%: 15%\n",
    "print(\"Splitting into train:test dataset\"\n",
    "    f\" with ratio of {TRAIN_SIZE:.2f}:{TEST_SIZE:.2f}\")\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    image_paths, label_paths, test_size=TEST_SIZE, random_state=RANDOM_SEED\n",
    ")\n",
    "print(f\"Total training images = {len(X_train)}\")\n",
    "print(f\"Total testing images = {len(X_test)}\")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Total images = 29\n",
      "Splitting into train:test dataset with ratio of 0.85:0.15\n",
      "Total training images = 24\n",
      "Total testing images = 5\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "source": [
    "SPLIT_IMAGE_DIR = os.path.join('Tensorflow', 'workspace', 'images')\n",
    "\n",
    "def copy_images(image_paths, label_paths, data_type):\n",
    "    assert data_type in (\"train\", \"valid\", \"test\")\n",
    "    image_dest = os.path.join(SPLIT_IMAGE_DIR, data_type)\n",
    "\n",
    "    print(f\"[INFO] Copying files from {IMAGE_DIR} to {image_dest}\")\n",
    "    if os.path.exists(image_dest):\n",
    "        # remove the existing images\n",
    "        shutil.rmtree(image_dest, ignore_errors=False)\n",
    "    \n",
    "    if not os.path.exists(image_dest):\n",
    "        # create new directories\n",
    "        os.makedirs(image_dest)\n",
    "\n",
    "    for image_path, label_path in zip(image_paths, label_paths):\n",
    "        # copy the image file and label file to the new directory\n",
    "        shutil.copy2(image_path, image_dest)\n",
    "        shutil.copy2(label_path, image_dest)\n",
    "\n",
    "copy_images(X_train, y_train, \"train\")\n",
    "copy_images(X_test, y_test, \"test\")\n",
    "print(\"[INFO] Files copied successfully.\")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[INFO] Copying files from Tensorflow\\workspace\\images\\collectedimages to Tensorflow\\workspace\\images\\train\n",
      "[INFO] Copying files from Tensorflow\\workspace\\images\\collectedimages to Tensorflow\\workspace\\images\\test\n",
      "[INFO] Files copied successfully.\n"
     ]
    }
   ],
   "metadata": {
    "tags": []
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# OPTIONAL - 7. Compress them for Colab Training"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "source": [
    "import os\n",
    "ARCHIVE_PATH = os.path.join('Tensorflow', 'workspace', 'images', 'archive.tar.gz')\n",
    "\n",
    "SPLIT_IMAGE_DIR = os.path.join('Tensorflow', 'workspace', 'images')\n",
    "TRAIN_PATH = os.path.join(SPLIT_IMAGE_DIR, \"train\")\n",
    "TEST_PATH = os.path.join(SPLIT_IMAGE_DIR, \"test\")\n",
    "\n",
    "# add the training and testing datasets to a tar file\n",
    "!tar -czf {ARCHIVE_PATH} {TRAIN_PATH} {TEST_PATH}\n",
    "print(f\"File saved at {ARCHIVE_PATH}\")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "File saved at Tensorflow\\workspace\\images\\archive.tar.gz\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "OR You can also download the following datasets, the first two are prepared by Anson:\n",
    "1. Coin detection dataset (small, 29 images, captured with iPhone): [Google Drive link](https://drive.google.com/file/d/1BJEUTWuSNl8rbmwxpPEtBT3AQKH34_Y8/view?usp=sharing)\n",
    "2. Coin detection dataset (large, 277 images, from Google Search): [Google Drive link](https://drive.google.com/file/d/1EwA81lgR2qWAAkcYb03FkGL1m4uT-z6h/view?usp=sharing)\n",
    "3. Face Mask Detection dataset (853 images): [Kaggle link](https://www.kaggle.com/andrewmvd/face-mask-detection)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "NOTE: This tar file is used to upload to Google Colab to use for training in the next notebook."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "You may also refer to the following Colab notebooks to examine the trained version for each of the 3 datasets used in this repo. But the procedure is exactly the same, with only a few modifications to accomodate the download the dataset, and also to modify the class labels for different datasets.\n",
    "1. Coin detection dataset (small, 29 images, captured with iPhone): [link](https://colab.research.google.com/drive/1M0IN3Ya3jT_7UOJ5N2wLepk2ypR78KFB?usp=sharing)\n",
    "2. Coin detection dataset (large, 277 images, from Google Search): [link](https://colab.research.google.com/drive/1aTliHKpwqxZaokf2YTmg0vwmRPUJnpzY?usp=sharing)\n",
    "3. Face Mask Detection dataset (853 images): [link](https://colab.research.google.com/drive/11ciR0XNAvICh5teg0AaFsK7sK8iSzVoh?usp=sharing)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Acknowledgement\n",
    "\n",
    "This notebook is inspired by the notebook in this [Github repo](https://github.com/nicknochnack/TFODCourse) by [Nicholas Renotte](https://youtu.be/yqkISICHH-U?list=PLEG-yoFSLEJ7inw5rNtyqaFAlOHkyWU3t)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "6d46af94c2bbce495f1e668725902fa517c90b1782bcfe2fce0dd9868df553d3"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.8.10 64-bit ('base': conda)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}