Food Recognition Benchmark 2022
Detectron2 training and submissions (Quick, Active)
v2 of baseline, now supports submissions via Colab for both quick and active participation
jerome_patel
🍕 Food Recognition Benchmark¶
Problem Statement¶
Detecting & Segmenting various kinds of food from an image. For ex. Someone got into new restaurent and get a food that he has never seen, well our DL model is in rescue, so our DL model will help indentifying which food it is from the class our model is being trained on!
Dataset¶
We will be using data from Food Recognition Challenge - A benchmark for image-based food recognition challange which is running since 2020.
https://www.aicrowd.com/challenges/food-recognition-benchmark-2022#datasets
We have a total of 39k training images with 3k validation set and 4k public-testing set. All the images are RGB and annotations exist in MS-COCO format.
Evaluation¶
The evaluation metrics is IOU aka. Intersection Over Union ( more about that later ).
The actualy metric is computed by averaging over all the precision and recall values for IOU which greater than 0.5.
https://www.aicrowd.com/challenges/food-recognition-challenge#evaluation-criteria
What does this notebook contains?¶
Setting our Workspace 💼
Data Exploration 🧐
- Reading Dataset
- Data Visualisations
Image Visulisation 🖼️
- Reading Images
Creating our Dataset 🔨
- Fixing the Dataset
- Creating our dataset
Creating our Model 🏭
- Creating R-CNN Model
- Setting up hyperparameters
Training the Model 🚂
- Setting up Tensorboard
- Start Training!
Evaluating the model 🧪
- Evaluating our Model
Testing the Model 💯
- Testing the Model
Submitting our predictions 📝
Setting our Workspace 💼¶
In this section we will be downloading our dataset, unzipping it & downliading detectron2 library and importing all libraries that we will be using
Downloading & Unzipping our Dataset¶
In [1]:
# Login to AIcrowd
!pip install aicrowd-cli > /dev/null
!aicrowd login
In [2]:
# List dataset for this challenge
!aicrowd dataset list -c food-recognition-benchmark-2022
# Download dataset
!aicrowd dataset download -c food-recognition-benchmark-2022
In [3]:
# Create data directory
!mkdir -p data/ data/train data/val data/test
!cp *test* data/test && cd data/test && echo "Extracting test dataset" && tar -xvf *test* > /dev/null
!cp *val* data/val && cd data/val && echo "Extracting val dataset" && tar -xvf *val* > /dev/null
!cp *train* data/train && cd data/train && echo "Extracting train dataset" && tar -xvf *train* > /dev/null
Mount the Google Drive¶
In [4]:
from google.colab import drive
drive.mount('/content/drive')
So, the data directory is something like this:
Importing Necessary Libraries¶
In [5]:
# Making sure that we are using GPUs
!nvidia-smi
In [6]:
!pip install cython pyyaml==5.1
!pip install -U pycocotools
import torch
TORCH_VERSION = ".".join(torch.__version__.split(".")[:2])
CUDA_VERSION = torch.__version__.split("+")[-1]
print("torch: ", TORCH_VERSION, "; cuda: ", CUDA_VERSION)
# Install detectron2 that matches the above pytorch version
# See https://detectron2.readthedocs.io/tutorials/install.html for instructions
!pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/$CUDA_VERSION/torch$TORCH_VERSION/index.html
# If there is not yet a detectron2 release that matches the given torch + CUDA version, you need to install a different pytorch.
#don't forget to restart the runtime
In [7]:
# You may need to restart your runtime prior to this, to let your installation take effect
# Some basic setup:
# Setup detectron2 logger
import detectron2
from detectron2.utils.logger import setup_logger
setup_logger()
# import some common libraries
import numpy as np
import pandas as pd
import cv2
import json
from tqdm.notebook import tqdm
import subprocess
import time
from pathlib import Path
# import some common detectron2 utilities
from detectron2 import model_zoo
from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.utils.visualizer import Visualizer
from detectron2.data import MetadataCatalog
from detectron2.utils.visualizer import ColorMode
from detectron2.data.datasets import register_coco_instances
from detectron2.engine import DefaultTrainer
from detectron2.evaluation import COCOEvaluator, inference_on_dataset
from detectron2.data import build_detection_test_loader
from detectron2.structures import Boxes, BoxMode
import pycocotools.mask as mask_util
# For reading annotations file
from pycocotools.coco import COCO
# utilities
from pprint import pprint # For beautiful print!
from collections import OrderedDict
import os
# For data visualisation
import matplotlib.pyplot as plt
import plotly.graph_objects as go
import plotly.express as px
from google.colab.patches import cv2_imshow
Data Exploration 🧐¶
In this section we are going to read our dataset & doing some data visualisations
Reading Data¶
In [8]:
# Reading annotations.json
TRAIN_ANNOTATIONS_PATH = "data/train/annotations.json"
TRAIN_IMAGE_DIRECTIORY = "data/train/images/"
VAL_ANNOTATIONS_PATH = "data/val/annotations.json"
VAL_IMAGE_DIRECTIORY = "data/val/images/"
train_coco = COCO(TRAIN_ANNOTATIONS_PATH)
In [9]:
# Reading the annotation files
with open(TRAIN_ANNOTATIONS_PATH) as f:
train_annotations_data = json.load(f)
with open(VAL_ANNOTATIONS_PATH) as f:
val_annotations_data = json.load(f)
train_annotations_data['annotations'][0]
Out[9]:
Data Format 🔍¶
Our COCO data format is something like this -
"info": {...},
"categories": [...],
"images": [...],
"annotations": [...],In which categories is like this
[
{'id': 2578,
'name': 'water',
'name_readable': 'Water',
'supercategory': 'food'},
{'id': 1157,
'name': 'pear',
'name_readable': 'Pear',
'supercategory': 'food'},
...
{'id': 1190,
'name': 'peach',
'name_readable': 'Peach',
'supercategory': 'food'}
]Info is empty ( not sure why )
images is like this
[
{'file_name': '065537.jpg',
'height': 464,
'id': 65537,
'width': 464},
{'file_name': '065539.jpg',
'height': 464,
'id': 65539,
'width': 464},
...
{'file_name': '069900.jpg',
'height': 391,
'id': 69900,
'width': 392},
]Annotations is like this
{'area': 44320.0,
'bbox': [86.5, 127.49999999999999, 286.0, 170.0],
'category_id': 2578,
'id': 102434,
'image_id': 65537,
'iscrowd': 0,
'segmentation': [[235.99999999999997,
372.5,
169.0,
372.5,
...
368.5,
264.0,
371.5]]}In [10]:
# Reading all classes
category_ids = train_coco.loadCats(train_coco.getCatIds())
category_names = [_["name_readable"] for _ in category_ids]
print("## Categories\n-", "\n- ".join(category_names))
In [11]:
# Getting all categoriy with respective to their total images
no_images_per_category = {}
for n, i in enumerate(train_coco.getCatIds()):
imgIds = train_coco.getImgIds(catIds=i)
label = category_names[n]
no_images_per_category[label] = len(imgIds)
img_info = pd.DataFrame(train_coco.loadImgs(train_coco.getImgIds()))
no_images_per_category = OrderedDict(sorted(no_images_per_category.items(), key=lambda x: -1*x[1]))
# Top 30 categories, based on number of images
i = 0
for k, v in no_images_per_category.items():
print(k, v)
i += 1
if i > 30:
break
Data Visualisations¶
In [12]:
fig = go.Figure([go.Bar(x=list(no_images_per_category.keys())[:50], y=list(no_images_per_category.values())[:50])])
fig.update_layout(
title="No of Image per class",)
fig.show()
fig = go.Figure([go.Bar(x=list(no_images_per_category.keys())[50:200], y=list(no_images_per_category.values())[50:200])])
fig.update_layout(
title="No of Image per class",)
fig.show()
fig = go.Figure([go.Bar(x=list(no_images_per_category.keys())[200:], y=list(no_images_per_category.values())[200:])])
fig.update_layout(
title="No of Image per class",)
fig.show()
In [13]:
pprint(f"Average number of image per class : { sum(list(no_images_per_category.values())) / len(list(no_images_per_category.values())) }")
pprint(f"Highest number of image per class is : { list(no_images_per_category.keys())[0]} of { list(no_images_per_category.values())[0] }")
pprint(f"Lowest number of image per class is : Veggie Burger of { sorted(list(no_images_per_category.values()))[0] }")
In [14]:
fig = go.Figure(data=[go.Pie(labels=list(no_images_per_category.keys())[:50], values=list(no_images_per_category.values())[:50],
hole=.3, textposition='inside', )], )
fig.update_layout(
title="No of Image per class Top 50 ( In pie )",)
fig.show()
In [15]:
fig = go.Figure()
fig.add_trace(go.Histogram(x=img_info['height'],text='height'))
fig.add_trace(go.Histogram(x=img_info['width'],text='width'))
# Overlay both histograms
fig.update_layout(barmode='stack', title="Histogram of Image width & height",)
fig.show()
Image Visulisation 🖼️¶
In this section we are going to do image visualisations!
In [16]:
print(img_info)
print(img_info.describe())
In [17]:
len(train_annotations_data['annotations'][2]['segmentation']), len(train_annotations_data['annotations'][2]['bbox'])
Out[17]:
In [18]:
img_no = 11
annIds = train_coco.getAnnIds(imgIds=train_annotations_data['images'][img_no]['id'])
anns = train_coco.loadAnns(annIds)
# load and render the image
plt.imshow(plt.imread(TRAIN_IMAGE_DIRECTIORY+train_annotations_data['images'][img_no]['file_name']))
plt.axis('off')
# Render annotations on top of the image
train_coco.showAnns(anns)
In [19]:
w, h = 15, 15 # Setting width and height of every image
rows, cols = 5, 5 # Setting the number of image rows & cols
fig = plt.figure(figsize=(15, 15)) # Making the figure with size
plt.title("Images")
plt.axis('off')
# Going thought every cell in rows and cols
for i in range(1, cols * rows+1):
annIds = train_coco.getAnnIds(imgIds=img_info['id'][i])
anns = train_coco.loadAnns(annIds)
fig.add_subplot(rows, cols, i)
# Show the image
img = plt.imread(TRAIN_IMAGE_DIRECTIORY+img_info['file_name'][i])
for i in anns:
[x,y,w,h] = i['bbox']
#create rectagle bbox of size given in dataset
cv2.rectangle(img, (int(x), int(y)), (int(x+h), int(y+w)), (255,0,0), 2)
plt.imshow(img)
# Render annotations on top of the image
train_coco.showAnns(anns)
# Setting the axis off
plt.axis("off")
# Showing the figure
plt.show()