AI Blitz #6

Learn from started code

Using started codes shared by admin and improve accuracy for the image classifier problems

By demarsylvain

Using started codes shared by admin and improve accuracy 98% for the image problems

Learn from started codes

  • Motivation

As a chess player (♟♞ ELO ± 1,600) and data scientist (R pseudo expert), this AI Blitz caught my interest! But my expertise on image/video analysis is weak. Although I know the theory of neural network, I never have the chance to practice.

Could this challenge be a first introduction ?

--> Thanks to baseline scripts shared by the admin (big thank you to 👍 @Ashivani 👍), the answer is yes.

  • Context

A) 3 of the 5 challenges are about image binary classification. From a chessboard picture, we would like to estimate which player (black or white):

  1. have more pieces?
  2. have more points?
  3. is the winner?

B) One challenge is about image transcription: describe FEN notation from a chessboard image. 

C) The last one is about video transcription: descrive piece moves from a short video.

  • AICrowd connexion 🔌
In [1]:
!pip install --upgrade fastai git+https://gitlab.aicrowd.com/yoogottamk/aicrowd-cli.git >/dev/null
%load_ext aicrowd.magic
API_KEY = '6ffcf3610910595c9ff5945edf539c47'
%aicrowd login --api-key $API_KEY
  Running command git clone -q https://gitlab.aicrowd.com/yoogottamk/aicrowd-cli.git /tmp/pip-req-build-w6npo_0q
Verifying API Key...
API Key valid
Saved API Key successfully!

A) Image binary classification

For this king of problem, a starting solution is to used a pre-trained model. Even if it was not trained with chess pictures, its previous learnings could be robust enough for other problems. As chess pictures of these challenges are quite clean (similar size, easily readible, etc.), it could work without specifically building new layer.

The baseline propose to use AlexNet, a Convolutional Neural Network (CNN) designed for classify images. Let's see what it will done on these chess pictures.

Import Packages 📦

In [ ]:
import pandas as pd
from fastai.vision.all import *
from fastai.data.core import *
import os

Access Data ♚♕♜♘♝♙

We start by downloading the zipfile, and unzip them. Let's start with the first challenge (Pieces).

  • Pieces
In [ ]:
%aicrowd dataset download --challenge chess-pieces -j 3
In [ ]:
!rm -rf data
!mkdir data
!mkdir data/pieces

!unzip train.zip  -d data/pieces/ 
!unzip val.zip -d data/pieces/ 
!unzip test.zip  -d data/pieces/
In [ ]:
!mv train.csv data/pieces/train.csv
!mv val.csv data/pieces/val.csv
!mv sample_submission.csv data/pieces/sample_submission.csv

We can visualize a summary table, containing image names and labels.

In [ ]:
train_df = pd.read_csv("data/pieces/train.csv")
train_df['ImageID'] = train_df['ImageID'].astype(str)+".jpg"

We can also visualise some training images (chessboard with label)

In [ ]:
dls = ImageDataLoaders.from_df(train_df, path="data/pieces/train", bs=8)

Pre-trained model 💪

In [ ]:
learn = cnn_learner(dls, alexnet, metrics = F1Score())
Downloading: "https://download.pytorch.org/models/alexnet-owt-4df8aa71.pth" to /root/.cache/torch/hub/checkpoints/alexnet-owt-4df8aa71.pth

epoch train_loss valid_loss f1_score time
0 0.734799 0.712334 0.557088 01:49

For the 3 image binary classification problem (Pieces, Points and WinPrediction), the accuracy with this model is around 60 %. As it's my first Python neural network code, I start by playing with it, trying to improve the accuracy.

Improvments tests 🧪

There are several ways to (try to) improve such a solution:

  • Increase number of epochs of the neural network

  • Increase number of training images, by transforming existing ones for instance

    aug_transforms(do_flip = False)
  • Test different parameters of the neural network (learning rate for instance)

    learn.fit_one_cycle(2, lr_max = 1e-3)
  • Test other pre-trained classifier

Results 📝

  • Pieces

With the different options described above, I managed to increase the accuracy for the first challenge - which players have more pieces - to an unexpected level (99.99 % 😜) using the pre-trained model ResNet-50 and only 3 epochs (see below). There are several participants with a similar performance (and it should increase day after day).

  • Points and WinPrediction

With a similar approch, I reached accuracy of 98.8% for Points challenge, and 94% for WinPrediction. Some other participants manage to get better accuracy (close 100 %), meaning there are other improvments I have to do.

At the beginning of the training, accuracy increase with epoch number, but quickly (7-8 epochs) converge, meaning the algorithm will overfit if continue. Other kind of improvment should be explore.


  • Code
In [ ]:
dls = ImageDataLoaders.from_df(train_df, path="data/pieces/train", bs=8)
learn = cnn_learner(dls, models.resnet50, metrics = F1Score())
Downloading: "https://download.pytorch.org/models/resnet50-19c8e357.pth" to /root/.cache/torch/hub/checkpoints/resnet50-19c8e357.pth

In [ ]:
epoch train_loss valid_loss f1_score time
0 0.666329 0.609515 0.669332 07:12
epoch train_loss valid_loss f1_score time
0 0.205505 0.051047 0.998147 09:59
1 0.119291 0.007659 0.999876 09:59
2 0.086349 0.006662 1.000000 10:00

Submission ✉

We can use this solution to predict on the test dataset, and submit on the challenge to be sure our solution has a similar accuracy, and it didn't overfitting the training dataset.

In [ ]:
test_imgs_name = get_image_files("data/pieces/test")
test_dls = dls.test_dl(test_imgs_name)
label_to_class_mapping = {v: k for v, k in enumerate(dls.vocab)}
test_img_ids = [re.sub(r"\D", "", str(img_name)) for img_name in test_imgs_name]
_,_,results = learn.get_preds(dl = test_dls, with_decoded = True)
results = [label_to_class_mapping[i] for i in results.numpy()]
In [ ]:
submission = pd.DataFrame({"ImageID":test_img_ids, "label":results})
submission.to_csv("submission.csv", index=False)
%aicrowd submission create -c chess-pieces -f submission.csv

B) Image transcription

(on going)

Import Packages 📦

In [2]:
from tqdm.notebook import tqdm
import numpy as np
import os
import glob
import re
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from random import shuffle
from skimage.util.shape import view_as_blocks
from skimage import io, transform
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten
from keras.layers.convolutional import Convolution2D
import warnings
import pandas as pd

Access Data

In [ ]:
%aicrowd dataset download --challenge chess-configuration -j 3

!mkdir data
!unzip train.zip  -d data/config/ 
!unzip val.zip -d data/config/ 
!unzip test.zip  -d data/config/ 

!mv train.zip data/config/train.zip
!mv train.csv data/config/train.csv
!mv val.csv data/config/val.csv
!mv val.zip data/config/val.zip
!mv test.zip data/config/test.zip
!mv sample_submission.csv data/config/sample_submission.csv
In [5]:
train = glob.glob("data/config/train/*.jpg")
test = glob.glob("data/config/test/*.jpg")
In [9]:
train_csv = pd.read_csv("data/config/train.csv")
train_csv['ImageID'] = train_csv['ImageID'].astype(str)+".jpg"
ImageID label
0 0.jpg 1rbqkb1r/p1p1n1pp/1pn1p3/1P1p1p2/3P4/N3B2P/P1P...
1 1.jpg 2bk4/2q1p3/3p3P/5r2/r5nP/P2K2N1/8/2q1NB1R
2 2.jpg 3rnq2/3k1p2/5rP1/pppp1P2/P1BP2P1/RPP1K2N/3B3P/...
3 3.jpg r4br1/1p2ppp1/3k2Pp/p2P3n/2pP1N2/P1P1K3/nP5P/1...
4 4.jpg 4kn2/8/p6b/1p6/P1p1p1pr/1P2P1N1/1R1r3P/1K2R1N1
... ... ...
39995 39995.jpg r7/B6k/2pp1bpP/8/1p6/1P3P2/5N2/5KRB
39996 39996.jpg r5B1/1b2k1b1/4Npp1/p3P3/P2p1P2/1R2P3/2P2KqP/1Q...
39997 39997.jpg 6n1/k7/1pP5/1P1pP1n1/p4PBP/R6b/2Kb4/3N2R1
39998 39998.jpg r1b1qk2/3n4/1pp1nb2/p3p1Nr/2p3Pp/PP2P2P/R2PBP1...
39999 39999.jpg 1r1n3r/p3k3/P7/2p1P3/1P3p1p/7P/2K2R2/3Q4

40000 rows × 2 columns

In [14]:
f, axarr = plt.subplots(1,3, figsize=(80, 80))

for i in range(0,3):
    axarr[i].set_title(train_csv['label'][i], fontsize=50, pad=30)
    axarr[i].imshow(mpimg.imread('data/config/train/' + train_csv['ImageID'][i]))