Loading

Data Purchasing Challenge 2022

WANDB - Build better models faster with experiment tracking

Track, compare, and visualize ML experiments with a few lines of code. Baseline as an ILLUSTRATION.

gaurav_singhal

I am sure everyone is trying different things to increase their performance. Some of them work, some of them don't, but how do you track each experiment. 

Wandb is a MLOps platform. It helps you build better models faster with experiment tracking, dataset versioning, and model management. It enables you to track the progress of your experiments in real-time on your web browser.  Setting up wandb will take just 10-15 mins and in this notebook, I will show you how you can set it up with an illustration of this challenge. 

I am using baseline as an illustration. If you want to learn how to make a baseline submission (video tutorial) please refer to this notebook: https://www.aicrowd.com/showcase/create-your-baseline-with-0-4-on-lb-git-repo-and-video

How to use this notebook 📝

You can use this notebook as stand-alone colab solution, however I have not written a code to make a submission.

Content of this notebook

  1. Download relevant data
  2. Install requirements
  3. Initialize datasets
  4. Setup the wandb (Shown in below cells)
  5. Baseline model in run.sh
  6. Execution pipeline You can copy the relevant code to your codebase during submission.

Note: I have written code for both CPU and CUDA. Search for this key: "CHANGE CPU CUDA HERE" and make changes as per your machine. You just have to comment and uncomment a very few lines.

Setting up WANDB

Steps:

  1. Create a free account on Wandb (https://wandb.ai/)
  2. Create a project in your profile (https://wandb.ai/USERNAME). You will get an API key which you should use for authentication while running the experiment.
  3. Install wandb (pip install wandb)
  4. Create wandb configuration (To track the Hyper-Parameters of your model). It's a free end dictionary which will store anything you want. Example: name of experient, learning rate, batch size, model name, optimizer name, transformations used, etc. An example is shown below
     wandb_config={
         "model": "ResNet101",
         "learning_rate": self.LEARNING_RATE,
         "samples_per_gpu": self.BATCH_SIZE,
         "workers_per_gpu": self.NUM_WORKERS,
         "augmentations": "NO",
         "description": "Pre-Training. No augmentation"
     }
  5. Initialize your project in run.py. An example is shown below
     wandb.init(project="zew-data-purchase", entity="krazy", name=model_name, config=wandb_config)
    here,
    • project-> The one you created in step 2
    • entity-> Your wandb username
    • name-> Name of your experiment
  6. Track the performance. Here you can put all the metrics you want to analyse. For example, loss at each step/epoch, accuracy at nth epoch, hamming loss at nth etc. An example is shown below.
    For loss
     wandb.log({
         "Step": (self.epoch*5000)+self.global_step,
         "Loss": loss,
     })
    For performance
     wandb.log({
         "Epoch": self.epoch+1,
         "Accuracy": accuracy_score,
         "Hamming Loss": hamming_loss_score,
         "Match ratio": exact_match_ratio_score
     })
  7. Analyse the performance with different graphs in your Wandb project. (https://wandb.ai/USERNAME/PROJECT-NAME)

1) Login to AIcrowd 🤩

In [1]:
#@title Login to AIcrowd
!pip install -U aicrowd-cli > /dev/null
!aicrowd login 2> /dev/null
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-colab 1.0.0 requires requests~=2.23.0, but you have requests 2.27.1 which is incompatible.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
Please login here: https://api.aicrowd.com/auth/haKT8-uY-hsX6z30NGXp-zU8uxpOIFgzAjbTutOokMU
API Key valid
Gitlab access token valid
Saved details successfully!

2) Setup magically, run the below cell 😉

In [2]:
#@title Magic Box ⬛ { vertical-output: true, display-mode: "form" }
try:
  import os
  if first_run and os.path.exists("/content/data-purchasing-challenge-2022-starter-kit/data/training"):
    first_run = False
except:
  first_run = True

if first_run:
  %cd /content/
  !git clone http://gitlab.aicrowd.com/zew/data-purchasing-challenge-2022-starter-kit.git > /dev/null
  %cd data-purchasing-challenge-2022-starter-kit
  !aicrowd dataset list -c data-purchasing-challenge-2022
  !aicrowd dataset download -c data-purchasing-challenge-2022
  !mkdir -p data/
  !mv *.tar.gz data/ && cd data && echo "Extracting dataset" && ls *.tar.gz | xargs -n1 -I{} bash -c "tar -xvf {} > /dev/null"


def run_pre_training_phase():
  from run import ZEWDPCBaseRun
  run = ZEWDPCBaseRun()
  run.pre_training_phase = pre_training_phase
  run.pre_training_phase(self=run, training_dataset=training_dataset)
  # NOTE:It is critical that the checkpointing works in a self-contained way
  #      As, the evaluators might choose to run the different phases separately.
  run.save_checkpoint("/tmp/pretrainig_phase_checkpoint.pickle")

def run_purchase_phase():
  from run import ZEWDPCBaseRun
  run = ZEWDPCBaseRun()
  run.pre_training_phase = pre_training_phase
  run.purchase_phase = purchase_phase
  run.load_checkpoint("/tmp/pretrainig_phase_checkpoint.pickle")
  # Hacky way to make it work in notebook
  unlabelled_dataset.purchases = set()
  run.purchase_phase(self=run, unlabelled_dataset=unlabelled_dataset, training_dataset=training_dataset, budget=3000)
  run.save_checkpoint("/tmp/purchase_phase_checkpoint.pickle")
  del run

def run_prediction_phase():
  from run import ZEWDPCBaseRun
  run = ZEWDPCBaseRun()
  run.pre_training_phase = pre_training_phase
  run.purchase_phase = purchase_phase
  run.prediction_phase = prediction_phase
  run.load_checkpoint("/tmp/purchase_phase_checkpoint.pickle")
  run.prediction_phase(self=run, test_dataset=val_dataset)
  del run
/content
Cloning into 'data-purchasing-challenge-2022-starter-kit'...
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 111 (delta 3), reused 1 (delta 0), pack-reused 99
Receiving objects: 100% (111/111), 39.76 KiB | 515.00 KiB/s, done.
Resolving deltas: 100% (57/57), done.
/content/data-purchasing-challenge-2022-starter-kit
                 Datasets for challenge #1024                                   
┌───┬───────────────────┬──────────────────────────┬─────────┐                  
│ #  Title              Description                  Size │                  
├───┼───────────────────┼──────────────────────────┼─────────┤                  
│ 0 │ validation.tar.gz │ Validation dataset       │ 182 MiB │                  
│ 1 │ unlabelled.tar.gz │ Unlabelled image dataset │ 609 MiB │                  
│ 2 │ training.tar.gz   │ Training data            │ 304 MiB │                  
│ 3 │ debug.tar.gz      │ Debug dataset            │ 6.1 MiB │                  
└───┴───────────────────┴──────────────────────────┴─────────┘                  
validation.tar.gz: 100% 191M/191M [00:10<00:00, 18.5MB/s]
unlabelled.tar.gz: 100% 638M/638M [00:57<00:00, 11.1MB/s]
training.tar.gz: 100% 319M/319M [00:15<00:00, 20.2MB/s]
debug.tar.gz: 100% 6.43M/6.43M [00:00<00:00, 9.65MB/s]
Extracting dataset

3) Writing your code implementation! ✍️

a) Runtime Packages

In [3]:
#@title a) Runtime Packages<br/><small>Important: Add the packages required by your code here. (space separated)</small> { run: "auto", display-mode: "form" }
apt_packages = "build-essential vim" #@param {type:"string"}
pip_packages = "scikit-image pandas timeout-decorator==0.5.0 numpy" #@param {type:"string"}

!apt install -y $apt_packages git-lfs
!pip install $pip_packages
Reading package lists... Done
Building dependency tree       
Reading state information... Done
build-essential is already the newest version (12.4ubuntu1).
The following packages were automatically installed and are no longer required:
  cuda-command-line-tools-10-0 cuda-command-line-tools-10-1
  cuda-command-line-tools-11-0 cuda-compiler-10-0 cuda-compiler-10-1
  cuda-compiler-11-0 cuda-cuobjdump-10-0 cuda-cuobjdump-10-1
  cuda-cuobjdump-11-0 cuda-cupti-10-0 cuda-cupti-10-1 cuda-cupti-11-0
  cuda-cupti-dev-11-0 cuda-documentation-10-0 cuda-documentation-10-1
  cuda-documentation-11-0 cuda-documentation-11-1 cuda-gdb-10-0 cuda-gdb-10-1
  cuda-gdb-11-0 cuda-gpu-library-advisor-10-0 cuda-gpu-library-advisor-10-1
  cuda-libraries-10-0 cuda-libraries-10-1 cuda-libraries-11-0
  cuda-memcheck-10-0 cuda-memcheck-10-1 cuda-memcheck-11-0 cuda-nsight-10-0
  cuda-nsight-10-1 cuda-nsight-11-0 cuda-nsight-11-1 cuda-nsight-compute-10-0
  cuda-nsight-compute-10-1 cuda-nsight-compute-11-0 cuda-nsight-compute-11-1
  cuda-nsight-systems-10-1 cuda-nsight-systems-11-0 cuda-nsight-systems-11-1
  cuda-nvcc-10-0 cuda-nvcc-10-1 cuda-nvcc-11-0 cuda-nvdisasm-10-0
  cuda-nvdisasm-10-1 cuda-nvdisasm-11-0 cuda-nvml-dev-10-0 cuda-nvml-dev-10-1
  cuda-nvml-dev-11-0 cuda-nvprof-10-0 cuda-nvprof-10-1 cuda-nvprof-11-0
  cuda-nvprune-10-0 cuda-nvprune-10-1 cuda-nvprune-11-0 cuda-nvtx-10-0
  cuda-nvtx-10-1 cuda-nvtx-11-0 cuda-nvvp-10-0 cuda-nvvp-10-1 cuda-nvvp-11-0
  cuda-nvvp-11-1 cuda-samples-10-0 cuda-samples-10-1 cuda-samples-11-0
  cuda-samples-11-1 cuda-sanitizer-11-0 cuda-sanitizer-api-10-1
  cuda-toolkit-10-0 cuda-toolkit-10-1 cuda-toolkit-11-0 cuda-toolkit-11-1
  cuda-tools-10-0 cuda-tools-10-1 cuda-tools-11-0 cuda-tools-11-1
  cuda-visual-tools-10-0 cuda-visual-tools-10-1 cuda-visual-tools-11-0
  cuda-visual-tools-11-1 default-jre dkms freeglut3 freeglut3-dev
  keyboard-configuration libargon2-0 libcap2 libcryptsetup12
  libdevmapper1.02.1 libfontenc1 libidn11 libip4tc0 libjansson4
  libnvidia-cfg1-510 libnvidia-common-460 libnvidia-common-510
  libnvidia-extra-510 libnvidia-fbc1-510 libnvidia-gl-510 libpam-systemd
  libpolkit-agent-1-0 libpolkit-backend-1-0 libpolkit-gobject-1-0 libxfont2
  libxi-dev libxkbfile1 libxmu-dev libxmu-headers libxnvctrl0 libxtst6
  nsight-compute-2020.2.1 nsight-compute-2022.1.0 nsight-systems-2020.3.2
  nsight-systems-2020.3.4 nsight-systems-2021.5.2 nvidia-dkms-510
  nvidia-kernel-common-510 nvidia-kernel-source-510 nvidia-modprobe
  nvidia-settings openjdk-11-jre policykit-1 policykit-1-gnome python3-xkit
  screen-resolution-extra systemd systemd-sysv udev x11-xkb-utils
  xserver-common xserver-xorg-core-hwe-18.04 xserver-xorg-video-nvidia-510
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  libgpm2 vim-common vim-runtime xxd
Suggested packages:
  gpm ctags vim-doc vim-scripts
The following NEW packages will be installed:
  git-lfs libgpm2 vim vim-common vim-runtime xxd
0 upgraded, 6 newly installed, 0 to remove and 39 not upgraded.
Need to get 8,854 kB of archives.
After this operation, 40.2 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 xxd amd64 2:8.0.1453-1ubuntu1.8 [49.9 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 vim-common all 2:8.0.1453-1ubuntu1.8 [71.1 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 git-lfs amd64 2.3.4-1 [2,129 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgpm2 amd64 1.20.7-5 [15.1 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 vim-runtime all 2:8.0.1453-1ubuntu1.8 [5,435 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 vim amd64 2:8.0.1453-1ubuntu1.8 [1,154 kB]
Fetched 8,854 kB in 1s (7,299 kB/s)
Selecting previously unselected package xxd.
(Reading database ... 155113 files and directories currently installed.)
Preparing to unpack .../0-xxd_2%3a8.0.1453-1ubuntu1.8_amd64.deb ...
Unpacking xxd (2:8.0.1453-1ubuntu1.8) ...
Selecting previously unselected package vim-common.
Preparing to unpack .../1-vim-common_2%3a8.0.1453-1ubuntu1.8_all.deb ...
Unpacking vim-common (2:8.0.1453-1ubuntu1.8) ...
Selecting previously unselected package git-lfs.
Preparing to unpack .../2-git-lfs_2.3.4-1_amd64.deb ...
Unpacking git-lfs (2.3.4-1) ...
Selecting previously unselected package libgpm2:amd64.
Preparing to unpack .../3-libgpm2_1.20.7-5_amd64.deb ...
Unpacking libgpm2:amd64 (1.20.7-5) ...
Selecting previously unselected package vim-runtime.
Preparing to unpack .../4-vim-runtime_2%3a8.0.1453-1ubuntu1.8_all.deb ...
Adding 'diversion of /usr/share/vim/vim80/doc/help.txt to /usr/share/vim/vim80/doc/help.txt.vim-tiny by vim-runtime'
Adding 'diversion of /usr/share/vim/vim80/doc/tags to /usr/share/vim/vim80/doc/tags.vim-tiny by vim-runtime'
Unpacking vim-runtime (2:8.0.1453-1ubuntu1.8) ...
Selecting previously unselected package vim.
Preparing to unpack .../5-vim_2%3a8.0.1453-1ubuntu1.8_amd64.deb ...
Unpacking vim (2:8.0.1453-1ubuntu1.8) ...
Setting up git-lfs (2.3.4-1) ...
Setting up xxd (2:8.0.1453-1ubuntu1.8) ...
Setting up libgpm2:amd64 (1.20.7-5) ...
Setting up vim-common (2:8.0.1453-1ubuntu1.8) ...
Setting up vim-runtime (2:8.0.1453-1ubuntu1.8) ...
Setting up vim (2:8.0.1453-1ubuntu1.8) ...
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vim (vim) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vimdiff (vimdiff) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/rvim (rvim) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/rview (rview) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vi (vi) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/view (view) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/ex (ex) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/editor (editor) in auto mode
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for mime-support (3.60ubuntu1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.3) ...
/sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link

Requirement already satisfied: scikit-image in /usr/local/lib/python3.7/dist-packages (0.18.3)
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (1.3.5)
Collecting timeout-decorator==0.5.0
  Downloading timeout-decorator-0.5.0.tar.gz (4.8 kB)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (1.21.5)
Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image) (2.4.1)
Requirement already satisfied: pillow!=7.1.0,!=7.1.1,>=4.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image) (7.1.2)
Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.7/dist-packages (from scikit-image) (2021.11.2)
Requirement already satisfied: scipy>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from scikit-image) (1.4.1)
Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from scikit-image) (1.2.0)
Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image) (2.6.3)
Requirement already satisfied: matplotlib!=3.0.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image) (3.2.2)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image) (3.0.7)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image) (1.3.2)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image) (2.8.2)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image) (0.11.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib!=3.0.0,>=2.0.0->scikit-image) (1.15.0)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas) (2018.9)
Building wheels for collected packages: timeout-decorator
  Building wheel for timeout-decorator (setup.py) ... done
  Created wheel for timeout-decorator: filename=timeout_decorator-0.5.0-py3-none-any.whl size=5028 sha256=26a1e218629ba4c5771281c9907d578b966d5323bec6cac22d06d02d8ae409a5
  Stored in directory: /root/.cache/pip/wheels/7d/64/ac/de1dd54f9a6e48b846e9cb5e4176d6f063380e7f83d69807ad
Successfully built timeout-decorator
Installing collected packages: timeout-decorator
Successfully installed timeout-decorator-0.5.0

b) Load Dataset

The directory sturcture at this point looks like this:


Quick preview of images and labels.csv is as follows:

Let's initialise dataset instances.

In [4]:
from evaluator.dataset import ZEWDPCBaseDataset, ZEWDPCProtectedDataset
DATASET_SHUFFLE_SEED = 1022022

# Instantiate Training Dataset
training_dataset = ZEWDPCBaseDataset(
    images_dir="./data/debug/images",
    labels_path="./data/debug/labels.csv",
    shuffle_seed=DATASET_SHUFFLE_SEED,
)
# Instantiate Unlabelled Dataset
unlabelled_dataset = ZEWDPCProtectedDataset(
    images_dir="./data/debug/images",
    labels_path="./data/debug/labels.csv",
    budget=3000,  # Configurable Parameter
    shuffle_seed=DATASET_SHUFFLE_SEED,
)
# Instantiate Validation Dataset
val_dataset = ZEWDPCBaseDataset(
    images_dir="./data/debug/images",
    labels_path="./data/debug/labels.csv",
    drop_labels=True,
    shuffle_seed=DATASET_SHUFFLE_SEED,
)
val_dataset_gt = ZEWDPCBaseDataset(
    images_dir="./data/debug/images",
    labels_path="./data/debug/labels.csv",
    drop_labels=False,
    shuffle_seed=DATASET_SHUFFLE_SEED,
)

c) pre_training_phase

Pre-train your model on the available labelled dataset here.

Hook for the Pre-Training Phase of the Competition, where you have access to a training_dataset, an instance of the ZEWDPCBaseDataset class (see dataset.py for more details).

You are allowed to pre-train on this data while you prepare for the purchase phase of the competition.

If you train some models, you can instantiate them as self.model, as long as you implement self-contained checkpointing in the self.save_checkpoint and self.load_checkpoint hooks, as the hooks for the different phases of the competition, can be called in other executions of the BaseRun.

Base code

In [5]:
import torch
from torch import nn
from torchvision import models
from torch.optim import Adam, SGD, lr_scheduler
from torchvision import transforms
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter
import numpy as np
import abc
import datetime
from tqdm import tqdm
from sklearn.metrics import accuracy_score
from sklearn.metrics import hamming_loss

from evaluator.dataset import ZEWDPCBaseDataset, ZEWDPCProtectedDataset
In [6]:
class ResNet101(nn.Module):

    def __init__(self, num_labels):
        super(ResNet101, self).__init__()
        self.network = models.resnet101(pretrained=False, num_classes=num_labels)

        model_dict = self.network.state_dict()
        self.network.load_state_dict(model_dict)
        
        for param in self.network.parameters():
            param.requires_grad = False
        for param in self.network.layer4.parameters():
            param.requires_grad = True
        self.network.fc.requires_grad = True

    def forward(self, x):
        x = self.network(x)
        return x

class AverageMeter(object):
    def __init__(self, num_classes):
        super(AverageMeter, self).__init__()
        self.num_classes = num_classes

    def reset(self):
        self._right_pred_counter = np.zeros(self.num_classes)  # right predicted image per-class counter
        self._pred_counter = np.zeros(self.num_classes)    # predicted image per-class counter
        self._gt_counter = np.zeros(self.num_classes)  # ground-truth image per-class counter

    def update(self, confidence, gt_label):
        self._count(confidence, gt_label)

    def compute(self):
        self._op = sum(self._right_pred_counter) / sum(self._pred_counter)
        self._or = sum(self._right_pred_counter) / sum(self._gt_counter)
        self._of1 = 2 * self._op * self._or / (self._op + self._or)
        self._right_pred_counter = np.maximum(self._right_pred_counter, np.finfo(np.float64).eps)
        self._pred_counter = np.maximum(self._pred_counter, np.finfo(np.float64).eps)
        self._gt_counter = np.maximum(self._gt_counter, np.finfo(np.float64).eps)
        self._cp = np.mean(self._right_pred_counter / self._pred_counter)
        self._cr = np.mean(self._right_pred_counter / self._gt_counter)
        self._cf1 = 2 * self._cp * self._cr / (self._cp + self._cr)

    @abc.abstractmethod
    def _count(self, confidence, gt_label):
        pass

    @property
    def op(self):   # overall precision
        return self._op

    @property   # overall recall
    def or_(self):
        return self._or

    @property   # overall F1
    def of1(self):
        return self._of1

    @property   # per-class precision
    def cp(self):
        return self._cp

    @property   # per-class recall
    def cr(self):
        return self._cr

    @property   # per-class F1
    def cf1(self):
        return self._cf1

Training class

In [7]:
# Install wandb
!pip install wandb
Collecting wandb
  Downloading wandb-0.12.10-py2.py3-none-any.whl (1.7 MB)
     |████████████████████████████████| 1.7 MB 4.8 MB/s 
Collecting pathtools
  Downloading pathtools-0.1.2.tar.gz (11 kB)
Requirement already satisfied: PyYAML in /usr/local/lib/python3.7/dist-packages (from wandb) (3.13)
Requirement already satisfied: Click!=8.0.0,>=7.0 in /usr/local/lib/python3.7/dist-packages (from wandb) (7.1.2)
Requirement already satisfied: protobuf>=3.12.0 in /usr/local/lib/python3.7/dist-packages (from wandb) (3.17.3)
Requirement already satisfied: six>=1.13.0 in /usr/local/lib/python3.7/dist-packages (from wandb) (1.15.0)
Requirement already satisfied: psutil>=5.0.0 in /usr/local/lib/python3.7/dist-packages (from wandb) (5.4.8)
Collecting sentry-sdk>=1.0.0
  Downloading sentry_sdk-1.5.5-py2.py3-none-any.whl (144 kB)
     |████████████████████████████████| 144 kB 48.9 MB/s 
Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.7/dist-packages (from wandb) (2.8.2)
Collecting docker-pycreds>=0.4.0
  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)
Requirement already satisfied: promise<3,>=2.0 in /usr/local/lib/python3.7/dist-packages (from wandb) (2.3)
Requirement already satisfied: requests<3,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from wandb) (2.27.1)
Collecting shortuuid>=0.5.0
  Downloading shortuuid-1.0.8-py3-none-any.whl (9.5 kB)
Collecting yaspin>=1.0.0
  Downloading yaspin-2.1.0-py3-none-any.whl (18 kB)
Requirement already satisfied: GitPython>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from wandb) (3.1.18)
Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.7/dist-packages (from GitPython>=1.0.0->wandb) (4.0.9)
Requirement already satisfied: typing-extensions>=3.7.4.0 in /usr/local/lib/python3.7/dist-packages (from GitPython>=1.0.0->wandb) (3.10.0.2)
Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.7/dist-packages (from gitdb<5,>=4.0.1->GitPython>=1.0.0->wandb) (5.0.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.0.0->wandb) (1.24.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.0.0->wandb) (2021.10.8)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.0.0->wandb) (2.10)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.0.0->wandb) (2.0.11)
Requirement already satisfied: termcolor<2.0.0,>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from yaspin>=1.0.0->wandb) (1.1.0)
Building wheels for collected packages: pathtools
  Building wheel for pathtools (setup.py) ... done
  Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8806 sha256=bfc30b9810374f20fb40ef3214b800d7fdb4fe9fd58a88e0696a432cb931f9a4
  Stored in directory: /root/.cache/pip/wheels/3e/31/09/fa59cef12cdcfecc627b3d24273699f390e71828921b2cbba2
Successfully built pathtools
Installing collected packages: yaspin, shortuuid, sentry-sdk, pathtools, docker-pycreds, wandb
Successfully installed docker-pycreds-0.4.0 pathtools-0.1.2 sentry-sdk-1.5.5 shortuuid-1.0.8 wandb-0.12.10 yaspin-2.1.0
In [10]:
import wandb

class ZEWDPCBaseRun:

    def __init__(self):
        self.evaluation_state = {}
        # Model parameters
        self.BATCH_SIZE = 32
        self.NUM_WORKERS = 2
        self.LEARNING_RATE = 0.001
        self.NUM_CLASSES = 4
        self.TOPK= 3
        self.THRESHOLD = 0.5
        self.NUM_EPOCS = 50
        self.EVAL_FREQ = 5

        self.model = ResNet101(num_labels = self.NUM_CLASSES)
        ## CHANGE CPU CUDA HERE
        # self.model.cuda()
        self.model.cpu()

        self.trainable_parameters = filter(lambda param: param.requires_grad, self.model.parameters())
        self.optimizer = Adam(self.trainable_parameters, lr=self.LEARNING_RATE)
        
        self.epoch = 0

        self.lr_scheduler_ = lr_scheduler.ReduceLROnPlateau(
            self.optimizer, mode='max', patience=2, verbose=True
        )
        self.criterion = nn.BCEWithLogitsLoss()

        # WandB setup
        model_name = "ResNet101"
        wandb_config={
            "model": model_name,
            "learning_rate": self.LEARNING_RATE,
            "samples_per_gpu": self.BATCH_SIZE,
            "workers_per_gpu": self.NUM_WORKERS,
            "augmentations": "NO",
            "description": "Pre-Training. No augmentation"
        }
        wandb.init(project="temp", entity="krazy", name=model_name, config=wandb_config)
        

    def pre_training_phase(
        self, training_dataset: ZEWDPCBaseDataset, register_progress=lambda x: False
    ):
        print("\n================> Pre-Training Phase\n")
        # Creating transformations
        train_transform = transforms.Compose([
            transforms.ToTensor(),
        ])
        training_dataset.set_transform(train_transform)
        train_loader = DataLoader(
            dataset=training_dataset,
            batch_size=self.BATCH_SIZE,
            shuffle=False,
            num_workers=self.NUM_WORKERS,
        )

        def run_epoch():
            for _, batch in enumerate(train_loader):

                ## CHANGE CPU CUDA HERE
                # x, y = batch["image"].cuda(), batch["label"]
                x, y = batch["image"].cpu(), batch["label"]

                pred_y = self.model(x)
                # Change the shape of true labels here. Because for last batch the no. of images can be less
                y = torch.cat(y, dim=0).reshape(
                    self.NUM_CLASSES, pred_y.shape[0]
                    ).T.type(torch.FloatTensor)
                
                ## CHANGE CPU CUDA HERE. Comment for CPU
                # y = y.cuda()
                loss = self.criterion(pred_y, y)
                self.optimizer.zero_grad()
                loss.backward()
                self.optimizer.step()

                # 416 = BATCH_SIZE*13
                if self.global_step % 416 == 0:
                    wandb.log({
                        "Step": (self.epoch*5000)+self.global_step,
                        "Loss": loss,
                    })
                    print("[{}] Training [epoch {}, step {}], loss: {:4f}".format(
                        datetime.datetime.now(), self.epoch, self.global_step, loss))
                self.global_step += self.BATCH_SIZE
        
        epoch_range = tqdm(range(self.epoch, self.NUM_EPOCS))
        for i in epoch_range:
            epoch_range.set_description(f"Epoch: {i}")
            self.global_step = 0
            run_epoch()
            register_progress(i) # Epoch as progress
            if (i+1)%self.EVAL_FREQ == 0:
                predictions = self.prediction_phase(val_dataset)
                self.evaluation(predictions)
            self.epoch += 1
        print("Execution Complete of Training Phase.")


    def purchase_phase(
        self,
        unlabelled_dataset: ZEWDPCProtectedDataset,
        training_dataset: ZEWDPCBaseDataset,
        budget=1000,
        register_progress=lambda x: False,
    ):
        """
        # Purchase Phase
        -------------------------
        In this phase of the competition, you have access to
        the unlabelled_dataset (an instance of `ZEWDPCProtectedDataset`)
        and the training_dataset (an instance of `ZEWDPCBaseDataset`)
        {see datasets.py for more details}, and a purchase budget.

        You can iterate over both the datasets and access the images without restrictions.
        However, you can probe the labels of the unlabelled_dataset only until you
        run out of the label purchasing budget.

        PARTICIPANT_TODO: Add your code here
        """
        print("\n================> Purchase Phase | Budget = {}\n".format(budget))

        register_progress(0.0) #Register Progress
        
        for sample in tqdm(unlabelled_dataset):
            idx = sample["idx"]
            
            # image = unlabelled_dataset.__getitem__(idx)
            # print(image)

            # Budgeting & Purchasing Labels
            if budget > 0:
                label = unlabelled_dataset.purchase_label(idx)

            budget -= 1
        register_progress(1.0) #Register Progress
        print("Execution Complete of Purchase Phase.")

    def prediction_phase(
        self,
        test_dataset: ZEWDPCBaseDataset,
        register_progress=lambda x: False,
    ):
        """
        # Prediction Phase
        -------------------------
        In this phase of the competition, you have access to the test dataset, and you
        are supposed to make predictions using your trained models.

        Returns:
            np.ndarray of shape (n, 4)
                where n is the number of samples in the test set
                and 4 refers to the 4 labels to be predicted for each sample
                for the multi-label classification problem.

        PARTICIPANT_TODO: Add your code here
        """
        print(
            "\n================> Prediction Phase : - on {} images\n".format(
                len(test_dataset)
            )
        )
        
        test_transform = transforms.Compose([
            transforms.ToTensor(),
        ])
        test_dataset.set_transform(test_transform)
        test_loader = DataLoader(
            dataset=test_dataset,
            batch_size=self.BATCH_SIZE,
            shuffle=False,
            num_workers=self.NUM_WORKERS,
        )
    
        def convert_to_label(preds):
            return np.array((torch.sigmoid(preds) > 0.5), dtype=int).tolist()

        predictions = []
        self.model.eval()
        with torch.no_grad():
            for _, batch in enumerate(test_loader):
                ## CHANGE CPU CUDA HERE
                X= batch['image'].cpu()
                # X = batch['image'].cuda()

                pred_y = self.model(X)

                # Convert to labels
                pred_y_labels = []
                for arr in pred_y:
                    ## CHANGE CPU CUDA HERE
                    # pred_y_labels.append(convert_to_label(arr.cpu())) # For CUDA
                    pred_y_labels.append(convert_to_label(arr)) # For CPU

                # Save the results
                predictions.extend(pred_y_labels)
            register_progress(1.0)

        predictions = np.array(predictions) # random predictions
        print("Execution Complete of Purchase Phase.")
        return predictions

    def evaluation(self, predictions):
        from evaluator.evaluation_metrics import accuracy_score, hamming_loss, exact_match_ratio

        y_true = val_dataset_gt._get_all_labels()
        y_pred = predictions

        accuracy_score = accuracy_score(y_true, y_pred)
        hamming_loss_score = hamming_loss(y_true, y_pred)
        exact_match_ratio_score = exact_match_ratio(y_true, y_pred)

        wandb.log({
            "Epoch": self.epoch+1,
            "Accuracy": accuracy_score,
            "Hamming Loss": hamming_loss_score,
            "Match ratio": exact_match_ratio_score
        })

        print("Accuracy Score : ", accuracy_score)
        print("Hamming Loss : ", hamming_loss_score)
        print("Exact Match Ratio : ", exact_match_ratio_score)

    def save_checkpoint(self, checkpoint_path):
        """
        Saves the checkpoint in the checkpoint_path directory. Each checkpoint will be saved for epoch_x
        """
        save_dict = {
            'epoch': self.epoch + 1,
            'model_state_dict': self.model.state_dict(),
            'optim_state_dict': self.optimizer.state_dict(),
        }
        torch.save(save_dict, checkpoint_path)
        print(f"Checkpont epoch:{self.epoch} Model saved at {checkpoint_path}")

    def load_checkpoint(self, checkpoint_path):
        """
        Load the latest checkpoint from the experiment
        """
        ## CHANGE CPU CUDA HERE
        # checkpoint_model = torch.load(checkpoint_path, map_location="cuda:0")
        checkpoint_model = torch.load(checkpoint_path, map_location="cpu")
        
        self.latest_epoch = checkpoint_model['epoch']
        self.model.load_state_dict(checkpoint_model['model_state_dict'])
        self.optimizer.load_state_dict(checkpoint_model['optim_state_dict'])
        print('loading checkpoint success (epoch {})'.format(self.latest_epoch))
In [ ]:
import tempfile
checkpoint_path = tempfile.NamedTemporaryFile(delete=False).name

# checkpoint_path = "/content/drive/MyDrive/data-purchasing-challenge-2022-starter-kit/experiments/baseline/debug.pt"

run = ZEWDPCBaseRun()
## Pre - Training process
run.pre_training_phase(training_dataset)
run.save_checkpoint(checkpoint_path)
del run

# ## Purchasing phase
run = ZEWDPCBaseRun()
run.load_checkpoint(checkpoint_path)
run.purchase_phase(unlabelled_dataset, training_dataset, budget=3000)
run.save_checkpoint(checkpoint_path)
del run

## Prediction phase
run = ZEWDPCBaseRun()
run.load_checkpoint(checkpoint_path)
predictions = run.prediction_phase(val_dataset)
assert type(predictions) == np.ndarray
assert predictions.shape == (len(val_dataset), 4)

## Evaluation Phase
from evaluator.evaluation_metrics import accuracy_score, hamming_loss, exact_match_ratio

y_true = val_dataset_gt._get_all_labels()
y_pred = predictions

accuracy_score = accuracy_score(y_true, y_pred)
hamming_loss_score = hamming_loss(y_true, y_pred)
exact_match_ratio_score = exact_match_ratio(y_true, y_pred)

print("Accuracy Score : ", accuracy_score)
print("Hamming Loss : ", hamming_loss_score)
print("Exact Match Ratio : ", exact_match_ratio_score)

Comments

You must login before you can post a comment.

Execute