#### Data Purchasing Challenge 2022

# Explainability: How does your model actually learn?

For a given sample, what features contribute to the decision? How to improve your model.

Explainability is concerned with finding out what features contribute to the decision of a model for a given example. Essentially, we are interested in finding out how the chosen parameters effect the decision process of the model. Instead of only considering the output, we will also look into the intermediate layers of the model and try to visualise them. The tool that will help us achieving this goal is called SHAP (SHapley Additive exPlanations). The main goal is to shed light on to a black box model, such as neural network, and try to find out how to improve the model and its decision making process.

# Explainability: How does your model actually learn?¶

Neural networks are obviously widely used. They have been applied successfully in various areas, such as computer vision, natural language processing and robotics, and contributed substantially to improving these areas. Unfortunately, many people take neural networks for granted and they don't pay much attention to the underlying network and how it really learns. It is viewed as a black box model. Luckily, there are tools out there that help investigate, or rather visualise how the model actually learns. Knowing how your chosen parameters effect your model will help you greatly in improving it.

## Data Purchasing Challenge¶

A key point in this challenge is to learn a good representation of the given training images. The number of the training images are kept small. Learning a CNN from scratch may constitute a big challenge due to the small number of available training images. Naturally, a pre-trained model on a much larger dataset will come in handy. Torchvision provides us with a decent amount of pre-trained architectures. Before randomly picking one of the pre-trained architectures, you should ask yourself if there is a more reliable strategy than just randomly picking any available architecture from the given list. At the end you may be wasting a lot of valuable time picking the *wrong* pre-trained model. In short being able to answer some of the following questions will greatly help you in improving your model, or find some hidden issues in your approach and finally, this may lead to a significant improvement.

- What do the various layers of the pre-trained model focus on?
- How do different optimizers (e.g. Adam, SGD,...) effect the model and therefore the final result?
- How does the learning rate effect the model and the result?
- Which layers of the pre-trained model should be held frozen aka no weight updates?
- How do different loss functions effect the various layers of the model?
- How does the number of epochs effect the various layers?
- ...

These are sort of question a ML engineer may ask and should be able to answer these questions to certain extent. In the following chapter I will introduce you a neat tool called SHAP (SHapley Additive exPlanations) which can help you in answering some of the questions.

## SHAP¶

Quote: ***SHAP (SHapley Additive exPlanations)** is a game theoretic approach to explain the output of any machine learning model. It connects optimal credit allocation with local explanations using the classic Shapley values from game theory and their related extensions.*

SHAP provides many tools to visualise a ML algorithm. The one we focuse on here is called the **GradientExplainer**

### Load Dependencies¶

```
num_threads = 8
import math
import os
import random
os.environ["OMP_NUM_THREADS"] = f"{num_threads}"
os.environ["OPENBLAS_NUM_THREADS"] = f"{num_threads}"
os.environ["MKL_NUM_THREADS"] = f"{num_threads}"
os.environ["VECLIB_MAXIMUM_THREADS"] = f"{num_threads}"
os.environ["NUMEXPR_NUM_THREADS"] = f"{num_threads}"
import time
import numpy as np
import pandas as pd
import shap
import torch
import torch.nn as nn
from torchvision import models, transforms
from evaluator.dataset import ZEWDPCBaseDataset
from evaluator.evaluation_metrics import accuracy_score, hamming_loss, exact_match_ratio
seed = 42
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.set_num_threads(num_threads)
```

### Helper function¶

```
# This function's only purpose is to encapsulate the training procedure
def train(training_dataset: ZEWDPCBaseDataset, train_idx, val_idx, model, optimizer, batch_size, epochs, criterion, device):
start_time = time.time()
n_train_it, n_val_it = math.ceil(train_idx.shape[0] / batch_size), math.ceil(val_idx.shape[0] / batch_size)
for epoch in range(epochs):
model.train()
train_predictions, val_predictions = [], []
random_idx = np.random.permutation(train_idx.shape[0])
train_idx = train_idx[random_idx]
for batch in range(n_train_it):
optimizer.zero_grad()
data, y_true = [], []
for i in train_idx[batch*batch_size:(batch+1)*batch_size]:
sample = training_dataset[i]
data.append(sample['image'])
y_true.append(sample['label'])
data = torch.stack(data, dim=0).to(device)
y_true = torch.tensor(y_true).to(device)
y_preds = model(data)
train_predictions.append(y_preds.cpu().detach())
loss = criterion(y_preds, y_true)
loss.backward()
optimizer.step()
# print(f"==> [Train] Epoch {epoch+1}/{epochs} | Batch {batch+1}/{n_train_it} | Loss {loss:.6f} | Passed time {(time.time() - start_time)/60:.2f}")
with torch.no_grad():
train_predictions = torch.concat(train_predictions, dim=0)
y_true = torch.from_numpy(training_dataset._get_all_labels()[train_idx]).float()
loss = criterion(train_predictions, y_true)
train_predictions[train_predictions <= .5] = 0
train_predictions[train_predictions > .5] = 1
acc = accuracy_score(training_dataset._get_all_labels()[train_idx], train_predictions)
print(f"==> [Train] Epoch {epoch+1}/{epochs} | Loss {loss:.6f} | Acc {acc:.3f} | Passed time {(time.time() - start_time)/60:.2f} min.")
model.eval()
for batch in range(n_val_it):
data, y_true = [], []
for i in val_idx[batch*batch_size:(batch+1)*batch_size]:
sample = training_dataset[i]
data.append(sample['image'])
y_true.append(sample['label'])
data = torch.stack(data, dim=0).to(device)
y_true = torch.tensor(y_true).to(device)
y_preds = model(data)
val_predictions.append(y_preds.cpu())
loss = criterion(y_preds, y_true)
# print(f"==> [Val] Epoch {epoch+1}/{epochs} | Batch {batch+1}/{n_train_it} | Loss {loss:.6f} | Passed time {(time.time() - start_time)/60:.2f}")
val_predictions = torch.concat(val_predictions, dim=0)
y_true = torch.from_numpy(training_dataset._get_all_labels()[val_idx])
loss = criterion(val_predictions, y_true)
val_predictions[val_predictions <= .5] = 0
val_predictions[val_predictions > .5] = 1
acc = accuracy_score(training_dataset._get_all_labels()[val_idx], val_predictions)
print(f"==> [Val] Epoch {epoch+1}/{epochs} | Loss {loss:.6f} | Acc {acc:.3f} | Passed time {(time.time() - start_time)/60:.2f} min.")
```

```
def normalize_data(image):
# make sure the pixel values are within the interval [0,1]
if image.max() > 1:
image /= 255
image = (image - mean.numpy()) / std.numpy()
return torch.tensor(image.swapaxes(-1, 1).swapaxes(2, 3)).float()
```

### Set Parameters¶

```
device = 'cuda:0' if torch.cuda.is_available else 'cpu'
device = torch.device(device)
```

### Load Dataset¶

```
mean, std = torch.tensor([0.485, 0.456, 0.406]), torch.tensor([0.229, 0.224, 0.225])
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize(mean=mean, std=std)
])
class_names = np.array(['scratch_small', 'scratch_large', 'dent_small', 'dent_large'])
training_dataset = ZEWDPCBaseDataset(
images_dir="./data/training/images",
labels_path="./data/training/labels.csv",
shuffle_seed=seed,
transform=transform
)
n_samples = len(training_dataset)
random_idx = np.random.permutation(n_samples)
train_idx, val_idx = random_idx[:math.floor(n_samples*.9)], random_idx[math.floor(n_samples*.9):]
```

### Load Torchvision's Pre-Trained Model: ResNet18¶

```
model = models.resnet18(True, False)
model.fc = nn.Linear(512, 4)
model = model.to(device)
```

### Train ResNet18¶

```
optimizer, batch_size, epochs, criterion = torch.optim.Adam(model.parameters(), lr=1e-3), 32, 5, nn.MultiLabelSoftMarginLoss()
train(training_dataset, train_idx, val_idx, model, optimizer, batch_size, epochs, criterion, device)
```

### Prepare Validation Data for Visualisation with SHAP¶

```
shap.initjs()
```

```
X_val, y_val, filename =[], [], []
for i in val_idx:
sample = training_dataset[i]
X_val.append(sample['image'])
y_val.append(sample['label'])
filename.append(training_dataset._get_filename(i))
X_val = torch.stack(X_val, dim=0)*std.reshape((1,-1, 1, 1)) + mean.reshape((1, -1, 1, 1))
X_val = X_val.permute(0,2,3,1).numpy()
```

### Visualise Trained ResNet18 Model with SHAP¶

```
model = model.to(torch.device('cpu'))
```

#### ResNet18 First Layer Visualisation¶

```
shap_gradient_explainer = shap.GradientExplainer((model, model.layer1), normalize_data(X_val))
```

```
shap_values, index = shap_gradient_explainer.shap_values(torch.from_numpy(X_val[122:125]).permute(0,3,1,2).float(), ranked_outputs=4, nsamples=100)
```

```
shap_values = [np.swapaxes(np.swapaxes(s, 2, 3), 1, -1) for s in shap_values]
```

```
shap.image_plot(shap_values, X_val[122:125], class_names[index])
```

In the case of the first sample, the first layer focuses on the pronounced dent with some dispersion around it. In the case of the second one it focuses on the various scratches and in the last case it is rather dispersed with no real focus as there is nothing to focus on.

#### ResNet18 Second Layer Visualisation¶

```
shap_gradient_explainer = shap.GradientExplainer((model, model.layer2), normalize_data(X_val))
```

```
shap_values, index = shap_gradient_explainer.shap_values(torch.from_numpy(X_val[122:125]).permute(0,3,1,2).float(), ranked_outputs=4, nsamples=100)
```

```
shap_values = [np.swapaxes(np.swapaxes(s, 2, 3), 1, -1) for s in shap_values]
```

```
shap.image_plot(shap_values, X_val[122:125], class_names[index])
```