Loading

ADDI Alzheimers Detection Challenge

What about neural networks? (0.694 LogLoss , 0.497 F1)

Trying out neural networks, imputing NaN values with KNNs, and exploring the data!

mmi333

An exploration of what neural networks can bring to the competition.

Training a Nerual Network (and other stuff)

Table of contents:

  1. Quick EDA & Discussion
  2. Impute the NaNs!
  3. Neural Networks!
  4. Cross validation.
  5. Submit!

Before we begin, a quick tip when working with ADDI workspaces.
There's a clipboard button on the top right that allows you to copy things from local clipboard to workspace clipboard so you don't have to type everything manually.
Obvious, right? I discovered that too late. Maybe it will save someone's time!

First, let's categorize the problem:

  1. It's a 3-class classifcation problem
  2. with class imbalance
  3. lots of missing values

The best solutions will most likely be big ensembles with good cross-validation (don't overfit to the leaderboard!) and some smart feature engineering.

It's also always a good idea to check top solutions for previous competitions with similar properties, here are some:

  1. Class imbalance (Code for winning solutions available)
  2. Extreme Class imbalance
  3. Multiclass classification

We'll be using a library called deeptables for the neural network. It's made for fitting deep learning models on tabular data fast and with a lot of options. Let's try it!

Define preprocessing code 💻

The code that is common between the training and the prediction sections should be defined here. During evaluation, we completely skip the training section. Please make sure to add any common logic between the training and prediction sections here.

Import common packages

Please import packages that are common for training and prediction phases here.

In [1]:
!pip install -q -U aicrowd-cli numpy pandas scikit-learn deeptables
In [2]:
%load_ext aicrowd.magic
In [3]:
import os

# Please use the absolute for the location of the dataset.
# Or you can use relative path with `os.getcwd() + "test_data/validation.csv"`
AICROWD_DATASET_PATH = os.getenv("DATASET_PATH", "/ds_shared_drive/validation.csv")
AICROWD_PREDICTIONS_PATH = os.getenv("PREDICTIONS_PATH", "predictions.csv")
AICROWD_ASSETS_DIR = "assets"
AICROWD_API_KEY = "" # Get your key from https://www.aicrowd.com/participants/me
In [4]:
import joblib
import numpy as np
import pandas as pd
import seaborn as sns
from matplotlib import pyplot as plt
from sklearn.metrics import log_loss
from sklearn.impute import KNNImputer
from deeptables.models import deeptable
from sklearn.model_selection import StratifiedKFold

Training phase ⚙️

You can define your training code here. This sections will be skipped during evaluation.

Load training data

Let's start by loading the data and encoding the diagnosis labels.

In [5]:
train = pd.read_csv(AICROWD_DATASET_PATH.replace("validation", "train"))
valid = pd.read_csv(AICROWD_DATASET_PATH)
valid_gt = pd.read_csv(AICROWD_DATASET_PATH.replace("validation", "validation_ground_truth"))
train.loc[train.diagnosis == "normal", "diagnosis"] = 0
train.loc[train.diagnosis == "pre_alzheimer", "diagnosis"] = 1
train.loc[train.diagnosis == "post_alzheimer", "diagnosis"] = 2


valid_gt.loc[valid_gt.diagnosis == "normal", "diagnosis"] = 0
valid_gt.loc[valid_gt.diagnosis == "pre_alzheimer", "diagnosis"] = 1
valid_gt.loc[valid_gt.diagnosis == "post_alzheimer", "diagnosis"] = 2

Now let's do a quick EDA and start with the NaN values


We'll see how many NaN values are there per row and per column:
In [6]:
null_row_count = train.isna().sum(axis=1).value_counts()
null_row_count = null_row_count.sort_values(ascending=False).head(20)
fig = sns.barplot(x=null_row_count.index, y=null_row_count)
plt.xlabel("Number of NaNs")
plt.ylabel("Number of rows")
plt.title("Number of NaNs per row")
plt.show(fig)
In [7]:
#We'll replace column names with their indices for easier plots
train_ind_cols = train.copy() 
train_ind_cols.columns = [x for x in range(len(train.columns))] 

null_column_count = train_ind_cols.isna().sum(axis=0)

null_column_count = null_column_count.sort_values(ascending=False).head(20)
fig = sns.barplot(x=null_column_count.index, y=null_column_count)
plt.xlabel("Column Index")
plt.ylabel("Number of NaNs in column")
plt.title("Number of NaNs per column")
plt.show(fig)
In [8]:
#And here are the names of the columns
pd.Series(train.iloc[:,null_column_count.index].columns, index = null_column_count.index)
Out[8]:
82     between_digits_angle_ccw_sum
90               single_hand_length
97          minute_proximity_from_2
80      between_digits_angle_cw_sum
96           hour_proximity_from_11
119                       time_diff
91                  clockhand_ratio
98              hour_pointing_digit
100           minute_pointing_digit
94            deviation_from_centre
120               centre_dot_detect
92                   clockhand_diff
93              angle_between_hands
95      intersection_pos_rel_centre
88                 hour_hand_length
89               minute_hand_length
26                 euc_dist_digit_1
50                   height_digit_1
38                     area_digit_1
62                    width_digit_1
dtype: object

As we can see, there are lots of nan values in each row and column. There are many ways to deal with them. We'll try here to use KNNImputer on continuous values, where it basically replaces NaNs with the closest value based on other rows. For categorical columns (we'll define it as less than 20 unique values) we'll just fill by most frequent category.

Now let's take a look at the number of digits drawn for each class:

In [9]:
#percentage of each class with at least one missing digit
(train.number_of_digits[train.number_of_digits < 12].groupby(train.diagnosis).count() / [train.diagnosis.value_counts()[0],train.diagnosis.value_counts()[1],train.diagnosis.value_counts()[2]]) * 100
Out[9]:
diagnosis
0    58.526660
1    82.380952
2    85.813751
Name: number_of_digits, dtype: float64

More than half of the normal category (0) have at least 1 missing digit!

This is a reminder that normal is not normal. As mentioned in the competition overview, normal means "not an Alzheimer's patient", and it doesn't rule out other neurodegenerative diseases. This could help when doing feature engineering.

Train your model

Now let's do some preprocessing. We'll use the KNNImputer implementation from sklearn for continuous values, and fill categorical values with the most common value.

In [10]:
most_commons = []
continuous,categorical = [],[]

for i in train.columns[1:-1]:
    #Define categorical as less than 20 unique values
    if len(train[i].unique()) < 20:
        categorical.append(i)
    else:
        continuous.append(i)
        
        
for i in categorical:
    most_common = train[i].value_counts().keys()[0]
    most_commons.append(most_common)
    train[i] = train[i].fillna(most_common)
imputer = KNNImputer() #Here's the fun part (It takes some time)
train[continuous] = imputer.fit_transform(train[continuous])

#One-hot encode the only non-numeric value we have
one_hot = pd.get_dummies(train["intersection_pos_rel_centre"])
train = train.drop("intersection_pos_rel_centre", axis=1)
train = train.join(one_hot)

meta_train = [categorical,continuous,most_commons]

for i,col in enumerate(categorical): 
    #We'll fill valid and test with most common in train
    valid[col] = valid[col].fillna(most_commons[i])
    
imputer = KNNImputer()
valid[continuous] = imputer.fit_transform(valid[continuous])

one_hot = pd.get_dummies(valid["intersection_pos_rel_centre"])
valid = valid.drop("intersection_pos_rel_centre", axis=1)
valid = valid.join(one_hot)

A little bit of feature engineering: Add variance of "euc_distance" and "dist from cent" columns

In [11]:
m = [col for col in train.columns if col.endswith("dist from cen")]
train["dist from cen_var"] = train[m].var(axis=1)
m = [col for col in train.columns if col.startswith("euc_dist_digit_")]
train["euc_dist_digit_var"] = train[m].var(axis=1)

m = [col for col in valid.columns if col.endswith("dist from cen")]
valid["dist from cen_var"] = valid[m].var(axis=1)
m = [col for col in valid.columns if col.startswith("euc_dist_digit_")]
valid["euc_dist_digit_var"] = valid[m].var(axis=1)

We have arrived at the fun part!

We'll use the apply_class_weight parameter to let deeptables deal with the imbalance for us!
For cross validation we'll do 10 folds and also we'll try predicting and scoring the valid set.
You can try experimenting with different nets, hyperparameters, etc. from the documentation.
There's also a fit_cross_validation function available you can try out!

There are lots of models to choose from (and combine). And you could also tune hyperparameters for most of these models. Happy tinkering!

In [279]:
from sklearn.model_selection import StratifiedKFold as skf
skf = StratifiedKFold(n_splits=10, random_state=0, shuffle=True)

X_train = train.drop(["diagnosis", "row_id"], axis=1)
y_train = train['diagnosis'].astype(int)

dnn_params = {"hidden_units":(
                                                        (512,0,False),
                                                        (256,0,False),
                                                        (128,0,False),
                                                        (64,0,False),
                                                        (32,0,False),
                                                        (16,0,False),
                                                        (8,0,False),



),
                                         "dnn_activation":"relu"}

conf = deeptable.ModelConfig(nets=['dnn_nets', 'autoint_nets'],
                             optimizer="RMSprop",
                            earlystopping_patience=5,
                            apply_class_weight=True,
                            monitor_metric="val_loss",
                            dnn_params=dnn_params)


score = 0.0
for fold, (itrain, ivalid) in enumerate(skf.split(X_train, y_train)):
    print("-"*40)
    print(f"Running for fold {fold}")   
    xtr,ytr = X_train.iloc[itrain], y_train.iloc[itrain]
    xv,yv  =  X_train.iloc[ivalid], y_train.iloc[ivalid]
    dt = deeptable.DeepTable(config=conf)
    model, history = dt.fit(xtr,ytr,epochs=100,verbose=0, validation_data=(xv,yv))
    ll = log_loss(yv.values.astype(int), dt.predict_proba(xv))
    print("-"*40)
    print(ll)
    score += ll
    dt.save(f"nn_{fold}")
    
print(score/10)
----------------------------------------
Running for fold 0
3 class detected, inferred as a [multiclass classification] task
Preparing features taken 0.15832805633544922s
Imputation taken 0.1703338623046875s
Categorical encoding taken 3.361701965332031e-05s
Calc classes weight.
Examples:
Total:29499
class 0:0.35009078933314347
class 1:26.013227513227516
class 2:9.509671179883947
Injected a callback [EarlyStopping]. monitor:val_loss, patience:5, mode:min
>>>>>>>>>>>>>>>>>>>>>> Model Desc <<<<<<<<<<<<<<<<<<<<<<< 
---------------------------------------------------------
inputs:
---------------------------------------------------------
['input_continuous_all: (123)']
---------------------------------------------------------
embeddings:
---------------------------------------------------------
None
---------------------------------------------------------
dense: dropout: 0
batch_normalization: False
---------------------------------------------------------
concat_embed_dense: shape: (None, 123)
---------------------------------------------------------
nets: ['dnn_nets', 'autoint_nets']
---------------------------------------------------------
dnn: input_shape (None, 123), output_shape (None, 8)
autoint: input_shape None, output_shape None
---------------------------------------------------------
stacking_op: add
---------------------------------------------------------
output: activation: softmax, output_shape: (None, 3), use_bias: True
loss: categorical_crossentropy
optimizer: RMSprop
---------------------------------------------------------

Restoring model weights from the end of the best epoch.
Epoch 00006: early stopping
Model has been saved to:dt_output/dt_20210512 021721_dnn_nets_autoint_nets/dnn_nets+autoint_nets.h5
----------------------------------------
0.3943079789251681
----------------------------------------
Running for fold 1
3 class detected, inferred as a [multiclass classification] task
Preparing features taken 0.1658332347869873s
Imputation taken 0.1638166904449463s
Categorical encoding taken 1.7404556274414062e-05s
Calc classes weight.
Examples:
Total:29499
class 0:0.35009078933314347
class 1:26.013227513227516
class 2:9.509671179883947
Injected a callback [EarlyStopping]. monitor:val_loss, patience:5, mode:min
>>>>>>>>>>>>>>>>>>>>>> Model Desc <<<<<<<<<<<<<<<<<<<<<<< 
---------------------------------------------------------
inputs:
---------------------------------------------------------
['input_continuous_all: (123)']
---------------------------------------------------------
embeddings:
---------------------------------------------------------
None
---------------------------------------------------------
dense: dropout: 0
batch_normalization: False
---------------------------------------------------------
concat_embed_dense: shape: (None, 123)
---------------------------------------------------------
nets: ['dnn_nets', 'autoint_nets']
---------------------------------------------------------
dnn: input_shape (None, 123), output_shape (None, 8)
autoint: input_shape None, output_shape None
---------------------------------------------------------
stacking_op: add
---------------------------------------------------------
output: activation: softmax, output_shape: (None, 3), use_bias: True
loss: categorical_crossentropy
optimizer: RMSprop
---------------------------------------------------------

Restoring model weights from the end of the best epoch.
Epoch 00006: early stopping
Model has been saved to:dt_output/dt_20210512 021731_dnn_nets_autoint_nets/dnn_nets+autoint_nets.h5
----------------------------------------
0.7343751654799149
----------------------------------------
Running for fold 2
3 class detected, inferred as a [multiclass classification] task
Preparing features taken 0.17086172103881836s
Imputation taken 0.16071772575378418s
Categorical encoding taken 1.811981201171875e-05s
Calc classes weight.
Examples:
Total:29499
class 0:0.35009078933314347
class 1:26.013227513227516
class 2:9.509671179883947
Injected a callback [EarlyStopping]. monitor:val_loss, patience:5, mode:min
>>>>>>>>>>>>>>>>>>>>>> Model Desc <<<<<<<<<<<<<<<<<<<<<<< 
---------------------------------------------------------
inputs:
---------------------------------------------------------
['input_continuous_all: (123)']
---------------------------------------------------------
embeddings:
---------------------------------------------------------
None
---------------------------------------------------------
dense: dropout: 0
batch_normalization: False
---------------------------------------------------------
concat_embed_dense: shape: (None, 123)
---------------------------------------------------------
nets: ['dnn_nets', 'autoint_nets']
---------------------------------------------------------
dnn: input_shape (None, 123), output_shape (None, 8)
autoint: input_shape None, output_shape None
---------------------------------------------------------
stacking_op: add
---------------------------------------------------------
output: activation: softmax, output_shape: (None, 3), use_bias: True
loss: categorical_crossentropy
optimizer: RMSprop
---------------------------------------------------------

Restoring model weights from the end of the best epoch.
Epoch 00007: early stopping
Model has been saved to:dt_output/dt_20210512 021739_dnn_nets_autoint_nets/dnn_nets+autoint_nets.h5
----------------------------------------
0.39416817651995434
----------------------------------------
Running for fold 3
3 class detected, inferred as a [multiclass classification] task
Preparing features taken 0.15629076957702637s
Imputation taken 0.15803003311157227s
Categorical encoding taken 1.811981201171875e-05s
Calc classes weight.
Examples:
Total:29499
class 0:0.35009078933314347
class 1:26.013227513227516
class 2:9.509671179883947
Injected a callback [EarlyStopping]. monitor:val_loss, patience:5, mode:min
>>>>>>>>>>>>>>>>>>>>>> Model Desc <<<<<<<<<<<<<<<<<<<<<<< 
---------------------------------------------------------
inputs:
---------------------------------------------------------
['input_continuous_all: (123)']
---------------------------------------------------------
embeddings:
---------------------------------------------------------
None
---------------------------------------------------------
dense: dropout: 0
batch_normalization: False
---------------------------------------------------------
concat_embed_dense: shape: (None, 123)
---------------------------------------------------------
nets: ['dnn_nets', 'autoint_nets']
---------------------------------------------------------
dnn: input_shape (None, 123), output_shape (None, 8)
autoint: input_shape None, output_shape None
---------------------------------------------------------
stacking_op: add
---------------------------------------------------------
output: activation: softmax, output_shape: (None, 3), use_bias: True
loss: categorical_crossentropy
optimizer: RMSprop
---------------------------------------------------------

Restoring model weights from the end of the best epoch.
Epoch 00006: early stopping
Model has been saved to:dt_output/dt_20210512 021749_dnn_nets_autoint_nets/dnn_nets+autoint_nets.h5
----------------------------------------
0.5215039837384121
----------------------------------------
Running for fold 4
3 class detected, inferred as a [multiclass classification] task
Preparing features taken 0.15322494506835938s
Imputation taken 0.16054177284240723s
Categorical encoding taken 1.8835067749023438e-05s
Calc classes weight.
Examples:
Total:29499
class 0:0.35009078933314347
class 1:26.013227513227516
class 2:9.509671179883947
Injected a callback [EarlyStopping]. monitor:val_loss, patience:5, mode:min
>>>>>>>>>>>>>>>>>>>>>> Model Desc <<<<<<<<<<<<<<<<<<<<<<< 
---------------------------------------------------------
inputs:
---------------------------------------------------------
['input_continuous_all: (123)']
---------------------------------------------------------
embeddings:
---------------------------------------------------------
None
---------------------------------------------------------
dense: dropout: 0
batch_normalization: False
---------------------------------------------------------
concat_embed_dense: shape: (None, 123)
---------------------------------------------------------
nets: ['dnn_nets', 'autoint_nets']
---------------------------------------------------------
dnn: input_shape (None, 123), output_shape (None, 8)
autoint: input_shape None, output_shape None
---------------------------------------------------------
stacking_op: add
---------------------------------------------------------
output: activation: softmax, output_shape: (None, 3), use_bias: True
loss: categorical_crossentropy
optimizer: RMSprop
---------------------------------------------------------

Restoring model weights from the end of the best epoch.
Epoch 00008: early stopping
Model has been saved to:dt_output/dt_20210512 021757_dnn_nets_autoint_nets/dnn_nets+autoint_nets.h5
----------------------------------------
0.733602980090243
----------------------------------------
Running for fold 5
3 class detected, inferred as a [multiclass classification] task
Preparing features taken 0.15276718139648438s
Imputation taken 0.4602482318878174s
Categorical encoding taken 1.8835067749023438e-05s
Calc classes weight.
Examples:
Total:29499
class 0:0.35009078933314347
class 1:26.013227513227516
class 2:9.509671179883947
Injected a callback [EarlyStopping]. monitor:val_loss, patience:5, mode:min
>>>>>>>>>>>>>>>>>>>>>> Model Desc <<<<<<<<<<<<<<<<<<<<<<< 
---------------------------------------------------------
inputs:
---------------------------------------------------------
['input_continuous_all: (123)']
---------------------------------------------------------
embeddings:
---------------------------------------------------------
None
---------------------------------------------------------
dense: dropout: 0
batch_normalization: False
---------------------------------------------------------
concat_embed_dense: shape: (None, 123)
---------------------------------------------------------
nets: ['dnn_nets', 'autoint_nets']
---------------------------------------------------------
dnn: input_shape (None, 123), output_shape (None, 8)
autoint: input_shape None, output_shape None
---------------------------------------------------------
stacking_op: add
---------------------------------------------------------
output: activation: softmax, output_shape: (None, 3), use_bias: True
loss: categorical_crossentropy
optimizer: RMSprop
---------------------------------------------------------

Restoring model weights from the end of the best epoch.
Epoch 00010: early stopping
Model has been saved to:dt_output/dt_20210512 021808_dnn_nets_autoint_nets/dnn_nets+autoint_nets.h5
----------------------------------------
0.6290689401180699
----------------------------------------
Running for fold 6
3 class detected, inferred as a [multiclass classification] task
Preparing features taken 0.15475225448608398s
Imputation taken 0.15557456016540527s
Categorical encoding taken 1.811981201171875e-05s
Calc classes weight.
Examples:
Total:29499
class 0:0.35009078933314347
class 1:26.013227513227516
class 2:9.509671179883947
Injected a callback [EarlyStopping]. monitor:val_loss, patience:5, mode:min
>>>>>>>>>>>>>>>>>>>>>> Model Desc <<<<<<<<<<<<<<<<<<<<<<< 
---------------------------------------------------------
inputs:
---------------------------------------------------------
['input_continuous_all: (123)']
---------------------------------------------------------
embeddings:
---------------------------------------------------------
None
---------------------------------------------------------
dense: dropout: 0
batch_normalization: False
---------------------------------------------------------
concat_embed_dense: shape: (None, 123)
---------------------------------------------------------
nets: ['dnn_nets', 'autoint_nets']
---------------------------------------------------------
dnn: input_shape (None, 123), output_shape (None, 8)
autoint: input_shape None, output_shape None
---------------------------------------------------------
stacking_op: add
---------------------------------------------------------
output: activation: softmax, output_shape: (None, 3), use_bias: True
loss: categorical_crossentropy
optimizer: RMSprop
---------------------------------------------------------

Restoring model weights from the end of the best epoch.
Epoch 00007: early stopping
Model has been saved to:dt_output/dt_20210512 021821_dnn_nets_autoint_nets/dnn_nets+autoint_nets.h5
----------------------------------------
0.6096622519273308
----------------------------------------
Running for fold 7
3 class detected, inferred as a [multiclass classification] task
Preparing features taken 0.1522071361541748s
Imputation taken 0.15454792976379395s
Categorical encoding taken 1.71661376953125e-05s
Calc classes weight.
Examples:
Total:29500
class 0:0.35010265721982886
class 1:26.01410934744268
class 2:9.500805152979066
Injected a callback [EarlyStopping]. monitor:val_loss, patience:5, mode:min
>>>>>>>>>>>>>>>>>>>>>> Model Desc <<<<<<<<<<<<<<<<<<<<<<< 
---------------------------------------------------------
inputs:
---------------------------------------------------------
['input_continuous_all: (123)']
---------------------------------------------------------
embeddings:
---------------------------------------------------------
None
---------------------------------------------------------
dense: dropout: 0
batch_normalization: False
---------------------------------------------------------
concat_embed_dense: shape: (None, 123)
---------------------------------------------------------
nets: ['dnn_nets', 'autoint_nets']
---------------------------------------------------------
dnn: input_shape (None, 123), output_shape (None, 8)
autoint: input_shape None, output_shape None
---------------------------------------------------------
stacking_op: add
---------------------------------------------------------
output: activation: softmax, output_shape: (None, 3), use_bias: True
loss: categorical_crossentropy
optimizer: RMSprop
---------------------------------------------------------

Restoring model weights from the end of the best epoch.
Epoch 00007: early stopping
Model has been saved to:dt_output/dt_20210512 021831_dnn_nets_autoint_nets/dnn_nets+autoint_nets.h5
----------------------------------------
0.6255286317306245
----------------------------------------
Running for fold 8
3 class detected, inferred as a [multiclass classification] task
Preparing features taken 0.156599760055542s
Imputation taken 0.15280437469482422s
Categorical encoding taken 1.7404556274414062e-05s
Calc classes weight.
Examples:
Total:29500
class 0:0.35009019272761793
class 1:26.01410934744268
class 2:9.509993552546744
Injected a callback [EarlyStopping]. monitor:val_loss, patience:5, mode:min
>>>>>>>>>>>>>>>>>>>>>> Model Desc <<<<<<<<<<<<<<<<<<<<<<< 
---------------------------------------------------------
inputs:
---------------------------------------------------------
['input_continuous_all: (123)']
---------------------------------------------------------
embeddings:
---------------------------------------------------------
None
---------------------------------------------------------
dense: dropout: 0
batch_normalization: False
---------------------------------------------------------
concat_embed_dense: shape: (None, 123)
---------------------------------------------------------
nets: ['dnn_nets', 'autoint_nets']
---------------------------------------------------------
dnn: input_shape (None, 123), output_shape (None, 8)
autoint: input_shape None, output_shape None
---------------------------------------------------------
stacking_op: add
---------------------------------------------------------
output: activation: softmax, output_shape: (None, 3), use_bias: True
loss: categorical_crossentropy
optimizer: RMSprop
---------------------------------------------------------

Restoring model weights from the end of the best epoch.
Epoch 00010: early stopping
Model has been saved to:dt_output/dt_20210512 021841_dnn_nets_autoint_nets/dnn_nets+autoint_nets.h5
----------------------------------------
0.6583830074230764
----------------------------------------
Running for fold 9
3 class detected, inferred as a [multiclass classification] task
Preparing features taken 0.1545114517211914s
Imputation taken 0.15814733505249023s
Categorical encoding taken 1.7404556274414062e-05s
Calc classes weight.
Examples:
Total:29500
class 0:0.35009019272761793
class 1:26.01410934744268
class 2:9.509993552546744
Injected a callback [EarlyStopping]. monitor:val_loss, patience:5, mode:min
>>>>>>>>>>>>>>>>>>>>>> Model Desc <<<<<<<<<<<<<<<<<<<<<<< 
---------------------------------------------------------
inputs:
---------------------------------------------------------
['input_continuous_all: (123)']
---------------------------------------------------------
embeddings:
---------------------------------------------------------
None
---------------------------------------------------------
dense: dropout: 0
batch_normalization: False
---------------------------------------------------------
concat_embed_dense: shape: (None, 123)
---------------------------------------------------------
nets: ['dnn_nets', 'autoint_nets']
---------------------------------------------------------
dnn: input_shape (None, 123), output_shape (None, 8)
autoint: input_shape None, output_shape None
---------------------------------------------------------
stacking_op: add
---------------------------------------------------------
output: activation: softmax, output_shape: (None, 3), use_bias: True
loss: categorical_crossentropy
optimizer: RMSprop
---------------------------------------------------------

Restoring model weights from the end of the best epoch.
Epoch 00010: early stopping
Model has been saved to:dt_output/dt_20210512 021854_dnn_nets_autoint_nets/dnn_nets+autoint_nets.h5
----------------------------------------
0.5063319144940494
0.5806933030446844
In [280]:
dts = []
for i in range(10):
    model_filename = f'nn_{i}'
    dt = deeptable.DeepTable.load(model_filename)
    dts.append(dt)
f = True
preds = 0.0
for dt in dts:
    if f:
        preds = dt.predict_proba(valid.iloc[:,1:])
        f = False
    else:
        preds = np.add(preds,dt.predict_proba(valid.iloc[:,1:]))
preds = preds / 10.0
Load model from disk:nn_0/dnn_nets+autoint_nets.h5.
Load model from disk:nn_1/dnn_nets+autoint_nets.h5.
Load model from disk:nn_2/dnn_nets+autoint_nets.h5.
Load model from disk:nn_3/dnn_nets+autoint_nets.h5.
Load model from disk:nn_4/dnn_nets+autoint_nets.h5.
Load model from disk:nn_5/dnn_nets+autoint_nets.h5.
Load model from disk:nn_6/dnn_nets+autoint_nets.h5.
Load model from disk:nn_7/dnn_nets+autoint_nets.h5.
Load model from disk:nn_8/dnn_nets+autoint_nets.h5.
Load model from disk:nn_9/dnn_nets+autoint_nets.h5.
WARNING:tensorflow:5 out of the last 36 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f33d0f7daf0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f33d12331f0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f33284dfa60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f33482e8af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f33d1233ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f33d0f7d9d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f3348665af0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.

One more thing to note, the model can produce wildly different results each time it's trained (but you can save weights after training), is there a better way to deal with the randomness? I'll leave that to you.

In [281]:
from sklearn.metrics import log_loss
log_loss(valid_gt.diagnosis.values.astype(int),preds)
Out[281]:
0.6604266336736995

Save your trained model

In [282]:
meta = {
    "categorical": meta_train[0],
    "continuous": meta_train[1],
    "most_commons": meta_train[2],
}
!cp nn* {AICROWD_ASSETS_DIR} -r
meta_filename  = f'{AICROWD_ASSETS_DIR}/nn_meta.pkl'
joblib.dump(meta, meta_filename)
Out[282]:
['assets/nn_meta.pkl']

Prediction phase 🔎

Please make sure to save the weights from the training section in your assets directory and load them in this section

In [283]:
meta_filename  = f'{AICROWD_ASSETS_DIR}/nn_meta.pkl'
meta = joblib.load(meta_filename)

dts = []
for i in range(10):
    model_filename = f'{AICROWD_ASSETS_DIR}/nn_{i}'
    dt = deeptable.DeepTable.load(model_filename)
    dts.append(dt)

categorical = meta['categorical']
continuous = meta['continuous']
most_commons = meta['most_commons']
Load model from disk:assets/nn_0/dnn_nets+autoint_nets.h5.
Load model from disk:assets/nn_1/dnn_nets+autoint_nets.h5.
Load model from disk:assets/nn_2/dnn_nets+autoint_nets.h5.
Load model from disk:assets/nn_3/dnn_nets+autoint_nets.h5.
Load model from disk:assets/nn_4/dnn_nets+autoint_nets.h5.
Load model from disk:assets/nn_5/dnn_nets+autoint_nets.h5.
Load model from disk:assets/nn_6/dnn_nets+autoint_nets.h5.
Load model from disk:assets/nn_7/dnn_nets+autoint_nets.h5.
Load model from disk:assets/nn_8/dnn_nets+autoint_nets.h5.
Load model from disk:assets/nn_9/dnn_nets+autoint_nets.h5.

Load test data

In [284]:
test_data = pd.read_csv(AICROWD_DATASET_PATH)

Generate predictions

In [285]:
for i,col in enumerate(categorical):
    test_data[col] = test_data[col].fillna(most_commons[i])
imputer = KNNImputer()
test_data[continuous] = imputer.fit_transform(test_data[continuous])
one_hot = pd.get_dummies(test_data["intersection_pos_rel_centre"])
test_data = test_data.drop("intersection_pos_rel_centre", axis=1)
test_data = test_data.join(one_hot)


m = [col for col in test_data.columns if col.endswith("dist from cen")]
test_data["dist from cen_var"] = test_data[m].var(axis=1)
m = [col for col in test_data.columns if col.startswith("euc_dist_digit_")]
test_data["euc_dist_digit_var"] = test_data[m].var(axis=1)


X_test = test_data.drop("row_id",axis=1)
preds = 0.0
f = True
for dt in dts:
    if f:
        preds = dt.predict_proba(X_test)
        f = False
    else:
        preds = np.add(preds,dt.predict_proba(X_test))
preds = preds / 10.0
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f33283c7b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f3380041040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f33485c8670> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f332851e700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f33485c8790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f332851eaf0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f33480ea280> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f33480ea820> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f332851e9d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f33482bf4c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
In [286]:
predictions = {
    "row_id": test_data["row_id"].values,
    "normal_diagnosis_probability": preds[:,0],
    "pre_alzheimer_diagnosis_probability": preds[:,1],
    "post_alzheimer_diagnosis_probability": preds[:,2]
}

predictions_df = pd.DataFrame.from_dict(predictions)

Save predictions 📨

In [287]:
predictions_df.to_csv(AICROWD_PREDICTIONS_PATH, index=False)

Submit to AIcrowd 🚀

NOTE: PLEASE SAVE THE NOTEBOOK BEFORE SUBMITTING IT (Ctrl + S)

In [288]:
!DATASET_PATH=$AICROWD_DATASET_PATH \
aicrowd notebook submit \
    --assets-dir $AICROWD_ASSETS_DIR \
    --challenge addi-alzheimers-detection-challenge
API Key valid
Saved API Key successfully!
Using notebook: /home/desktop0/nn.ipynb for submission...
Removing existing files from submission directory...
Scrubbing API keys from the notebook...
Collecting notebook...
Validating the submission...
Executing install.ipynb...
[NbConvertApp] Converting notebook /home/desktop0/submission/install.ipynb to notebook
[NbConvertApp] Executing notebook with kernel: python
[NbConvertApp] Writing 552 bytes to /home/desktop0/submission/install.nbconvert.ipynb
Executing predict.ipynb...
[NbConvertApp] Converting notebook /home/desktop0/submission/predict.ipynb to notebook
[NbConvertApp] Executing notebook with kernel: python
2021-05-12 02:24:13.428599: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-05-12 02:24:13.428647: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-05-12 02:24:14.889944: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-05-12 02:24:14.890235: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-05-12 02:24:14.890267: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-05-12 02:24:14.890315: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (linux-01): /proc/driver/nvidia/version does not exist
2021-05-12 02:24:14.922632: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-05-12 02:24:16.897449: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-05-12 02:24:16.897959: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2294685000 Hz
[NbConvertApp] Writing 16126 bytes to /home/desktop0/submission/predict.nbconvert.ipynb
submission.zip ━━━━━━━━━━━━━━━━━━━━━━ 100.0%17.6/17.6 MB2.5 MB/s0:00:00[0m • 0:00:01[36m0:00:01
                                                 ╭─────────────────────────╮                                                 
                                                 │ Successfully submitted! │                                                 
                                                 ╰─────────────────────────╯                                                 
                                                       Important links                                                       
┌──────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│  This submission │ https://www.aicrowd.com/challenges/addi-alzheimers-detection-challenge/submissions/136511              │
│                  │                                                                                                        │
│  All submissions │ https://www.aicrowd.com/challenges/addi-alzheimers-detection-challenge/submissions?my_submissions=true │
│                  │                                                                                                        │
│      Leaderboard │ https://www.aicrowd.com/challenges/addi-alzheimers-detection-challenge/leaderboards                    │
│                  │                                                                                                        │
│ Discussion forum │ https://discourse.aicrowd.com/c/addi-alzheimers-detection-challenge                                    │
│                  │                                                                                                        │
│   Challenge page │ https://www.aicrowd.com/challenges/addi-alzheimers-detection-challenge                                 │
└──────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Reflections

This was an exploration into what a neural network could do in this competition. The model shown has a bit higher log loss on the public leaderboard. Maybe with some hyperparameter tuning it will help in an ensemble!


Comments

You must login before you can post a comment.

Execute