Loading

Emotion Detection

BERT for emotion detection

Fine tuning BERT pretrained on MLM for emotion detection task

falak

Google BERT is an important model ubiquitous across NLP tasks. In this notebook we'll use the HuggingFace `transformers` library to fine-tune pretrained BERT model for classification. The `transformers` library provides pretrained state-of-the-art BERT models. Here we finetune it for emotion detection task.

Fine-tuning of BERT for Sentiment Analysis

Introduction

Google BERT is an important model ubiquitous across NLP tasks. In this notebook we'll use the HuggingFace transformers library to fine-tune pretrained BERT model for classification. The transformers library provides pretrained state-of-the-art BERT models.

Reference:

Here are some useful blogs on transformers:

Setup

In [ ]:
# Install huggingface library
!pip install transformers

# AI crowd CLI for data download
!pip install aicrowd-cli
Requirement already satisfied: transformers in /opt/conda/lib/python3.7/site-packages (4.5.1)
Requirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.7/site-packages (from transformers) (4.59.0)
Requirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from transformers) (3.0.12)
Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.7/site-packages (from transformers) (1.19.5)
Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from transformers) (3.4.0)
Requirement already satisfied: sacremoses in /opt/conda/lib/python3.7/site-packages (from transformers) (0.0.45)
Requirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.7/site-packages (from transformers) (2021.3.17)
Requirement already satisfied: tokenizers<0.11,>=0.10.1 in /opt/conda/lib/python3.7/site-packages (from transformers) (0.10.2)
Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from transformers) (2.25.1)
Requirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from transformers) (20.9)
Requirement already satisfied: typing-extensions>=3.6.4 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->transformers) (3.7.4.3)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->transformers) (3.4.1)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->transformers) (2.4.7)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->transformers) (1.26.4)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests->transformers) (2020.12.5)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->transformers) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests->transformers) (4.0.0)
Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers) (1.15.0)
Requirement already satisfied: click in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers) (7.1.2)
Requirement already satisfied: joblib in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers) (1.0.1)
Collecting aicrowd-cli
  Downloading aicrowd_cli-0.1.7-py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 512 kB/s eta 0:00:01
Requirement already satisfied: toml<1,>=0.10.2 in /opt/conda/lib/python3.7/site-packages (from aicrowd-cli) (0.10.2)
Requirement already satisfied: gitpython<4,>=3.1.12 in /opt/conda/lib/python3.7/site-packages (from aicrowd-cli) (3.1.14)
Requirement already satisfied: tqdm<5,>=4.56.0 in /opt/conda/lib/python3.7/site-packages (from aicrowd-cli) (4.59.0)
Requirement already satisfied: requests<3,>=2.25.1 in /opt/conda/lib/python3.7/site-packages (from aicrowd-cli) (2.25.1)
Requirement already satisfied: click<8,>=7.1.2 in /opt/conda/lib/python3.7/site-packages (from aicrowd-cli) (7.1.2)
Collecting requests-toolbelt<1,>=0.9.1
  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 1.1 MB/s eta 0:00:01
Collecting rich<11,>=10.0.0
  Downloading rich-10.3.0-py3-none-any.whl (205 kB)
     |████████████████████████████████| 205 kB 2.6 MB/s eta 0:00:01
Requirement already satisfied: gitdb<5,>=4.0.1 in /opt/conda/lib/python3.7/site-packages (from gitpython<4,>=3.1.12->aicrowd-cli) (4.0.7)
Requirement already satisfied: smmap<5,>=3.0.1 in /opt/conda/lib/python3.7/site-packages (from gitdb<5,>=4.0.1->gitpython<4,>=3.1.12->aicrowd-cli) (3.0.5)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.25.1->aicrowd-cli) (1.26.4)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.25.1->aicrowd-cli) (4.0.0)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.25.1->aicrowd-cli) (2020.12.5)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.25.1->aicrowd-cli) (2.10)
Collecting commonmark<0.10.0,>=0.9.0
  Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
     |████████████████████████████████| 51 kB 2.2 MB/s eta 0:00:01
Requirement already satisfied: colorama<0.5.0,>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from rich<11,>=10.0.0->aicrowd-cli) (0.4.4)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in /opt/conda/lib/python3.7/site-packages (from rich<11,>=10.0.0->aicrowd-cli) (2.8.1)
Requirement already satisfied: typing-extensions<4.0.0,>=3.7.4 in /opt/conda/lib/python3.7/site-packages (from rich<11,>=10.0.0->aicrowd-cli) (3.7.4.3)
Installing collected packages: commonmark, rich, requests-toolbelt, aicrowd-cli
Successfully installed aicrowd-cli-0.1.7 commonmark-0.9.1 requests-toolbelt-0.9.1 rich-10.3.0
In [ ]:
import os
import re
import random
import time

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import torch
import torch.nn.functional as F
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler

from transformers import AdamW, get_linear_schedule_with_warmup
from transformers import BertTokenizer
from transformers import BertModel
from tqdm import tqdm

%matplotlib inline
In [ ]:
API_KEY = '32a51964ffee76a5eea432879125' # Please get your your API Key from [https://www.aicrowd.com/participants/me]
!aicrowd login --api-key $API_KEY

# Downloading the Dataset
!mkdir data
!aicrowd dataset download --challenge emotion-detection -j 3 -o data
API Key valid
Saved API Key successfully!
test.csv: 100%|███████████████████████████████| 642k/642k [00:00<00:00, 976kB/s]
train.csv:   0%|                                    | 0.00/2.30M [00:00<?, ?B/s]
val.csv:   0%|                                       | 0.00/262k [00:00<?, ?B/s]
val.csv: 100%|████████████████████████████████| 262k/262k [00:00<00:00, 497kB/s]
train.csv: 100%|███████████████████████████| 2.30M/2.30M [00:00<00:00, 2.37MB/s]
In [ ]:
train_dataset = pd.read_csv("data/train.csv")
validation_dataset = pd.read_csv("data/val.csv")[1:]
In [ ]:
X_train = train_dataset.text.values
y_train = train_dataset.label.values
X_val = validation_dataset.text.values
y_val = validation_dataset.label.values
In [ ]:
# Let's look at some examples
print(X_train[:20])
['takes no time to copy/paste a press release' "You're delusional"
 "Jazz fan here. I completely feel. Lindsay Mann cousins has more votes than Lindsay Mann, and Lindsay Mann hasn't even stepped on the court this year"
 'ah i was also confused but i think they mean friends around the same age'
 'Thank you so much. ♥️ that means a lot.' 'And I’ll be there!!!'
 'There are some amazingly cringey compilations of terrible dialogue from this show on yt'
 'Check the frame (FPS) limit option in the advanced graphic options menu.'
 'you made me think I was in the dbd subreddit with that statement idk why'
 'It was in your op.'
 'There is a reason 15 people liked my comment dear, those fifteen people all think you are pretty. I hope you see this... 👓'
 "He's not THAT bad on YouTube, but on Twitter... Let's just say that he's problematic"
 'As someone who is dairy intolerant (milk is in _everything _!!) this veganuary is awesome :D'
 'IVE NEVER BEEN MORE PROUD TO CAMPAIGN FOR A CANDIDATE!! I stg im so happy shes my congresswoman! finally dumped comstock!!!!'
 'See him live if you haven’t already. One of my favorites to see in concert.'
 'CNN has video of the arrest. Someone needs to put the cops theme over it.'
 "It wasn't fair on the rest of the list, nothing is good as Tim Hortons."
 "Finally got this in the mail today. I'm pretty excited. My favorite ballplayer."
 'Yes, once I even deleted my account because I was so nervous about down votes. Lately,every once in a while I comment encouraging good attitude.'
 'This is truly the most captivating trilogy I’ve ever read. Eager to read the third story.']

Set up GPU

Select GPU accelerator on colab

Runtime -> Change runtime type -> Hardware accelerator: GPU

In [ ]:
if torch.cuda.is_available():       
    device = torch.device("cuda")
    print(f'There are {torch.cuda.device_count()} GPU(s) available.')
    print('Device name:', torch.cuda.get_device_name(0))
There are 1 GPU(s) available.
Device name: Tesla P100-PCIE-16GB

Fine-tuning BERT

Tokenization and Input Formatting

In [ ]:
def text_preprocessing(text):
    """
    Remove emojis from the text in preprocessing.
    """
    text = text.lower()  
    # Remove emojis
    regrex_pattern = re.compile(pattern = "["
        u"\U0001F600-\U0001F64F"  # emoticons
        u"\U0001F300-\U0001F5FF"  # symbols & pictographs
        u"\U0001F680-\U0001F6FF"  # transport & map symbols
        u"\U0001F1E0-\U0001F1FF"  # flags (iOS)
                           "]+", flags = re.UNICODE)
    text = regrex_pattern.sub(r'',text)
    return text
In [ ]:
text = 'There is a reason 15 people liked my comment dear. I hope you see this... 👓'
print('Original: ', text)
print('Processed: ', text_preprocessing(text))
Original:  There is a reason 15 people liked my comment dear. I hope you see this... 👓
Processed:  there is a reason 15 people liked my comment dear. i hope you see this... 

BERT Tokenizer

In order to apply the pre-trained BERT, we must use the tokenizer provided by the library. This is because (1) the model has a specific, fixed vocabulary and (2) the BERT tokenizer has a particular way of handling out-of-vocabulary words.

In addition, we are required to add special tokens to the start and end of each sentence, pad & truncate all sentences to a single constant length, and explicitly specify what are padding tokens with the "attention mask".

In [ ]:
# Load the BERT tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True, )

# Create a function to tokenize a set of texts
def preprocessing_for_bert(data):
    """Perform required preprocessing steps for pretrained BERT.
    @param    data (np.array): Array of texts to be processed.
    @return   input_ids (torch.Tensor): Tensor of token ids to be fed to a model.
    @return   attention_masks (torch.Tensor): Tensor of indices specifying which
                  tokens should be attended to by the model.
    """
    # Create empty lists to store outputs
    input_ids = []
    attention_masks = []

    # For every sentence...
    for sent in data:
        # `encode_plus` will:
        #    (1) Tokenize the sentence
        #    (2) Add the `[CLS]` and `[SEP]` token to the start and end
        #    (3) Truncate/Pad sentence to max length
        #    (4) Map tokens to their IDs
        #    (5) Create attention mask
        #    (6) Return a dictionary of outputs
        encoded_sent = tokenizer.encode_plus(
            text=text_preprocessing(sent),  # Preprocess sentence
            add_special_tokens=True,        # Add `[CLS]` and `[SEP]`
            max_length=MAX_LEN,                  # Max length to truncate/pad
            pad_to_max_length=True,         # Pad sentence to max length
            return_attention_mask=True,      # Return attention mask
            truncation=True
            )
        
        # Add the outputs to the lists
        input_ids.append(encoded_sent.get('input_ids'))
        attention_masks.append(encoded_sent.get('attention_mask'))

    # Convert lists to tensors
    input_ids = torch.tensor(input_ids)
    attention_masks = torch.tensor(attention_masks)

    return input_ids, attention_masks

Before tokenizing, we need to specify the maximum length of our sentences.

In [ ]:
# Concatenate train data and test data
all_data = np.concatenate([X_train, X_val])

# Encode our concatenated data
encoded_data = [tokenizer.encode(sent, add_special_tokens=True) for sent in all_data]

# Find the maximum length
max_len = max([len(sent) for sent in encoded_data])
print('Max length: ', max_len)
Max length:  316
In [ ]:
# Analyse the text and find the optimal length for truncating
# len_dist = [len(sent) for sent in encoded_tweets]
# plt.hist(len_dist, 50)
# print(sorted(len_dist)[::-2])

Now let's tokenize our data.

In [ ]:
# Specify `MAX_LEN`
MAX_LEN = 35

# Test the preprocessing function
token_ids = list(preprocessing_for_bert([X_train[0]])[0].squeeze().numpy())
print('Original: ', X_train[0])
print('Token IDs: ', token_ids)
Original:  takes no time to copy/paste a press release
Token IDs:  [101, 3138, 2053, 2051, 2000, 6100, 1013, 19351, 1037, 2811, 2713, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
/opt/conda/lib/python3.7/site-packages/transformers/tokenization_utils_base.py:2079: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
  FutureWarning,
In [ ]:
# Run function `preprocessing_for_bert` on the train set and the validation set
print('Tokenizing data...')
train_inputs, train_masks = preprocessing_for_bert(X_train)
val_inputs, val_masks = preprocessing_for_bert(X_val)

# Convert other data types to torch.Tensor
train_labels = torch.tensor(y_train)
val_labels = torch.tensor(y_val)
Tokenizing data...

Train Our Model

BERT-base consists of 12 transformer layers, each transformer layer takes in a list of token embeddings, and produces the same number of embeddings with the same hidden size (or dimensions) on the output. The output of the final transformer layer of the [CLS] token is used as the features of the sequence to feed a classifier.

In [ ]:
%%time

# Create the BertClassfier class
class BertClassifier(nn.Module):
    """Bert Model for Classification Tasks.
    """
    def __init__(self, freeze_bert=False):
        """
        @param    bert: a BertModel object
        @param    classifier: a torch.nn.Module classifier
        @param    freeze_bert (bool): Set `False` to fine-tune the BERT model
        """
        super(BertClassifier, self).__init__()
        # Specify hidden size of BERT, hidden size of our classifier, and number of labels
        D_in, H, D_out = 768, 64, 2

        # Instantiate BERT model
        self.bert = BertModel.from_pretrained('bert-base-uncased')

        # Instantiate an one-layer feed-forward classifier
        self.classifier = nn.Sequential(
            nn.Linear(D_in, H),
            nn.ReLU(),
            #nn.Dropout(0.5),
            nn.Linear(H, D_out)
        )

        # Freeze the BERT model
        if freeze_bert:
            for param in self.bert.parameters():
                param.requires_grad = False
        
    def forward(self, input_ids, attention_mask):
        """
        Feed input to BERT and the classifier to compute logits.
        @param    input_ids (torch.Tensor): an input tensor with shape (batch_size,
                      max_length)
        @param    attention_mask (torch.Tensor): a tensor that hold attention mask
                      information with shape (batch_size, max_length)
        @return   logits (torch.Tensor): an output tensor with shape (batch_size,
                      num_labels)
        """
        # Feed input to BERT
        outputs = self.bert(input_ids=input_ids,
                            attention_mask=attention_mask)
        
        # Extract the last hidden state of the token `[CLS]` for classification task
        last_hidden_state_cls = outputs[0][:, 0, :]

        # Feed input to classifier to compute logits
        logits = self.classifier(last_hidden_state_cls)

        return logits
    

def initialize_model(epochs=4):
    """Initialize the Bert Classifier, the optimizer and the learning rate scheduler.
    """
    # Instantiate Bert Classifier
    bert_classifier = BertClassifier(freeze_bert=False)

    # Tell PyTorch to run the model on GPU
    bert_classifier.to(device)

    # Create the optimizer
    optimizer = AdamW(bert_classifier.parameters(),
                      lr=5e-5,    # Default learning rate
                      eps=1e-8    # Default epsilon value
                      )

    # Total number of training steps
    total_steps = len(train_dataloader) * epochs

    # Set up the learning rate scheduler
    scheduler = get_linear_schedule_with_warmup(optimizer,
                                                num_warmup_steps=0, # Default value
                                                num_training_steps=total_steps)
    return bert_classifier, optimizer, scheduler
CPU times: user 35 µs, sys: 0 ns, total: 35 µs
Wall time: 39.1 µs

Training/ Eval Loop

The script below is commented with the details of our training, evaluation and predict steps.

In [ ]:
# Specify loss function
loss_fn = nn.CrossEntropyLoss()

def set_seed(seed_value=42):
    """Set seed for reproducibility.
    """
    random.seed(seed_value)
    np.random.seed(seed_value)
    torch.manual_seed(seed_value)
    torch.cuda.manual_seed_all(seed_value)

def train(model, train_dataloader, val_dataloader=None, epochs=4, evaluation=False):
    """Train the BertClassifier model.
    """
    # Start training loop
    print("Start training...\n")
    for epoch_i in range(epochs):
        print(f"{'Epoch':^7} | {'Batch':^7} | {'Train Loss':^12} | {'Val Loss':^10} | {'Val Acc':^9} | {'Elapsed':^9}")
        print("-"*70)

        # Measure the elapsed time of each epoch
        t0_epoch, t0_batch = time.time(), time.time()

        # Reset tracking variables at the beginning of each epoch
        total_loss, batch_loss, batch_counts = 0, 0, 0

        # Put the model into the training mode
        model.train()

        # For each batch of training data...
        for step, batch in enumerate(train_dataloader):
            batch_counts +=1
            # Load batch to GPU
            b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)

            # Zero out any previously calculated gradients
            model.zero_grad()

            # Perform a forward pass. This will return logits.
            logits = model(b_input_ids, b_attn_mask)

            # Compute loss and accumulate the loss values
            loss = loss_fn(logits, b_labels)
            batch_loss += loss.item()
            total_loss += loss.item()

            # Perform a backward pass to calculate gradients
            loss.backward()

            # Clip the norm of the gradients to 1.0 to prevent "exploding gradients"
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

            # Update parameters and the learning rate
            optimizer.step()
            scheduler.step()

            # Print the loss values and time elapsed for every 20 batches
            if (step % 20 == 0 and step != 0) or (step == len(train_dataloader) - 1):
                # Calculate time elapsed for 20 batches
                time_elapsed = time.time() - t0_batch

                # Print training results
                print(f"{epoch_i + 1:^7} | {step:^7} | {batch_loss / batch_counts:^12.6f} | {'-':^10} | {'-':^9} | {time_elapsed:^9.2f}")

                # Reset batch tracking variables
                batch_loss, batch_counts = 0, 0
                t0_batch = time.time()

        # Calculate the average loss over the entire training data
        avg_train_loss = total_loss / len(train_dataloader)

        print("-"*70)
        if evaluation == True:
            # After the completion of each training epoch, measure the model's performance
            # on our validation set.
            val_loss, val_accuracy = evaluate(model, val_dataloader)

            # Print performance over the entire training data
            time_elapsed = time.time() - t0_epoch
            
            print(f"{epoch_i + 1:^7} | {'-':^7} | {avg_train_loss:^12.6f} | {val_loss:^10.6f} | {val_accuracy:^9.2f} | {time_elapsed:^9.2f}")
            print("-"*70)
        print("\n")
    
    print("Training complete!")


def evaluate(model, val_dataloader):
    """After the completion of each training epoch, measure the model's performance
    on our validation set.
    """
    model.eval()

    # Tracking variables
    val_accuracy = []
    val_loss = []

    # For each batch in our validation set...
    for batch in val_dataloader:
        # Load batch to GPU
        b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)

        # Compute logits
        with torch.no_grad():
            logits = model(b_input_ids, b_attn_mask)

        # Compute loss
        loss = loss_fn(logits, b_labels)
        val_loss.append(loss.item())

        # Get the predictions
        preds = torch.argmax(logits, dim=1).flatten()

        # Calculate the accuracy rate
        accuracy = (preds == b_labels).cpu().numpy().mean() * 100
        val_accuracy.append(accuracy)

    # Compute the average accuracy and loss over the validation set.
    val_loss = np.mean(val_loss)
    val_accuracy = np.mean(val_accuracy)

    return val_loss, val_accuracy

def bert_predict(model, test_dataloader):
    """Perform a forward pass on the trained BERT model to predict probabilities
    on the test set.
    """
    # Put the model into the evaluation mode. The dropout layers are disabled during
    # the test time.
    model.eval()

    all_logits = []

    # For each batch in our test set...
    for batch in test_dataloader:
        # Load batch to GPU
        b_input_ids, b_attn_mask = tuple(t.to(device) for t in batch)[:2]

        # Compute logits
        with torch.no_grad():
            logits = model(b_input_ids, b_attn_mask)
        all_logits.append(logits)
    
    # Concatenate logits from each batch
    all_logits = torch.cat(all_logits, dim=0)

    # Apply softmax to calculate probabilities
    probs = F.softmax(all_logits, dim=1).cpu().numpy()

    return probs

Now, let's create data loaders and start training our BertClassifier!

In [ ]:
set_seed(42)
batch_size = 32

# Create the DataLoader for our training set
train_data = TensorDataset(train_inputs, train_masks, train_labels)
train_sampler = RandomSampler(train_data)
train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)

# Create the DataLoader for our validation set
val_data = TensorDataset(val_inputs, val_masks, val_labels)
val_sampler = SequentialSampler(val_data)
val_dataloader = DataLoader(val_data, sampler=val_sampler, batch_size=batch_size)

# Concatenate the train set and the validation set
full_train_data = torch.utils.data.ConcatDataset([train_data, val_data])
full_train_sampler = RandomSampler(full_train_data)
full_train_dataloader = DataLoader(full_train_data, sampler=full_train_sampler, batch_size=32)

# Train the Bert Classifier on the entire training data

bert_classifier, optimizer, scheduler = initialize_model(epochs=2)
train(bert_classifier, train_dataloader, epochs=1)
# train(bert_classifier, full_train_dataloader, epochs=1)
evaluate(bert_classifier, val_dataloader)
Start training...

 Epoch  |  Batch  |  Train Loss  |  Val Loss  |  Val Acc  |  Elapsed 
----------------------------------------------------------------------
   1    |   20    |   0.551077   |     -      |     -     |   4.62   
   1    |   40    |   0.450243   |     -      |     -     |   3.51   
   1    |   60    |   0.374716   |     -      |     -     |   3.67   
   1    |   80    |   0.369399   |     -      |     -     |   3.38   
   1    |   100   |   0.383739   |     -      |     -     |   3.57   
   1    |   120   |   0.358121   |     -      |     -     |   3.38   
   1    |   140   |   0.345789   |     -      |     -     |   3.36   
   1    |   160   |   0.329952   |     -      |     -     |   3.49   
   1    |   180   |   0.363243   |     -      |     -     |   3.37   
   1    |   200   |   0.320596   |     -      |     -     |   3.41   
   1    |   220   |   0.330034   |     -      |     -     |   3.39   
   1    |   240   |   0.342855   |     -      |     -     |   3.64   
   1    |   260   |   0.326514   |     -      |     -     |   3.56   
   1    |   280   |   0.372858   |     -      |     -     |   3.38   
   1    |   300   |   0.307736   |     -      |     -     |   3.49   
   1    |   320   |   0.319178   |     -      |     -     |   3.37   
   1    |   340   |   0.340524   |     -      |     -     |   3.39   
   1    |   360   |   0.371839   |     -      |     -     |   3.59   
   1    |   380   |   0.342046   |     -      |     -     |   3.51   
   1    |   400   |   0.312744   |     -      |     -     |   3.38   
   1    |   420   |   0.329041   |     -      |     -     |   3.52   
   1    |   440   |   0.312006   |     -      |     -     |   3.58   
   1    |   460   |   0.337357   |     -      |     -     |   3.39   
   1    |   480   |   0.354741   |     -      |     -     |   3.52   
   1    |   500   |   0.330441   |     -      |     -     |   3.39   
   1    |   520   |   0.335171   |     -      |     -     |   3.36   
   1    |   540   |   0.348457   |     -      |     -     |   3.53   
   1    |   560   |   0.311118   |     -      |     -     |   3.40   
   1    |   580   |   0.313956   |     -      |     -     |   3.39   
   1    |   600   |   0.310986   |     -      |     -     |   3.39   
   1    |   620   |   0.344323   |     -      |     -     |   3.64   
   1    |   640   |   0.301641   |     -      |     -     |   3.52   
   1    |   660   |   0.290095   |     -      |     -     |   3.36   
   1    |   680   |   0.334232   |     -      |     -     |   3.51   
   1    |   700   |   0.310011   |     -      |     -     |   3.38   
   1    |   720   |   0.340676   |     -      |     -     |   3.40   
   1    |   740   |   0.315238   |     -      |     -     |   3.51   
   1    |   760   |   0.311741   |     -      |     -     |   3.36   
   1    |   780   |   0.310916   |     -      |     -     |   3.37   
   1    |   800   |   0.327071   |     -      |     -     |   3.49   
   1    |   820   |   0.326617   |     -      |     -     |   3.63   
   1    |   840   |   0.294517   |     -      |     -     |   3.37   
   1    |   860   |   0.312432   |     -      |     -     |   3.52   
   1    |   880   |   0.308134   |     -      |     -     |   3.39   
   1    |   900   |   0.257281   |     -      |     -     |   3.40   
   1    |   920   |   0.336866   |     -      |     -     |   3.37   
   1    |   940   |   0.288061   |     -      |     -     |   3.57   
   1    |   960   |   0.314042   |     -      |     -     |   3.39   
   1    |   976   |   0.264756   |     -      |     -     |   2.68   
----------------------------------------------------------------------


Training complete!
Out[ ]:
(0.3071859935554889, 86.84059633027523)

Predictions on Test Set

In [ ]:
# Run `preprocessing_for_bert` on the test set
test_data = pd.read_csv('data/test.csv')
print('Tokenizing data...')
test_inputs, test_masks = preprocessing_for_bert(test_data.text)

# Create the DataLoader for our test set
test_dataset = TensorDataset(test_inputs, test_masks)
test_sampler = SequentialSampler(test_dataset)
test_dataloader = DataLoader(test_dataset, sampler=test_sampler, batch_size=32)
Tokenizing data...
In [ ]:
# Compute predicted probabilities on the test set
probs = bert_predict(bert_classifier, test_dataloader)

# Get predictions from the probabilities
threshold = 0.5
preds = np.where(probs[:, 1] > threshold, 1, 0)
test_data['label'] = preds
In [ ]:
!mkdir assets

# Saving the sample submission in assets directory
test_data.to_csv(os.path.join("assets", "submission.csv"), index=False)
In [ ]:
!aicrowd notebook submit -c emotion-detection -a assets --no-verify
Using notebook: /kaggle/working/__notebook_source__.ipynb for submission...
Scrubbing API keys from the notebook...
Collecting notebook...
submission.zip ━━━━━━━━━━━━━━━━━━━━ 100.0%279.4/277.7 KB1.3 MB/s0:00:0000:0100:01
                                                  ╭─────────────────────────╮                                                  
                                                  │ Successfully submitted! │                                                  
                                                  ╰─────────────────────────╯                                                  
                                                        Important links                                                        
┌──────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│  This submission │ https://www.aicrowd.com/challenges/ai-blitz-9/problems/emotion-detection/submissions/146170              │
│                  │                                                                                                          │
│  All submissions │ https://www.aicrowd.com/challenges/ai-blitz-9/problems/emotion-detection/submissions?my_submissions=true │
│                  │                                                                                                          │
│      Leaderboard │ https://www.aicrowd.com/challenges/ai-blitz-9/problems/emotion-detection/leaderboards                    │
│                  │                                                                                                          │
│ Discussion forum │ https://discourse.aicrowd.com/c/ai-blitz-9                                                               │
│                  │                                                                                                          │
│   Challenge page │ https://www.aicrowd.com/challenges/ai-blitz-9/problems/emotion-detection                                 │
└──────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────┘
In [ ]:


Comments

neelima_shah
4 months ago

Thank you! This was great to get started on the competition (and get a jump on the baseline :))

You must login before you can post a comment.

Execute