Lidar Car Detection
Solution for submission 156597
A detailed solution for submission 156597 submitted for challenge Lidar Car Detection
konstantin_diachkov
Starter Code for Lidar Car Detection
What we are going to Learn¶
- Learning about how lidar works
- Using scikit-learn for binary classification.
Note : Create a copy of the notebook and use the copy for submission. Go to File > Save a Copy in Drive to create a new copy
Downloading Dataset¶
Installing aicrowd-cli
In [4]:
!pip install aicrowd-cli
%load_ext aicrowd.magic
In [5]:
%aicrowd login
In [3]:
!rm -rf data
!mkdir data
%aicrowd ds dl -c lidar-car-detection -o data
Importing Libraries¶
In [6]:
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
import os
import matplotlib.pyplot as plt
import plotly.graph_objects as go
import random
%matplotlib notebook
import tensorflow as tf
from tensorflow.keras.layers import *
from tensorflow.keras.models import Model, load_model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import to_categorical
import tensorflow.keras.backend as K
In [ ]:
# !pip install catboost
Reading the dataset¶
In [ ]:
# Reading the training dataset
# Reading the training dataset
train_data = np.load("data/train.npz", allow_pickle=True)
test_data = np.load("data/test.npz", allow_pickle=True)
train_data = train_data['train']
test_data = test_data['test']
train_data.shape, test_data.shape
Out[ ]:
Visualizing the dataset¶
In this section, we will be visualizing a sample 3D lidar data
In [ ]:
# Getting a random 3D lidar sample data
INDEX = random.randint(0, train_data.shape[0] - 1)
# Getting the individual x, y and z points.
x = train_data[INDEX][0][:, 0].tolist()
y = train_data[INDEX][0][:, 1].tolist()
z = train_data[INDEX][0][:, 2].tolist()
# Label for the corrosponding sample ( no. of cars )
label = train_data[INDEX][1]
# Generating the 3D graph
fig = go.Figure(data=[go.Scatter3d(x=x, y=y, z=z,
mode='markers',
marker=dict(
size=1,
colorscale='Viridis',
opacity=0.8))])
print("No. of cars : ", label)
fig.show()
Can you try finding cars in this 3d data ?
In [ ]:
print(train_data[:, 1].max())
plt.figure()
plt.hist(train_data[:, 1])
plt.show()
In [ ]:
X = np.array([sample for sample in train_data[:, 0].tolist()])
Y = to_categorical(np.array(train_data[:, 1], dtype=np.int), num_classes=8)
X_test = np.array([sample for sample in test_data.tolist()])
m,n,r = X.shape
out_arr = np.column_stack((np.repeat(np.arange(m),n), X.reshape(m*n,-1)))
df = pd.DataFrame(out_arr)
df.columns = ["idx", "x", "y", "z"]
m,n,r = X_test.shape
out_arr = np.column_stack((np.repeat(np.arange(m),n), X_test.reshape(m*n,-1)))
df_test = pd.DataFrame(out_arr)
df_test.columns = ["idx", "x", "y", "z"]
In [ ]:
X.shape, X_test.shape, Y.shape
Out[ ]:
In [ ]:
# Normalize
df["x"] = df["x"] / 60 + 0.5
df["y"] = df["y"] / 60 + 0.5
df["z"] = df["z"] / 16 + 0.2
df_test["x"] = df_test["x"] / 60 + 0.5
df_test["y"] = df_test["y"] / 60 + 0.5
df_test["z"] = df_test["z"] / 16 + 0.2
In [ ]:
# Drop low / high z values rows
df = df.drop(df[(df["z"] > 0.22) | (df["z"] < 0.10)].index).reset_index()
df = df.drop(["z",], axis=1).reset_index()
df_test = df_test.drop(df_test[(df_test["z"] > 0.22) | (df_test["z"] < 0.10)].index).reset_index()
df_test = df_test.drop(["z",], axis=1).reset_index()
In [ ]:
# Turn 3D points cloud to flat images
img_size = 128
images = np.zeros((int(df["idx"].max()+1), img_size, img_size, 1))
images_test = np.zeros((int(df_test["idx"].max()+1), img_size, img_size, 1))
def make_img(points, images):
for i, r in points.iterrows():
images[int(r["idx"])][int(r["x"]*img_size)][int(r["y"]*img_size)] = [1]
df.groupby("idx").apply(lambda x: make_img(x, images))
df_test.groupby("idx").apply(lambda x: make_img(x, images_test))
Out[ ]:
In [ ]:
plt.figure()
plt.imshow(images[INDEX].squeeze(), cmap="gray")
plt.show()
In [ ]:
# Data augmentation (horizontal flip)
augmented_images = np.vstack((np.rot90(images, axes=(1, 2)), images))
augmented_images = np.vstack((np.rot90(images, k=2, axes=(1, 2)), augmented_images))
augmented_images = np.vstack((np.rot90(images, k=3, axes=(1, 2)), augmented_images))
augmented_Y = np.hstack((Y, Y, Y, Y))
In [ ]:
augmented_images.shape, augmented_Y.shape
Out[ ]:
In [ ]:
Splitting the dataset¶
In [ ]:
# Getting the 3d points and flattening the points into 1d array ( using only 100 training samples for faster training )
X = train_data[:, 0]
X = [i.flatten() for i in X]
# labels
y = train_data[:, 1]
In [ ]:
# Splitting the dataset into training and testing
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2)
Training the model¶
In [ ]:
model = RandomForestRegressor(verbose=True, n_jobs=-1, n_estimators=70)
In [ ]:
# from catboost import CatBoostRegressor
# model = CatBoostRegressor(loss_function='MAE',verbose=1, depth=3, iterations=300) # task_type="GPU",
model.fit(X_train, y_train)#, eval_set=(X_val, y_val))
Out[ ]:
Validation¶
In [ ]:
model.score(X_val, y_val)
Out[ ]:
Generating the predictions¶
In [ ]:
# Loading the test data
test_data = np.load("/content/data/test.npz", allow_pickle=True)
test_data = test_data['test']
test_data.shape
Out[ ]:
In [ ]:
# flattening the points into 1d array
X_test = X = [i.flatten() for i in test_data]
In [ ]:
# Generating the predictions
predictions = model.predict(X_test)
predictions.shape
Out[ ]:
In [ ]:
submission = pd.DataFrame({"label":predictions})
submission
Out[ ]:
In [7]:
# Saving the predictions
!rm -rf assets
!mkdir assets
# submission.to_csv(os.path.join("assets", "submission.csv"))
Submitting our Predictions¶
Note : Please save the notebook before submitting it (Ctrl + S)
In [ ]:
In [ ]:
Content
Comments
You must login before you can post a comment.