無人駕駛之Behaviroal-Cloning實現

引言

本篇部落格採用了Keras2.1.5,tensorflow1.7,整個專案原始碼:github
本次部落格主要分享Udacity自動駕駛納米學位的第三個專案,轉角預測模型的實現。
該專案的任務是在給定道路影象的前提下,實現一個深度神經網路模型來預測汽車的轉角。Udacity開源了一個模擬器。該模擬器有兩種模式:訓練模式和自動駕駛模式。
在訓練模式中,我們將車開到軌道上,並儲存幀(.jpg檔案)以及相應的轉向角(儲存在.csv檔案中)。事實上,其他引數值被記錄:油門,速度,剎車。在自主模式下,採用訓練模型預測轉向角和驅動車輛。
這裡給大家提供模擬器下載連結及訓練資料下載連結(大家可以自己駕駛汽車並採集資料):

基本實現思路

我們先來看看總體的軟體架構設計,我們根據實際情況,採用監督學習方法。架構圖如下所示
這裡寫圖片描述
模擬器一共有三個攝像頭,通過提供的資料,我們將圖片輸入到CNN,標籤是每次記錄的轉角資訊。通過訓練CNN模型,我們就能根據圖片來預測對應行駛的轉角了。
如下所示:
這裡寫圖片描述

詳細解釋及實現

import lib

這裡匯入一些庫

import numpy as np
import matplotlib.pyplot as plt
from keras.optimizers import Adam
from keras.layers.core import Dense,Activation
from keras.layers import Input,Conv2D,MaxPooling2D,Flatten,PReLU
from keras.models import Sequential,Model
from keras import backend as K
from keras.regularizers import l2
import os.path
import csv
import cv2
import glob
from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split
import json
from keras import callbacks
seed = 42
%matplotlib inline

Data exploration

主要看一下圖片資料以及一些駕駛資料

data_path = '../../data_collected/'
# 開啟CSV檔案,這裡可以用csv庫,或者pandas庫
with open(data_path 'driving_log.csv','r') as csvfile:
file_reader = csv.reader(csvfile,delimiter=',')
log = []
for row in file_reader:
log.append(row)
log = np.array(log)
log = log[1:,:] #remove the header
# 列印出log檔案中有多少張圖片以及轉角資料
print('DataSet: \n {} images| Numbers of steering data:{}'.format(len(log)*3,len(log)))
# 驗證數量是否匹配
ls_imgs = glob.glob(data_path 'IMG/*.jpg')
#assert len(ls_imgs)==len(log)*3 #這裡我們發現實際圖片比log中記錄的圖片多出了三張
len(ls_imgs)
DataSet: 
22362 images| Numbers of steering data:7454
22365

Image Visualization

這裡我們將圖片顯示出來並,配上對應的轉角

# 隨機選擇20張圖片
n_imgs =20
steering_col = 3
throttle_col = 4
ls_imgs = np.random.choice(np.arange(len(log)),size=n_imgs,replace = False)
_,ax = plt.subplots(4,5,figsize=(10,5))
col,row = 0,0
for i in ls_imgs:
img = cv2.imread(log[i,0])
#print(data_path log[i,0])
img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)#cv2.imread()預設讀取影象的格式是BGR;numpy.imread()讀取影象的格式是RGB
ax[row,col].imshow(img)
sub_title = 'st.{:.2f}|thr.{:.2f}'.format(float(log[i,steering_col]),float(log[i,throttle_col]))
ax[row,col].text(3,-5,sub_title,fontsize=9)
ax[row,col].get_xaxis().set_ticks([])
ax[row,col].get_yaxis().set_ticks([])
if col == 4:
row,col = row 1,0
else:
col =1
plt.show()

png

_,ax = plt.subplots(1,2,figsize=(10,5))
ax[0].plot(np.arange(200),log[200:400,steering_col].astype(float),'r-o')
ax[0].set_xlabel('frame',fontsize=14)
ax[0].set_ylabel('steering angle(rad)',fontsize=14)
ax[1].plot(np.arange(200),log[200:400,throttle_col].astype(float),'b-o')
ax[1].set_xlabel('frame',fontsize=14)
ax[1].set_ylabel('throttle',fontsize=14)
plt.show()

png

steering angle exploration

# 設定轉角值得精度為小數點後4位
precision = 0.0001
steering_angle = log[:,3].astype(float)*1/precision
steering_angle = steering_angle.astype(int)*precision
unique_st_angles = np.unique(steering_angle)
# 看一下轉角的分佈情況
print('Number of unique angles value(precision = 0.0001):{}'.format(len(unique_st_angles)))
# 畫出轉角的分佈直方圖
hist = []
for i in unique_st_angles:
count = np.sum(steering_angle==i)
hist.append([i,count])
hist = np.array(hist)
_,ax = plt.subplots(1,2,figsize=(15,5))
ax[0].bar(hist[:,0],hist[:,1],width=0.01,color='b')
ax[0].set_xlabel('steering angle (rad)',fontsize=14)
ax[0].set_ylabel('count',fontsize=14)
ax[1].bar(hist[:,0],np.log(hist[:,1]),width=0.01,color='b')
ax[1].set_xlabel('steering angle (rad)',fontsize=14)
ax[1].set_ylabel('log(count)',fontsize=14)
plt.show()
cnt_neg = np.sum(steering_angle<0)
cnt_pos = np.sum(steering_angle>0)
print('Count of positive angles:{}|Negative angles:{}'.format(cnt_pos,cnt_neg))
Number of unique angles value(precision = 0.0001):1508

png

Count of positive angles:255|Negative angles:1482

在分類問題中,我們常常需要去檢查樣本是否是均衡分佈的。比如在上文中的樣本中負角度的數量是正角度的7倍!模型就會有著極大地可能性去預測0值及負值。由於我一開始沒考慮到這塊,直接導致我使用模型進行自動駕駛時,車一開始便往左行駛。
因此我們準備直接使用官方提供的資料集進行進一步的分析。連結如下:data_set

我們來重新探索下資料集,並看下資料的分佈

data_path = 'data/'
# 開啟CSV檔案,這裡可以用csv庫,或者pandas庫
with open(data_path 'driving_log.csv','r') as csvfile:
file_reader = csv.reader(csvfile,delimiter=',')
log = []
for row in file_reader:
log.append(row)
log = np.array(log)
log = log[1:,:] #remove the header
# 列印出log檔案中有多少張圖片以及轉角資料
print('DataSet: \n {} images| Numbers of steering data:{}'.format(len(log)*3,len(log)))
# 驗證數量是否匹配
ls_imgs = glob.glob(data_path 'IMG/*.jpg')
assert len(ls_imgs)==len(log)*3 
_,ax = plt.subplots(1,2,figsize=(10,5))
ax[0].plot(np.arange(200),log[200:400,steering_col].astype(float),'r-o')
ax[0].set_xlabel('frame',fontsize=14)
ax[0].set_ylabel('steering angle(rad)',fontsize=14)
ax[1].plot(np.arange(200),log[200:400,throttle_col].astype(float),'b-o')
ax[1].set_xlabel('frame',fontsize=14)
ax[1].set_ylabel('throttle',fontsize=14)
plt.show()
# 設定轉角值得精度為小數點後4位
precision = 0.0001
steering_angle = log[:,3].astype(float)*1/precision
steering_angle = steering_angle.astype(int)*precision
unique_st_angles = np.unique(steering_angle)
# 看一下轉角的分佈情況
print('Number of unique angles value(precision = 0.0001):{}'.format(len(unique_st_angles)))
# 畫出轉角的分佈直方圖
hist = []
for i in unique_st_angles:
count = np.sum(steering_angle==i)
hist.append([i,count])
hist = np.array(hist)
_,ax = plt.subplots(1,2,figsize=(15,5))
ax[0].bar(hist[:,0],hist[:,1],width=0.01,color='b')
ax[0].set_xlabel('steering angle (rad)',fontsize=14)
ax[0].set_ylabel('count',fontsize=14)
ax[1].bar(hist[:,0],np.log(hist[:,1]),width=0.01,color='b')
ax[1].set_xlabel('steering angle (rad)',fontsize=14)
ax[1].set_ylabel('log(count)',fontsize=14)
plt.show()
cnt_neg = np.sum(steering_angle<0)
cnt_pos = np.sum(steering_angle>0)
print('Count of positive angles:{}|Negative angles:{}'.format(cnt_pos,cnt_neg))
DataSet: 
24108 images| Numbers of steering data:8036

png

Number of unique angles value(precision = 0.0001):124

png

Count of positive angles:1900|Negative angles:1775

很明顯上圖中,負角度與正角度的分佈就比較均衡了

throttle = log[:,4].astype(float)
unique_throttle = np.unique(throttle)
print('Number of unique throttle opening value:{}'.format(len(unique_throttle)))
hist = []
for i in unique_throttle:
count = np.sum(throttle==i)
hist.append([i,count])
hist = np.array(hist)
_,ax = plt.subplots(1,2,figsize=(15,5))
ax[0].bar(hist[:,0],hist[:,1],width=0.01,color='b')
ax[0].set_xlabel('throttle opeing',fontsize=14)
ax[0].set_ylabel('count',fontsize=14)
ax[1].bar(hist[:,0],np.log(hist[:,1]),width=0.01,color='b')
ax[1].set_xlabel('throttle opening',fontsize=14)
ax[1].set_ylabel('log(count)',fontsize=14)
plt.show()
Number of unique throttle opening value:95

png

Data Preparation

DataSplit

在對影象進行預處理之前,我們可以將資料集按照5:1的比例劃分成訓練集與驗證集

Region of Interest

在模型的使用中,我們希望模型能夠專注於與駕駛相關的畫素或特徵上,例如,車道線。因此我們會將影象沿垂直方向進行裁剪,底部裁剪20px,頂部裁剪80px,並將影象resize到(128,128)

Data augmentation

接下里我們將從亮度、水平翻轉、cross-track error幾個角度來進行資料增強

# Variation of brightness
#這裡我們隨機選取一張圖片,生成9張不同brightness的圖片,這裡將亮度值調暗比例設定在0.1-1之間,值越小,圖片亮度越暗
n_imgs = 1
ls_imgs = np.random.choice(np.arange(len(log)),size=n_imgs,replace=False)
_,ax = plt.subplots(3,3,figsize=(12,5))
col,row = 0,0
for i in range(9):
img_adress = data_path   log[ls_imgs[0],0]
img = cv2.imread(img_adress)
img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)
hsv = cv2.cvtColor(img,cv2.COLOR_RGB2HSV)#HSV又稱作HSB,B指Brightness
alpha = np.random.uniform(low=0.1,high=1.0,size=None)
v = hsv[:,:,2]
v = v*alpha
hsv[:,:,2] = v.astype('uint8')
rgb = cv2.cvtColor(hsv.astype('uint8'),cv2.COLOR_HSV2RGB)
ax[row,col].imshow(rgb)
title_ = 'alpha beightness {:.2f}'.format(alpha)
ax[row,col].text(3,-8,title_,fontsize=10,color='black')
ax[row,col].get_xaxis().set_ticks([])
ax[row,col].get_yaxis().set_ticks([])
if col==2:
row,col = row 1,0
else:
col =1
plt.show()

png

#水平翻轉
#為了增大資料集,我們可以將圖片水平翻轉
n_imgs = 5
ls_imgs = np.random.choice(np.arange(len(log)),size=n_imgs,replace=False)
_,ax = plt.subplots(2,n_imgs,figsize=(12,3))
col,row = 0,0
for i in ls_imgs:
img = cv2.imread(data_path log[i,0])
img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)
ax[row,col].imshow(img)
ax[row,col].text(3,-5,float(log[i,3]),fontsize=10,color='black')
ax[row,col].get_xaxis().set_ticks([])
ax[row,col].get_yaxis().set_ticks([])
ax[row 1,col].imshow(cv2.flip(img,1))
ax[row 1,col].text(3,-5,-float(log[i,3]),fontsize=10,color='red')
ax[row 1,col].get_xaxis().set_ticks([])
ax[row 1,col].get_yaxis().set_ticks([])
col =1
plt.show()

png

# cross-track error
# 從理論上講,我們對左右攝像頭之間的轉角有一個粗略的估計,假如左攝像頭與參考路徑的夾角為S1,中間攝像頭夾角為s2,右邊攝像頭的夾角為s3,
# 那麼這些夾角之間存在這樣的公式:s1 = s2 0.25;s3 = s2-0.25
def angle_viewpoint(st_angle,camera_side='center'):
if camera_side == 'left':
return st_angle 0.25
if camera_side == 'right':
return st_angle-0.25
else:
return st_angle
n_imgs = 5
ls_imgs = np.random.choice(np.arange(len(log)),size=n_imgs,replace=False)
_,ax = plt.subplots(3,n_imgs,figsize=(12,5))
col,row = 0,0
for i in ls_imgs:
for k,viewpoint in enumerate(['left','center','right']):
img_adress = data_path   log[i,0].replace('center',viewpoint)
img = cv2.imread(img_adress)
img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)
ax[row k,col].imshow(img)
new_angle = angle_viewpoint(float(log[i,3]),viewpoint)
title_ = viewpoint   '|'   "{:.3f}".format(new_angle)
ax[row k,col].text(3,-8,title_,fontsize=10,color='black')
ax[row k,col].get_xaxis().set_ticks([])
ax[row k,col].get_yaxis().set_ticks([])
col =1
plt.show()

png

除了上述介紹的三種資料增強的方法,我們還可以對資料集做如下處理:
1.去除資料集中轉角為0的影象資料
2.縮小畫素值的範圍,原先畫素值的範圍是0-255,現在我們可以將畫素範圍縮小到-0.5-0.5
接下來我們把上述介紹的一些資料增強的方法,封裝成函式

def horizontal_flip(img,label):
'''
隨機水平翻轉影象,概率為0.5
img:原始影象 in array type
label: 原始影象的轉角值
'''
choice = np.random.choice([0,1])
if choice==1:
img,label = cv2.flip(img,1),-label
return (img,label)
def transf_brightness(img,label):
'''
將一張圖片的亮度值調暗,比例為0.1-1之間的隨機數
img:原始影象 in array type
label:原始影象的轉角值
'''
hsv = cv2.cvtColor(img,cv2.COLOR_RGB2HSV)
alpha = np.random.uniform(low=0.1,high=1.0,size=None)
v = hsv[:,:,2]
v = v*alpha
hsv[:,:,2] = v.astype('uint8')
rgb = cv2.cvtColor(hsv.astype('uint8'),cv2.COLOR_HSV2RGB)
return (rgb,label)
def center_RightLeft_swap(img_adress,label,label_corr=0.25):
'''
這裡我們規定汽車校準行駛距離為4m,中間攝像頭距離左右攝像頭的距離均為1m
img_adress:physical location of the original image file
label:steering angle value of original image
label_corr:correction of the steering angle to applied.default value=1/4
'''
swap = np.random.choice(['L','R','C'])
if swap=='L':
img_adress = img_adress.replace('center','left')
corrected_label = label   label_corr
return (img_adress,corrected_label)
elif swap=='R':
img_adress = img_adress.replace('center','right')
corrected_label = label - label_corr
return (img_adress,corrected_label)
else:
return (img_adress,label)
def filter_zero_steering(label,del_rate):
'''
label: list of steering angle value in the original dataset
del_rate:rate of deletion-del_rate=0.9 means delete 90% of example with steering angle=0
'''
steering_zero_idx = np.where(label==0)
steering_zero_idx = steering_zero_idx[0]
size_del = int(len(steering_zero_idx)*del_rate)
return np.random.choice(steering_zero_idx,size=size_del,replace=False)
def image_transformation(img_adress,label,data_dir):
#img swap
img_adress , label = center_RightLeft_swap(img_adress,label)
# Read img file
img = cv2.imread(data_dir img_adress)
img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)
img , label = transf_brightness(img,label)
#flip image
img , label = horizontal_flip(img,label)
return (img,label)

Modeling

這裡由於模型引數過多(博主的機子配置差),我實際實現程式碼的過程中,增加了一層卷積,並調整了kernel的大小,將引數個數降低了10倍
接下來我們將實現一個神經網路模型。我們定義一張圖片的輸入為128*128*3,模型由4個卷積層及3個全連線層構成,具體結構如下所述:

  • 第一層卷積層過濾器的尺寸為5×5,深度為8,步長為1,不使用全0填充,所以這一層輸出的尺寸為(124,124)深度為8.那麼這一層共有引數5x5x3x8 8=608
  • 第二層池化層,採用過濾器的大小為2×2,步長為2,不使用全0填充,所以這一層輸出的尺寸為(62,62)深度為8
  • 第三層卷積層過濾器的尺寸為5×5,深度為8,步長為1,不使用全0填充,所以這一層輸出尺寸為(58,58)深度為8,那麼這一層共有引數5x5x8x8 8=1608
  • 第四層池化層,採用過濾器的大小為2×2,步長為2,不使用全0填充,所以這一層輸出的尺寸為(29,29)深度為8
  • 第五層卷積層過濾器的尺寸為4×4,深度為16,步長為1,不使用全0填充,所以這一層輸出尺寸為(26,26)深度為16,那麼這一層共有引數4x4x8x16 16=2064
  • 第六層池化層,,採用過濾器的大小為2×2,步長為2,不使用全0填充,所以這一層輸出的尺寸為(13,13)深度為16
  • 第七層卷積層過濾器的尺寸為5×5,深度為16,步長為1,不是用全0填充,所以這一層輸出尺寸為(9,9)深度為16,那麼這一層共有引數5x5x16x16 16=6416
  • “展平層”,尺寸大小為1296
  • 第八層全連線層,神經元個數128,引數個數1296×128 128=166016
  • 第九層全連線層,神經元個數50,引數個數128×50 50=6450
  • 第十層全連線層,神經元個數10,引數個數50×10 10=510
  • 第十一層全連線層,神經元個數為1,引數個數是10×1 1=11

模型結構圖如下所示:
這裡寫圖片描述

def continuousSteering(img_sz,activation_fn='relu',l2_reg=[10**-3,10**-3]):
pool_size = (2,2)
model = Sequential()
model.add(Conv2D(filters=8,kernel_size=(5,5),strides=(1,1),padding='valid',name='conv1',input_shape=img_sz,kernel_regularizer=l2(l2_reg[1])))
if activation_fn=='elu':
model.add(Activation('elu'))
elif activation_fn=='prelu':
model.add(PReLU())
else:
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size,padding='valid'))
model.add(Conv2D(filters=8,kernel_size=(5,5),strides=(1,1),padding='valid',kernel_regularizer=l2(l2_reg[1])))
if activation_fn=='elu':
model.add(Activation('elu'))
elif activation_fn=='prelu':
model.add(PReLU())
else:
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=pool_size,padding='valid'))
model.add(Conv2D(filters=8,kernel_size=(5,5),strides=(1,1),padding='valid',kernel_regularizer=l2(l2_reg[1])))
if activation_fn=='elu':
model.add(Activation('elu'))
elif activation_fn=='prelu':
model.add(PReLU())
else:
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=pool_size,padding='valid'))
model.add(Conv2D(filters=16,kernel_size=(5,5),strides=(1,1),padding='valid',kernel_regularizer=l2(l2_reg[1])))
if activation_fn=='elu':
model.add(Activation('elu'))
elif activation_fn=='prelu':
model.add(PReLU())
else:
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=pool_size,padding='valid'))
model.add(Conv2D(filters=16,kernel_size=(4,4),strides=(1,1),padding='valid',kernel_regularizer=l2(l2_reg[1])))
if activation_fn=='elu':
model.add(Activation('elu'))
elif activation_fn=='prelu':
model.add(PReLU())
else:
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(128,kernel_regularizer=l2(l2_reg[0])))
if activation_fn=='elu':
model.add(Activation('elu'))
elif activation_fn=='prelu':
model.add(PReLU())
else:
model.add(Activation('relu'))
model.add(Dense(50,kernel_regularizer=l2(l2_reg[0])))
if activation_fn=='elu':
model.add(Activation('elu'))
elif activation_fn=='prelu':
model.add(PReLU())
else:
model.add(Activation('relu'))
model.add(Dense(10,kernel_regularizer=l2(l2_reg[0])))
if activation_fn=='elu':
model.add(Activation('elu'))
elif activation_fn=='prelu':
model.add(PReLU())
else:
model.add(Activation('relu'))
model.add(Dense(1,activation='linear',kernel_regularizer=l2(l2_reg[0]),kernel_initializer='he_normal'))
adam = Adam(lr=0.001)
model.compile(optimizer=adam,loss='mean_squared_error')
print('LightWeight Model is created and compiled..-activation:{}'.format(activation_fn))
return model
def batch_generator(x,y,batch_size,img_sz,training=True,del_rate=0.95,data_dir='data/',monitor=True,yieldXY=True):
'''
生成訓練batch:利用yield關鍵詞修飾
資料增強機制:水平翻轉、改變亮度、改變視角
在每一輪訓練中,在使用資料增強前,先刪除95%的轉角為0的資料。
x:被用作訓練的所有影象的地址
y:轉角
training:如果為True,生成資料增強後的訓練集;如果為false,生成驗證集
batch_size:
img_sz:生成的影象大小(height,width,channel)
del_rate:需要刪除轉角為0的樣本比例
data_dir:圖片的地址目錄
monitor:是否將特徵與標籤進行儲存
yieldXY:如果為真,yields (x,y),否則只yields x
'''
if training:
y_bag = []
x,y = shuffle(x,y)
rand_zero_idx = filter_zero_steering(y,del_rate)
new_x = np.delete(x,rand_zero_idx,axis=0)
new_y = np.delete(y,rand_zero_idx,axis=0)
else:
new_x = x
new_y = y
offset = 0
while True:
X = np.empty((batch_size,*img_sz))
Y = np.empty((batch_size,1))
#generate a batch
for example in range(batch_size):
img_adress,img_steering = new_x[example offset],new_y[example offset]
assert os.path.exists(data_dir img_adress),'Image file[' img_adress '] not found-'
if training:
img, img_steering = image_transformation(img_adress,img_steering,data_dir)
else:
img = cv2.imread(data_dir img_adress)
img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)
#crop、resize、scale
X[example,:,:,:] = cv2.resize(img[80:140,0:320],(img_sz[0],img_sz[1]))/255.0-0.5
Y[example] = img_steering
if training:
y_bag.append(img_steering)
'''
如果到達原始資料集末尾,就從頭開始迴圈
'''
if (example 1) offset > len(new_y)-1:
x , y = shuffle(x,y)
rand_zero_idx = filter_zero_steering(y,del_rate=del_rate)
new_x = x
new_y = y
new_x = np.delete(new_x,rand_zero_idx,axis=0)
new_y = np.delete(new_y,rand_zero_idx,axis=0)
offset = 0
if yieldXY:
yield (X,Y)
else:
yield X
offset = offset batch_size
if training:
np.save('y_bag.npy',np.array(y_bag))
np.save('Xbatch_sample.npy',X)#這裡儲存的是最後一個batch的images
#這裡設定一些引數
test_size = 0.2
img_sz = (128,128,3)
batch_size = 200
data_agumentation = 200
nb_epoch = 15
del_rate = 0.95
activation_fn = 'relu'
l2_reg = [0.00001,0]
print('Total number of samples per EPOCH:{}'.format(batch_size*data_agumentation))
x_ = log[:,0]
y_ = log[:,3].astype(float)
x_,y_ = shuffle(x_,y_)
X_train,X_val,y_train,y_val = train_test_split(x_,y_,test_size=test_size,random_state=seed)
print('Train set size:{}|Validation set size:{}'.format(len(X_train),len(X_val)))
sample_per_epoch = batch_size*data_agumentation
#這裡我們確保驗證集大小為batch_size的倍數
nb_val_samples = len(y_val) - len(y_val)%batch_size
Total number of samples per EPOCH:40000
Train set size:6428|Validation set size:1608
#train model
model = continuousSteering(img_sz,activation_fn = activation_fn,l2_reg=l2_reg)
print(model.summary())
'''
Callbacks:給予驗證集損失儲存最佳訓練輪數
1.如果驗證損失降低了,儲存模型或者2.當驗證損失連續5次epoch不再改善就停止訓練並儲存模型
'''
model_path = os.path.expanduser('model.h5')
save_best = callbacks.ModelCheckpoint(model_path,monitor='val_loss',verbose=1,save_best_only=True,mode='min')#Save the model after every epoch.
early_stop = callbacks.EarlyStopping(monitor='val_loss',min_delta=0,patience=15,verbose=0,mode='auto')
callbacks_list = [early_stop,save_best]
# batch generator default value:
history = model.fit_generator(batch_generator(X_train,y_train,batch_size,img_sz,training=True,del_rate=del_rate),
steps_per_epoch=200,
validation_steps= 80,
validation_data = batch_generator(X_val,y_val,batch_size,img_sz,training=False,monitor=False),
epochs = nb_epoch,verbose=1,callbacks=callbacks_list)
with open('model.json','w') as f:
f.write(model.to_json())
model.save('model_.h5')
print('Model saved!')
LightWeight Model is created and compiled..-activation:relu
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv1 (Conv2D)               (None, 124, 124, 8)       608       
_________________________________________________________________
activation_9 (Activation)    (None, 124, 124, 8)       0         
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 62, 62, 8)         0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 58, 58, 8)         1608      
_________________________________________________________________
activation_10 (Activation)   (None, 58, 58, 8)         0         
_________________________________________________________________
max_pooling2d_6 (MaxPooling2 (None, 29, 29, 8)         0         
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 25, 25, 8)         1608      
_________________________________________________________________
activation_11 (Activation)   (None, 25, 25, 8)         0         
_________________________________________________________________
max_pooling2d_7 (MaxPooling2 (None, 12, 12, 8)         0         
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 8, 8, 16)          3216      
_________________________________________________________________
activation_12 (Activation)   (None, 8, 8, 16)          0         
_________________________________________________________________
max_pooling2d_8 (MaxPooling2 (None, 4, 4, 16)          0         
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 1, 1, 16)          4112      
_________________________________________________________________
activation_13 (Activation)   (None, 1, 1, 16)          0         
_________________________________________________________________
flatten_2 (Flatten)          (None, 16)                0         
_________________________________________________________________
dense_5 (Dense)              (None, 128)               2176      
_________________________________________________________________
activation_14 (Activation)   (None, 128)               0         
_________________________________________________________________
dense_6 (Dense)              (None, 50)                6450      
_________________________________________________________________
activation_15 (Activation)   (None, 50)                0         
_________________________________________________________________
dense_7 (Dense)              (None, 10)                510       
_________________________________________________________________
activation_16 (Activation)   (None, 10)                0         
_________________________________________________________________
dense_8 (Dense)              (None, 1)                 11        
=================================================================
Total params: 20,299
Trainable params: 20,299
Non-trainable params: 0
_________________________________________________________________
None
Epoch 1/15
200/200 [==============================] - 133s 667ms/step - loss: 0.0420 - val_loss: 0.0267
Epoch 00001: val_loss improved from inf to 0.02667, saving model to model.h5
Epoch 2/15
200/200 [==============================] - 128s 641ms/step - loss: 0.0295 - val_loss: 0.0223
Epoch 00002: val_loss improved from 0.02667 to 0.02227, saving model to model.h5
Epoch 3/15
200/200 [==============================] - 129s 644ms/step - loss: 0.0284 - val_loss: 0.0246
Epoch 00003: val_loss did not improve
Epoch 4/15
200/200 [==============================] - 131s 653ms/step - loss: 0.0274 - val_loss: 0.0267
Epoch 00004: val_loss did not improve
Epoch 5/15
200/200 [==============================] - 132s 658ms/step - loss: 0.0271 - val_loss: 0.0253
Epoch 00005: val_loss did not improve
Epoch 6/15
200/200 [==============================] - 133s 667ms/step - loss: 0.0265 - val_loss: 0.0238
Epoch 00006: val_loss did not improve
Epoch 7/15
200/200 [==============================] - 133s 667ms/step - loss: 0.0264 - val_loss: 0.0239
Epoch 00007: val_loss did not improve
Epoch 8/15
200/200 [==============================] - 133s 667ms/step - loss: 0.0255 - val_loss: 0.0245
Epoch 00008: val_loss did not improve
Epoch 9/15
200/200 [==============================] - 134s 669ms/step - loss: 0.0252 - val_loss: 0.0236
Epoch 00009: val_loss did not improve
Epoch 10/15
200/200 [==============================] - 134s 671ms/step - loss: 0.0248 - val_loss: 0.0226
Epoch 00010: val_loss did not improve
Epoch 11/15
200/200 [==============================] - 134s 671ms/step - loss: 0.0244 - val_loss: 0.0213
Epoch 00011: val_loss improved from 0.02227 to 0.02133, saving model to model.h5
Epoch 12/15
200/200 [==============================] - 134s 670ms/step - loss: 0.0241 - val_loss: 0.0222
Epoch 00012: val_loss did not improve
Epoch 13/15
200/200 [==============================] - 134s 672ms/step - loss: 0.0233 - val_loss: 0.0213
Epoch 00013: val_loss improved from 0.02133 to 0.02127, saving model to model.h5
Epoch 14/15
200/200 [==============================] - 135s 673ms/step - loss: 0.0232 - val_loss: 0.0205
Epoch 00014: val_loss improved from 0.02127 to 0.02051, saving model to model.h5
Epoch 15/15
200/200 [==============================] - 136s 681ms/step - loss: 0.0232 - val_loss: 0.0217
Epoch 00015: val_loss did not improve
Model saved!
#from keras.models import load_model
model = load_model('model_.h5')
model.compile("adam", "mse")
model.load_weights('model_.h5')
#steering angle prediction
duration = 200
#select a random start frame
start_frame = np.random.choice(np.arange(0,len(log)-int(duration),1))
data4eval = log[start_frame:start_frame duration,[0,3]]
steering_pred = []
for img_adress in data4eval[:,0]:
img = cv2.imread('data/' img_adress)
img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)
img = img[80:140,0:320]#crop image
img = cv2.resize(img,(128,128))/255.0-0.5
transformed_image_array = img[None,:,:,:]
steering_angle = float(model.predict(transformed_image_array,batch_size=1))
steering_pred.append(steering_angle)
type(data4eval[0,1])
numpy.str_
plt.plot(np.arange(0, int(duration), 1), steering_pred, color='red', label='predicted')
plt.plot(np.arange(0, int(duration), 1), data4eval[:,1].astype(float), color='blue', label='actual')#此處data4eval[:,1]的資料型別為numpy.str_必須進行格式轉換
plt.xlabel('frames', fontsize=12)
plt.ylabel('steering angle (rad)', fontsize=12)
plt.legend()
plt.show()

png

#下面我們對預處理後的圖片進行視覺化
img_check = np.load('Xbatch_sample.npy')
y_check = np.load('y_bag.npy')
n_imgs = 20
ls_imgs = np.random.choice(np.arange(len(img_check)),size=n_imgs,replace=False)
_,ax = plt.subplots(4,5,figsize=(12,8))
col,row = 0,0
for i in ls_imgs:
img = (img_check[i,:,:,:] 0.5)*255#還原
ax[row,col].imshow(img.astype('uint8'))
title_ = 'st_angle:{:.2f}'.format(y_check[i])
ax[row,col].text(3,-8,title_,fontsize=10,color='black')
ax[row,col].get_xaxis().set_ticks([])
ax[row,col].get_yaxis().set_ticks([])
if col==4:
row,col = row 1,0
else:
col =1
plt.show()

png

#視覺化過濾器,這裡我們選擇特定層來視覺化
_,ax = plt.subplots(4,3,figsize=(5,7))
col,row = 0,0
print('Filters shape for the convolutional layers(format==[height,width,depth,nbr])')
for layer_idx in [0,3,6,9]:
weights = np.array(model.layers[laye r_idx].get_weights())[0]
convlayer = int(layer_idx/3 1)
print('\t conv{}:W{}'.format(convlayer,weights.shape))
for kernel_idx in [0,1,2]:
kernel = weights[:,:,0,kernel_idx]
ax[row,col].imshow(kernel,cmap='gray',interpolation='none')
ax[row,col].get_xaxis().set_ticks([])
ax[row,col].get_yaxis().set_ticks([])
ax[row,col].title.set_text('conv layer' str(convlayer))
col =1
row =1
col=0
plt.show()
Filters shape for the convolutional layers(format==[height,width,depth,nbr])
conv1:W(5, 5, 3, 8)
conv2:W(5, 5, 8, 8)
conv3:W(5, 5, 8, 8)
conv4:W(5, 5, 8, 16)

png

視覺化卷積層後的啟用層

這裡我們選取4張圖片, 並將模型的4個卷積層後的啟用層輸出給顯示出來

n_imgs = 4
ls_img = np.random.choice(np.arange(len(img_check)), size = n_imgs, replace=False)
_, ax = plt.subplots( 5, 4, figsize=(20,20) )
row, col = 0, 0
for k, img_id in enumerate(ls_img):
img_input = img_check[img_id, :,:,:]
img_input = img_input.reshape((1, *img_sz) )
for layer_id in [-1, 1, 4, 7, 10]:
if layer_id == -1: #show input image
img_4display = (img_input[0]   0.5) * 255 
ax[row, col].imshow(img_4display.astype('uint8'))
title_ = 'original image {} | {}'.format(k   1, img_4display.shape)
ax[row, col].text(3, -8, title_, fontsize=10, color='black')
ax[row, col].get_xaxis().set_ticks([])
ax[row, col].get_yaxis().set_ticks([])
row = row   1
else:
get_layer_output = K.function([model.layers[0].input], [model.layers[layer_id].output])
layer_output = get_layer_output([img_input])[0]
title_ = 'activation conv{} | {}'.format(layer_id, layer_output[0, :,:,0].shape)
ax[row, col].set_title(title_, fontsize=10, color='black')
ax[row, col].imshow(layer_output[0,:,:,0], cmap='gray') #display 1st activation map
ax[row, col].get_xaxis().set_ticks([])
ax[row, col].get_yaxis().set_ticks([])
if row == 4:
row = 0
col  = 1
else:
row  = 1

png