- 🔬 Data Science
- 🥠 Deep Learning and Object Detection
- 🛤️ Tracking
Introduction and objective
Vehicle detection and tracking is a common problem with multiple use cases. Government authorities and private establishment might want to understand the traffic flowing through a place to better develop its infrastructure for the ease and convenience of everyone. A road widening project, timing the traffic signals and construction of parking spaces are a few examples where analysing the traffic is integral to the project.
Traditionally, identification and tracking has been carried out manually. A person will stand at a point and note the count of the vehicles and their types. Recently, sensors have been put into use, but they only solve the counting problem. Sensors will not be able to detect the type of vehicle.
In this notebook, we'll demonstrate how we can use deep learning to detect vehicles and then track them in a video. We'll use a short video taken from live traffic camera feed.
Necessary imports
import os
import pandas as pd
from pathlib import Path
from arcgis.gis import GIS
from arcgis.learn import RetinaNet, prepare_data
gis = GIS('home')
Prepare data that will be used for training
You can download vehicle training data from here. Extract the downloaded file to get your training data.
Model training
Let's set a path to the folder that contains training images and their corresponding labels.
training_data = gis.content.get('ccaa060897e24b379a4ed2cfd263c15f')
training_data
filepath = training_data.download(file_name=training_data.name)
import zipfile
with zipfile.ZipFile(filepath, 'r') as zip_ref:
zip_ref.extractall(Path(filepath).parent)
data_path = Path(os.path.join(os.path.splitext(filepath)[0]))
We'll use the prepare_data
function to create a fastai databunch with the necessary parameters such as batch_size
, and chip_size
. A complete list of parameters can be found in the API reference.
The given dataset has 235 images of size 854x480 pixels. We will define a chip_size
of 480 pixels which will create random crops of 480x480 from the given images. This way we will maintain the aspect ratios of the objects but can miss out on objects when training the model for fewer epochs. To avoid cropping, we can set resize_to
=480 so that every chip is an entire frame and doesn't miss any object, but there is a risk of poor detection with smaller sized object.
data = prepare_data(data_path,
batch_size=4,
dataset_type="PASCAL_VOC_rectangles",
chip_size=480)
Please check your dataset. 9 images dont have the corresponding label files.
We see the warning above because there are a few images in our dataset with missing corresponding label files. These images will be ignored while loading the data. If it is a significant number, we might want to fix this issue by adding the label files for those images or removing those images.
We can use the classes
attribute of the data object to get information about the number of classes.
data.classes
['background', 'bicycle', 'bus', 'car', 'motorcycle', 'person', 'scooter', 'tempo', 'tractor', 'truck', 'van']
Visualize training data
To visualize and get a sense of the training data, we can use the data.show_batch
method.
data.show_batch()
In the previous cell, we see a sample of the dataset. We can observe, in the given chips, that the most common vehicles are cars and bicycles. It can also be noticed that the different instance of the vehicles have varying scales.
Load model architecture
arcgis.learn
provides us object detection models which are based on pretrained convnets, such as ResNet, that act as the backbones. We will use RetinaNet
with the default parameters to create our vehicle detection model. For more details on RetinaNet
check out How RetinaNet works? and the API reference.
retinanet = RetinaNet(data)
We will use the lr_find()
method to find an optimum learning rate. It is important to set a learning rate at which we can train a model with good accuracy and speed.
lr = retinanet.lr_find()
4.365158322401661e-05
Train the model
We will now train the RetinaNet
model using the suggested learning rate from the previous step. We can specify how many epochs we want to train for. Let's train the model for 100 epochs. Also, we can turn tensorboard
True if we want to visualize the training process in tensorboard.
retinanet.fit(100, lr=lr, tensorboard=True)
epoch | train_loss | valid_loss | time |
---|---|---|---|
0 | 2.651160 | 3.122699 | 00:33 |
1 | 2.727485 | 3.089710 | 00:32 |
2 | 2.744920 | 3.015922 | 00:32 |
3 | 2.671797 | 2.851994 | 00:31 |
4 | 2.457554 | 2.497410 | 00:31 |
5 | 2.381740 | 2.328834 | 00:31 |
6 | 2.060174 | 4.138567 | 00:31 |
7 | 1.792403 | 21.451857 | 00:31 |
8 | 1.712977 | 4.193508 | 00:31 |
9 | 1.608706 | 4.876813 | 00:32 |
10 | 1.496329 | 4.955950 | 00:32 |
11 | 1.575526 | 2.124239 | 00:33 |
12 | 1.448479 | 2.765982 | 00:31 |
13 | 1.356783 | 2.739088 | 00:31 |
14 | 1.296036 | 1.941170 | 00:32 |
15 | 1.235588 | 3.042969 | 00:32 |
16 | 1.177469 | 2.916740 | 00:32 |
17 | 1.163151 | 2.462182 | 00:32 |
18 | 1.124477 | 1.952319 | 00:32 |
19 | 1.055723 | 2.639346 | 00:32 |
20 | 0.976554 | 1.884056 | 00:32 |
21 | 0.865862 | 1.545389 | 00:32 |
22 | 0.885476 | 1.693674 | 00:32 |
23 | 0.861983 | 1.386624 | 00:32 |
24 | 0.812286 | 1.257245 | 00:33 |
25 | 0.794138 | 1.578588 | 00:32 |
26 | 0.765640 | 1.208835 | 00:34 |
27 | 0.702818 | 1.117395 | 00:32 |
28 | 0.669110 | 1.213653 | 00:33 |
29 | 0.674798 | 1.130191 | 00:32 |
30 | 0.675300 | 1.154881 | 00:32 |
31 | 0.680791 | 1.257907 | 00:33 |
32 | 0.655586 | 1.072347 | 00:32 |
33 | 0.586407 | 1.009210 | 00:32 |
34 | 0.570755 | 1.220290 | 00:33 |
35 | 0.590223 | 0.982790 | 00:34 |
36 | 0.575041 | 0.997690 | 00:33 |
37 | 0.585412 | 1.035814 | 00:33 |
38 | 0.572887 | 1.015082 | 00:33 |
39 | 0.552126 | 0.949728 | 00:32 |
40 | 0.535455 | 1.195224 | 00:33 |
41 | 0.499169 | 0.946746 | 00:33 |
42 | 0.527345 | 1.009812 | 00:34 |
43 | 0.547029 | 0.991675 | 00:33 |
44 | 0.515441 | 0.906661 | 00:33 |
45 | 0.547948 | 0.986166 | 00:33 |
46 | 0.517109 | 0.943002 | 00:33 |
47 | 0.474826 | 0.894875 | 00:33 |
48 | 0.440434 | 0.909886 | 00:33 |
49 | 0.441918 | 0.819840 | 00:33 |
50 | 0.433040 | 0.837711 | 00:33 |
51 | 0.424501 | 0.834161 | 00:33 |
52 | 0.442397 | 0.825194 | 00:33 |
53 | 0.438501 | 0.778577 | 00:34 |
54 | 0.425794 | 0.790809 | 00:33 |
55 | 0.405544 | 0.774125 | 00:34 |
56 | 0.397529 | 0.751094 | 00:34 |
57 | 0.386021 | 0.756899 | 00:33 |
58 | 0.395799 | 0.763772 | 00:33 |
59 | 0.385372 | 0.785581 | 00:35 |
60 | 0.379765 | 0.767338 | 00:34 |
61 | 0.369503 | 0.720050 | 00:33 |
62 | 0.367806 | 0.720712 | 00:35 |
63 | 0.378731 | 0.734859 | 00:34 |
64 | 0.368838 | 0.729135 | 00:33 |
65 | 0.344555 | 0.700024 | 00:35 |
66 | 0.340411 | 0.743908 | 00:35 |
67 | 0.350800 | 0.718764 | 00:34 |
68 | 0.364890 | 0.715524 | 00:35 |
69 | 0.337952 | 0.688673 | 00:34 |
70 | 0.348077 | 0.719215 | 00:35 |
71 | 0.323196 | 0.700020 | 00:34 |
72 | 0.361027 | 0.719423 | 00:35 |
73 | 0.367712 | 0.719814 | 00:35 |
74 | 0.367507 | 0.693808 | 00:35 |
75 | 0.347651 | 0.708264 | 00:35 |
76 | 0.345269 | 0.705601 | 00:34 |
77 | 0.341163 | 0.719633 | 00:34 |
78 | 0.321359 | 0.719021 | 00:34 |
79 | 0.325086 | 0.710695 | 00:34 |
80 | 0.307621 | 0.709985 | 00:34 |
81 | 0.312010 | 0.695209 | 00:34 |
82 | 0.308455 | 0.723050 | 00:34 |
83 | 0.333749 | 0.721235 | 00:34 |
84 | 0.323337 | 0.718696 | 00:33 |
85 | 0.330353 | 0.709316 | 00:34 |
86 | 0.337785 | 0.728645 | 00:33 |
87 | 0.299953 | 0.732279 | 00:33 |
88 | 0.309058 | 0.723001 | 00:33 |
89 | 0.341413 | 0.749138 | 00:33 |
90 | 0.332262 | 0.734328 | 00:33 |
91 | 0.306863 | 0.716808 | 00:33 |
92 | 0.300803 | 0.737754 | 00:33 |
93 | 0.313041 | 0.714918 | 00:33 |
94 | 0.329477 | 0.711772 | 00:33 |
95 | 0.321354 | 0.714558 | 00:33 |
96 | 0.321379 | 0.701373 | 00:34 |
97 | 0.301340 | 0.726296 | 00:33 |
98 | 0.297174 | 0.726158 | 00:33 |
99 | 0.310064 | 0.736690 | 00:33 |
After the training is complete, we can view the plot with training and validation losses.
retinanet.learn.recorder.plot_losses()
Visualize results on validation set
To see sample results we can use the show_results
method. This method displays the chips from the validation dataset with ground truth (left) and predictions (right). We can also specify the threshold to view predictions at different confidence levels. This visual analysis helps in assessing the qualitative results of the trained model.
retinanet.show_results(thresh=0.4)
To see the quantitative results of our model we will use the average_precision_score
method.
retinanet.average_precision_score(detect_thresh=0.4)
{'bicycle': 0.6121794875615674, 'bus': 0.0, 'car': 0.770548729309354, 'motorcycle': 0.0, 'person': 0.0, 'scooter': 0.0, 'tempo': 0.0, 'tractor': 0.0, 'truck': 1.0, 'van': 0.38429487869143486}
We can see the average precision for each class in the validation dataset. Note that while car and bicycle have a good score, van doesn't, and a few have a score of 0. Remember when we visualized the data using show_batch
we noted that the cars and bicycles were the most common objects. It means, the scores could be correlated with the number of examples of these objects we have in our training dataset.
Let's look at the number of instances of each class in the training data and it should explain.
all_classes = []
for i, bb in enumerate(data.train_ds.y):
all_classes += bb.data[1].tolist()
df = pd.value_counts(all_classes, sort=False)
df.index = [data.classes[i] for i in df.index]
df
bicycle 266 bus 19 car 756 motorcycle 33 person 24 scooter 6 tempo 1 tractor 4 truck 30 van 69 dtype: int64
It is evident that the classes that have a score of 0.0 have extremely low number of examples in the training dataset.
Save the model
Let's save the model by giving it a name and calling the save
method, so that we can load
it later whenever required. The model is saved by default in a directory called models
in the data_path
initialized earlier, but a custom path can be provided.
retinanet.save('vehicle_det_ep100_defaults')
WindowsPath('vehicle_detection/models/vehicle_det_ep100_defaults')
Inference and tracking
Multiple-object tracking can be performed using predict_video
function of the arcgis.learn
module. To enable tracking, set the track
parameter in the predict_video
function as track=True
.
The following options/parameters are available in the predict video function for the user to decide:-
-
vanish_frames
: The number of frames the object remains absent from the frame to be considered as vanished. -
detect_frames
: The number of frames an object remains present in the frame to start tracking. -
assignment_iou_thrd
: There might be multiple trackers detecting and tracking objects. The Intersection over Union (iou) threshold can be set to assign a tracker with the mentioned threshold value.
video_data = gis.content.get('1801dc029fed467ba67d6e39113202af')
video_data
videopath = video_data.download(file_name=video_data.name)
import zipfile
with zipfile.ZipFile(videopath, 'r') as zip_ref:
zip_ref.extractall(Path(videopath).parent)
video_file = os.path.join(os.path.splitext(videopath)[0], 'test.mp4')
retinanet.predict_video(input_video_path=video_file,
metadata_file='test.csv',
track=True,
visualize=True,
threshold=0.5,
resize=True)