- 🔬 Data Science
- 🥠 Deep Learning and Object Detection
Prerequisites
- Please refer to the prerequisites section in our guide for more information. This sample demonstrates how to do export training data and model inference using ArcGIS Image Server. Alternatively, they can be done using ArcGIS Pro as well.
- If you have already exported training samples using ArcGIS Pro, you can jump straight to the training section. The saved model can also be imported into ArcGIS Pro directly.
Introduction and objective
Deep Learning has achieved great success with state of the art results, but taking it to the field and solving real-world problems is still a challenge. Integration of the latest research in AI with ArcGIS opens up a world of opportunities. This notebook demonstrates an end-to-end deep learning workflow in using ArcGIS API for Python. The workflow consists of three major steps: (1) extracting training data, (2) train a deep learning object detection model, (3) deploy the model for inference and create maps. To better illustrate this process, we choose detecting swmming pools in Redlands, CA using remote sensing imagery.
Part 1 - export training data for deep learning
Import ArcGIS API for Python and get connected to your GIS
import arcgis
import sys
from datetime import datetime as dt
from arcgis import GIS, learn
from arcgis.raster import analytics
from arcgis.raster.functions import extract_band, apply, clip
from arcgis.raster.analytics import list_datastore_content
from arcgis.learn import SingleShotDetector, prepare_data, Model, list_models, detect_objects
arcgis.env.verbose = True
from arcgis.geocoding import geocodegis = GIS('home')Prepare data that will be used for training data export
To export training data, we need a labeled feature class that contains the bounding box for each object, and a raster layer that contains all the pixels and band information. In this swimming pool detection case, we have created feature class by hand labelling the bounding box of each swimming pool in Redlands using ArcGIS Pro and USA NAIP Imagery: Color Infrared as raster data.
pool_bb = gis.content.get('744c0dfd56c04057b99e6f954020f609')
pool_bb
Feature Layer Collection by api_data_owner
pool_bb_layer = pool_bb.layers[0]
pool_bb_layer.url'https://services7.arcgis.com/JEwYeAy2cc8qOe3o/arcgis/rest/services/pools_redlands/FeatureServer/0'
Now let's retrieve the NAIP image layer.
naip_item = gis.content.get('b4adf04924fb4fdc8acb14dc7e1c08f5')
naip_item
Tiled Imagery Layer by api_data_owner
# Visualize layer over map
m = gis.map("LOMA LINDA")
m.content.add(naip_item)
m.zoom_to_layer(naip_item)
m
m.basemap.basemap = 'gray-vector'Specify a folder name in raster store that will be used to store our training data
Make sure a raster store is ready on your raster analytics image server. This is where where the output subimages, also called chips, labels and metadata files are going to be stored.
ds = analytics.get_datastores(gis=gis)
ds<DatastoreManager for https://pythonapi.playground.esri.com/ra/admin>
ds.search()[<Datastore title:"/fileShares/ListDatastoreContent" type:"folder">, <Datastore title:"/rasterStores/RasterDataStore" type:"rasterStore">]
rasterstore = ds.get("/rasterStores/RasterDataStore")
rasterstore<Datastore title:"/rasterStores/RasterDataStore" type:"rasterStore">
samplefolder = "pool_chips_test"
samplefolder'pool_chips_test'
Export training data using arcgis.learn
With the feature class and raster layer, we are now ready to export training data using the export_training_data() method in arcgis.learn module. In addtion to feature class, raster layer, and output folder, we also need to speficy a few other parameters such as tile_size (size of the image chips), strid_size (distance to move in the X when creating the next image chip), chip_format (TIFF, PNG, or JPEG), metadata format (how we are going to store those bounding boxes). More detail can be found here.
Depending on the size of your data, tile and stride size, and computing resources, this opertation can take 15mins~2hrs in our experiment. Also, do not re-run it if you already run it once unless you would like to update the setting.
pool_bb
pool_bb_layer<FeatureLayer url:"https://services7.arcgis.com/JEwYeAy2cc8qOe3o/arcgis/rest/services/pools_redlands/FeatureServer/0">
export = learn.export_training_data(input_raster=naiplayer,
output_location=samplefolder,
input_class_data=pool_bb_layer,
chip_format="PNG",
tile_size={"x":448,"y":448},
stride_size={"x":224,"y":224},
metadata_format="PASCAL_VOC_rectangles",
classvalue_field = "Id",
buffer_radius = 6,
context={"startIndex": 0, "exportAllTiles": False},
gis = gis)
exportNow let's get into the raster store and look at what has been generated and exported.
samples = list_datastore_content(rasterstore.datapath + '/' + samplefolder + "/images", filter = "*png")
# print out the first five chips/subimages
samples['/rasterStores/RasterDataStore/pool_chips_test/images'][0:5]Submitted. Executing... Start Time: Wednesday, October 23, 2024 9:44:27 AM Running script ListDatastoreContent...
['/rasterStores/LocalRasterStore/pool_chips_yongyao/images/000000000.png', '/rasterStores/LocalRasterStore/pool_chips_yongyao/images/000000001.png', '/rasterStores/LocalRasterStore/pool_chips_yongyao/images/000000002.png', '/rasterStores/LocalRasterStore/pool_chips_yongyao/images/000000003.png', '/rasterStores/LocalRasterStore/pool_chips_yongyao/images/000000004.png']
labels = list_datastore_content(rasterstore.datapath + '/' + samplefolder + "/labels", filter = "*xml")
# print out the labels/bounding boxes for the first five chips
labels['/rasterStores/RasterDataStore/pool_chips_test/labels'][0:5]Start Time: Wednesday, October 23, 2024 9:44:29 AM Running script ListDatastoreContent... Completed script ListDatastoreContent... Succeeded at Wednesday, October 23, 2024 9:44:29 AM (Elapsed Time: 0.05 seconds)
['/rasterStores/LocalRasterStore/pool_chips_yongyao/labels/000000000.xml', '/rasterStores/LocalRasterStore/pool_chips_yongyao/labels/000000001.xml', '/rasterStores/LocalRasterStore/pool_chips_yongyao/labels/000000002.xml', '/rasterStores/LocalRasterStore/pool_chips_yongyao/labels/000000003.xml', '/rasterStores/LocalRasterStore/pool_chips_yongyao/labels/000000004.xml']
We can also create a image layer using one of this images and look at what it looks like. Note that a chip may or may not have a bounding box in it and one chip might have multiple boxes as well.
Part 2 - model training
If you've already done part 1, you should already have both the training chips and swimming pool labels. Please change the path to your own export training data folder that contains "images" and "labels" folder.
import os
from pathlib import Path
agol_gis = GIS('home')training_data = agol_gis.content.get('73a29df69b344ce8b94fdb4c9df7103d')
training_data
Image Collection by api_data_owner
filepath = training_data.download(file_name=training_data.name)import zipfile
with zipfile.ZipFile(filepath, 'r') as zip_ref:
zip_ref.extractall(Path(filepath).parent)data_path = Path(os.path.join(os.path.splitext(filepath)[0]))data = prepare_data(data_path, {0:'Pool'}, batch_size=32)
data.classes['background', 'Pool']
Visualize training data
To get a sense of what the training data looks like, arcgis.learn.show_batch() method randomly picks a few training chips and visualize them.
%%time
data.show_batch()CPU times: total: 36.1 s Wall time: 2.31 s
Load model architecture
Here we use Single Shot MultiBox Detector (SSD) [1], a well-recognized object detection algorithm, for swimming pool detection. A SSD model architecture using Resnet-34 as the base model has already been predefined in ArcGIS API for Python, which makes it easy to use.
Architecture of a convolutional neural network with a SSD detector [1]
ssd = SingleShotDetector(data, grids=[5], zooms=[1.0], ratios=[[1.0, 1.0]])Train a model through learning rate tuning and transfer learning
Learning rate is one of the most important hyperparameters in model training. Here we explore a range of learning rate to guide us to choose the best one.
ssd.lr_find()
0.001737800828749376
Based on the learning rate plot above, we can see that the loss starts going down from 1e-4. Therefore, we set learning rate to be a range from 1e-4 to 3e-3, which means we will apply smaller rates to the first few layers and larger rates for the last few layers, and intermediate rates for middle layers, which is the idea of transfer learning. Let's start with 20 epochs for the sake of time.
ssd.fit(20, lr=slice(1e-3, 3e-2))| epoch | train_loss | valid_loss | average_precision | time |
|---|---|---|---|---|
| 0 | 122.097435 | 110.688850 | 0.031559 | 01:43 |
| 1 | 97.619499 | 96.159203 | 0.077635 | 01:43 |
| 2 | 86.206436 | 101.102463 | 0.025981 | 01:42 |
| 3 | 77.945244 | 88.561409 | 0.094550 | 01:41 |
| 4 | 74.508583 | 82.104263 | 0.113272 | 01:44 |
| 5 | 63.735157 | 70.450706 | 0.153443 | 01:47 |
| 6 | 60.626919 | 74.208374 | 0.136342 | 01:45 |
| 7 | 58.279434 | 70.050827 | 0.140875 | 01:47 |
| 8 | 59.066772 | 75.767746 | 0.178750 | 01:43 |
| 9 | 57.460339 | 63.838734 | 0.149501 | 01:43 |
| 10 | 56.071575 | 62.930477 | 0.183645 | 01:44 |
| 11 | 56.584370 | 63.439034 | 0.106067 | 01:42 |
| 12 | 55.039093 | 63.658760 | 0.251810 | 01:43 |
| 13 | 51.613686 | 59.696857 | 0.238393 | 01:43 |
| 14 | 51.887936 | 59.459240 | 0.363338 | 01:43 |
| 15 | 50.366940 | 58.402035 | 0.349388 | 01:42 |
| 16 | 48.102745 | 57.059322 | 0.362674 | 01:42 |
| 17 | 46.950359 | 56.467030 | 0.414897 | 01:43 |
| 18 | 47.563656 | 57.227020 | 0.463681 | 01:43 |
| 19 | 45.963917 | 56.420990 | 0.467014 | 01:43 |
Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_0 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_1 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_3 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_4 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_5 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_7 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_9 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_10 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_13 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_14 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_15 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_16 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_17 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_19 Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\checkpoint_2024-10-22_11-35-07_epoch_19
Detect and visualize swimming pools in validation set
Now we have the model, let's look at how the model performs. Here we plot out 5 rows of images and a threshold of 0.3. Threshold is a measure of probablity that a swimming pool exists. Higher value meas more confidence.
ssd.show_results(thresh=0.3)
As we can see, with only 10 epochs, we are already seeing reasonable results. Further improvment can be acheived through more sophisticated hyperparameter tuning. Let's save the model for further training or inference later. The model should be saved into a models folder in your folder. By default, it will be saved into your data_path that you specified in the very beginning of this notebook.
ssd.save('ssd_5x5-10-deploy_2024', publish=True, gis=gis)Computing model metrics...
Created model files at ~\AppData\Local\Temp\detecting_swimming_pools_using_satellite_image_and_deep_learning\models\ssd_5x5-10-deploy_2024 Published DLPK Item Id: 0d7bdc4558c743039cf9e9d2a2209a2b
WindowsPath('~/AppData/Local/Temp/detecting_swimming_pools_using_satellite_image_and_deep_learning/models/ssd_5x5-10-deploy_2024')Part 3 - deployment and inference
detect_objects_model_package = gis.content.get('0d7bdc4558c743039cf9e9d2a2209a2b') # Replace the item ID from the above cell output to get your published model
detect_objects_model_package
Deep Learning Package by demos_deldev
Now we are ready to install the mode. Installation of the deep learning model item will unpack the model definition file, model file and the inference function script, and copy them to "trusted" location under the Raster Analytic Image Server site's system directory.
detect_objects_model = Model(detect_objects_model_package)detect_objects_model.query_info(gis=gis)Start Time: Tuesday, October 22, 2024 6:40:08 AM Succeeded at Tuesday, October 22, 2024 6:40:09 AM (Elapsed Time: 0.46 seconds) QueryDeepLearningModelInfo GP Job: c7021827fdd34465b2dcd351d4fe14a3 finished successfully.
{'modelInfo': {'Framework': 'arcgis.learn.models._inferencing',
'ModelType': 'ObjectDetection',
'ParameterInfo': [{'name': 'raster',
'dataType': 'raster',
'required': '1',
'displayName': 'Raster',
'description': 'Input Raster'},
{'name': 'model',
'dataType': 'string',
'required': '1',
'displayName': 'Input Model Definition (EMD) File',
'description': 'Input model definition (EMD) JSON file'},
{'name': 'device',
'dataType': 'numeric',
'required': '0',
'displayName': 'Device ID',
'description': 'Device ID'},
{'name': 'padding',
'dataType': 'numeric',
'value': '56',
'required': '0',
'displayName': 'Padding',
'description': 'Padding'},
{'name': 'threshold',
'dataType': 'numeric',
'value': '0.5',
'required': '0',
'displayName': 'Confidence Score Threshold [0.0, 1.0]',
'description': 'Confidence score threshold value [0.0, 1.0]'},
{'name': 'nms_overlap',
'dataType': 'numeric',
'value': '0.1',
'required': '0',
'displayName': 'NMS Overlap',
'description': 'Maximum allowed overlap within each chip'},
{'name': 'batch_size',
'dataType': 'numeric',
'required': '0',
'value': '64',
'displayName': 'Batch Size',
'description': 'Batch Size'},
{'name': 'exclude_pad_detections',
'dataType': 'string',
'required': '0',
'domain': ['True', 'False'],
'value': 'True',
'displayName': 'Filter Outer Padding Detections',
'description': 'Filter detections which are outside the specified padding'},
{'name': 'test_time_augmentation',
'dataType': 'string',
'required': '0',
'value': 'False',
'displayName': 'Perform test time augmentation while predicting',
'description': 'If True, will merge predictions from flipped and rotated images.'},
{'name': 'tta_scales',
'dataType': 'string',
'required': '0',
'value': '1',
'displayName': 'Perform test time augmentation while predicting using different scales',
'description': 'provide different scales separated by comma e.g. 0.9,1,1.1'}]}}Model inference
To test our model, let's get a raster image with some swimming pools.
naip_item.url'https://tiledimageservices7.arcgis.com/JEwYeAy2cc8qOe3o/arcgis/rest/services/naip_subset/ImageServer'
out_objects = detect_objects(input_raster=naip_item.layers[0],
model = detect_objects_model,
output_name = "ssd3_pooldetection_full_redlands_2024" + str(dt.now().microsecond),
context = {
"processorType" : "GPU", # The specified processor (CPU or GPU) will be used for the analysis.
"cellsize" : 0.42
}
)
out_objectsSubmitted. Executing... Start Time: Tuesday, October 22, 2024 9:43:54 AM Raster Analytics helper service: https://rasteranalysis6.arcgis.com/arcgis Running on ArcGIS Online. Read Data Store info from Registry. Publishing Privilege Check: OK DetectObjectsUsingDeepLearning GP Job: 8115f40ebec04474b7bcd1fea31545bc finished successfully.
<FeatureLayer url:"https://services6.arcgis.com/SMX5BErCXLM7eDtY/arcgis/rest/services/ssd3_pooldetection_full_redlands_202457278/FeatureServer/0">
Visualize detected pools on map
result_map = gis.map('Redlands, CA')
result_map.basemap.basemap='satellite'
result_map.content.add(out_objects)
result_map.zoom_to_layer(out_objects)
result_map.legend.enabled = True
result_map
Visualize attribute details of detected features
out_objects_df = out_objects.layers[0].query().sdf
out_objects_df| OID | Class | Confidence | SHAPE | |
|---|---|---|---|---|
| 0 | 1 | Pool | 88.909531 | {"rings": [[[-13052737.9628268, 4035468.901826... |
| 1 | 2 | Pool | 62.038022 | {"rings": [[[-13052736.7628268, 4035376.501826... |
| 2 | 3 | Pool | 61.712742 | {"rings": [[[-13052844.1628268, 4035625.501826... |
| 3 | 4 | Pool | 50.787354 | {"rings": [[[-13052952.7628268, 4035262.501826... |
| 4 | 5 | Pool | 50.531522 | {"rings": [[[-13053239.5628268, 4035490.501826... |
| 5 | 6 | Pool | 95.414031 | {"rings": [[[-13052735.5628268, 4035468.901826... |
| 6 | 7 | Pool | 70.555383 | {"rings": [[[-13052598.7628268, 4035341.101826... |
| 7 | 8 | Pool | 70.151544 | {"rings": [[[-13052542.9628268, 4035343.501826... |
| 8 | 9 | Pool | 69.603866 | {"rings": [[[-13052738.5628268, 4035369.901826... |
| 9 | 10 | Pool | 86.146212 | {"rings": [[[-13053114.1628268, 4035063.301826... |
| 10 | 11 | Pool | 84.137297 | {"rings": [[[-13052722.3628268, 4034880.301826... |
| 11 | 12 | Pool | 83.056879 | {"rings": [[[-13053245.5628268, 4034780.101826... |
| 12 | 13 | Pool | 80.139256 | {"rings": [[[-13052955.6622652, 4035257.931770... |
| 13 | 14 | Pool | 76.52266 | {"rings": [[[-13052940.1628268, 4035149.701826... |
| 14 | 15 | Pool | 72.765577 | {"rings": [[[-13052848.9628268, 4034790.301826... |
| 15 | 16 | Pool | 72.39576 | {"rings": [[[-13052799.7628268, 4035146.701826... |
| 16 | 17 | Pool | 71.398526 | {"rings": [[[-13052814.7628268, 4034966.101826... |
| 17 | 18 | Pool | 70.468491 | {"rings": [[[-13053174.7628268, 4035151.501826... |
| 18 | 19 | Pool | 68.482238 | {"rings": [[[-13052939.5628268, 4034858.101826... |
| 19 | 20 | Pool | 67.470229 | {"rings": [[[-13053062.5628268, 4034861.101826... |
| 20 | 21 | Pool | 67.176467 | {"rings": [[[-13052820.7628268, 4034865.301826... |
| 21 | 22 | Pool | 66.37097 | {"rings": [[[-13053144.1628268, 4034896.501826... |
| 22 | 23 | Pool | 65.359616 | {"rings": [[[-13052889.1628268, 4035153.901826... |
| 23 | 24 | Pool | 64.357823 | {"rings": [[[-13053141.1628268, 4034789.101826... |
| 24 | 25 | Pool | 63.244838 | {"rings": [[[-13053136.3628268, 4035061.501826... |
| 25 | 26 | Pool | 62.825352 | {"rings": [[[-13053142.3628268, 4034973.301826... |
| 26 | 27 | Pool | 61.125118 | {"rings": [[[-13052941.9628268, 4034792.101826... |
| 27 | 28 | Pool | 56.52678 | {"rings": [[[-13053040.9628268, 4034978.101826... |
| 28 | 29 | Pool | 54.094642 | {"rings": [[[-13053042.1628268, 4034904.301826... |
| 29 | 30 | Pool | 53.528374 | {"rings": [[[-13053174.7628268, 4035256.501826... |
| 30 | 31 | Pool | 50.961924 | {"rings": [[[-13052950.9628268, 4034932.501826... |
| 31 | 32 | Pool | 50.857979 | {"rings": [[[-13052829.7628268, 4034856.301826... |
| 32 | 33 | Pool | 50.729121 | {"rings": [[[-13053142.3628268, 4034984.701826... |
| 33 | 34 | Pool | 50.710213 | {"rings": [[[-13053114.7628268, 4034848.501826... |
| 34 | 35 | Pool | 50.623687 | {"rings": [[[-13053063.1628268, 4034876.701826... |
| 35 | 36 | Pool | 50.522973 | {"rings": [[[-13052958.7628268, 4034924.101826... |
| 36 | 37 | Pool | 71.883517 | {"rings": [[[-13052207.5628268, 4034845.501826... |
| 37 | 38 | Pool | 71.174616 | {"rings": [[[-13052589.352817, 4034948.3504839... |
| 38 | 39 | Pool | 64.985728 | {"rings": [[[-13052727.1628268, 4034978.101826... |
| 39 | 40 | Pool | 50.787622 | {"rings": [[[-13052725.9628268, 4034884.501826... |
| 40 | 41 | Pool | 50.718477 | {"rings": [[[-13052728.9628268, 4035107.701826... |
| 41 | 42 | Pool | 73.754281 | {"rings": [[[-13052212.9628268, 4034847.901826... |
| 42 | 43 | Pool | 66.257858 | {"rings": [[[-13052241.7629, 4034772.90182675]... |
Conclusion
In thise notebook, we have covered a lot of ground. In part 1, we discussed how to export training data for deep learning using ArcGIS python API and what the output looks like. In part 2, we demonstrated how to prepare the input data, train a object detection model, visualize the results, as well as apply the model to an unseen image using the deep learning module in ArcGIS API for Python. Then we covered how to install and publish this model and make it production-ready in part 3.
References
[1] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu: “SSD: Single Shot MultiBox Detector”, 2015; arXiv:1512.02325.