ArcGIS Developers
Dashboard

ArcGIS API for Python

Download the samples Try it live

Land Parcel Extraction using Edge Detection model

  • 🔬 Data Science
  • 🥠 Deep Learning and edge detection

Introduction

High-resolution remote sensing images provide useful spatial information for plot delineation; however, manual processing is time-consuming. Automatically extracting visible cadastral boundaries combined with (legal) adjudication and incorporation of local knowledge from human operators offers the potential to improve current cadastral mapping approaches in terms of time, cost, accuracy, and acceptance.

This sample shows how ArcGIS API for Python can be used to train a deep learning edge detection model to extract parcels from satellite imagery and thus more efficient approaches for cadastral mapping.

In this workflow we will basically have three steps.

  • Export training data
  • Train a model
  • Deploy model and extract land parcels

Necessary imports

In [1]:
import os, zipfile
from pathlib import Path

import arcgis
from arcgis import GIS
from arcgis.learn import BDCNEdgeDetector, HEDEdgeDetector, prepare_data

Connect to your GIS

In [2]:
gis = GIS("home")
ent_gis = GIS('https://pythonapi.playground.esri.com/portal')

Export land parcel boundaries data

Training data can be exported by using the Export Training Data For Deep Learning tool available in ArcGIS Pro as well as ArcGIS Image Server. The training data consisted of polyline buffered feature class with a 'class' attribute. For this example, we prepared training data in Classified Tiles format using a chip_size of 400px and cell_size of 20cm in ArcGIS Pro.

  • Input Raster : Esri World Imagery
  • Input Feature Class or Classified Raster : buffered polyline with class attribute
  • Class Value Field : field in the attributes containing class
  • Tile Size X & Tile Size Y : 400
  • Stride X & Stride Y : 128
  • Reference System : Map space
  • Meta Data Format : Classified tiles
  • Environments : Set optimum Cell Size, Processing Extent

Raster and parcels data used for exporting the training dataset are provided below

In [3]:
training_area_raster = ent_gis.content.get('8202ffe4fcaf4ba9bfefe2154f98e7b8')
training_area_raster
Out[3]:
train_area_edgedetection_tif
Training area raster for exporting training dataImagery Layer by api_data_owner
Last Modified: January 15, 2021
0 comments, 1 views
In [4]:
parcel_training_polygon = gis.content.get('ac0e639d6e9b43328605683efb37ff56')
parcel_training_polygon
Out[4]:
Parcels training data edge detection model
land parcels feature class data for edge detection modelFeature Layer Collection by api_data_owner
Last Modified: December 17, 2020
0 comments, 88 views

arcpy.ia.ExportTrainingDataForDeepLearning("Imagery", r"C:\sample\Data\Training Data 400px 20cm", "land_parcels_training_buffered.shp", "TIFF", 400, 400, 128, 128, "ONLY_TILES_WITH_FEATURES", "Classified_Tiles", 0, None, 0, 0, "MAP_SPACE", "NO_BLACKEN", "Fixed_Size")

This will create all the necessary files needed for the next step in the 'Output Folder', and we will now call it our training data.

Prepare data

Alternatively, we have provided a subset of training data containing a samples below and the parcel training polygon used for exporting data. You can use the data directly to run the experiments.

In [32]:
training_data = gis.content.get('ab003694c99f4484a2d30df53f8b4d03')
training_data
Out[32]:
data_for_edge_detection_model
Data for edge detection modelImage Collection by api_data_owner
Last Modified: December 11, 2020
0 comments, 0 views
In [ ]:
filepath = training_data.download(file_name=training_data.name)
In [ ]:
import zipfile
with zipfile.ZipFile(filepath, 'r') as zip_ref:
    zip_ref.extractall(Path(filepath).parent)
In [ ]:
data_path = Path(os.path.join(os.path.splitext(filepath)[0]))

We would specify the path to our training data and a few parameters.

  • path: path of folder containing training data.
  • chip_size: Same as per specified while exporting training data
  • batch_size: No of images your model will train on each step inside an epoch, it directly depends on the memory of your graphic card.
In [11]:
data = prepare_data(data_path, batch_size=2)

Visualize a few samples from your training data

The code below shows a few samples of our data with the same symbology as in ArcGIS Pro.

  • rows: No of rows we want to see the results for.
  • alpha: controls the opacity of labels(Classified imagery) over the drone imagery
In [10]:
data.show_batch(alpha=1)

Part 1 - Model training

Load BDCN or HED edge detector model architecture

There are two available edge detection models in arcgis.learn, the HEDEdgeDetector and BDCNEdgeDetector. If backbone is not specified, by default this model will be loaded on a pretrained ResNet type backbone.

In [12]:
#model = HEDEdgeDetector(data, backbone="vgg19")

#or

model = BDCNEdgeDetector(data, backbone="vgg19")

List of supported backbones, that could be used during training.

In [13]:
model.supported_backbones
Out[13]:
['resnet18',
 'resnet34',
 'resnet50',
 'resnet101',
 'resnet152',
 'vgg11',
 'vgg11_bn',
 'vgg13',
 'vgg13_bn',
 'vgg16',
 'vgg16_bn',
 'vgg19',
 'vgg19_bn']

Tuning for optimal learning rate

Optimization in deep learning is all about tuning 'hyperparameters'. In this step, we will find an 'optimum learning rate' for our model on the training data. Learning rate is a very important parameter, while training our model it will see the training data several times and adjust itself (the weights of the network). Too high learning rate will lead to the convergence of our model to a suboptimal solution and too low learning can slow down the convergence of our model. We can use the lr_find() method to find an optimum learning rate at which can train a robust model fast enough.

In [17]:
# Find Learning Rate
lr = model.lr_find()
In [16]:
lr
Out[16]:
0.007585775750291836

Fit the model on the data

To start with let us first train the model for 30 epochs. One epoch means the model will see the complete training set once and so on. If you feel the results are not satisfactory we can train it further for more number of epochs.

In [18]:
model.fit(epochs=30, lr=lr)
epoch train_loss valid_loss accuracy f1_score dice time
0 42379.250000 42792.605469 0.635095 0.221039 1.000000 04:13
1 41170.578125 40688.066406 0.702877 0.339028 1.000000 04:03
2 42477.429688 39820.054688 0.622671 0.364611 1.000000 04:05
3 39409.554688 37903.304688 0.663775 0.425070 1.000000 04:01
4 38365.433594 37327.031250 0.704567 0.461380 1.000000 04:02
5 38277.031250 36422.820312 0.774611 0.519772 1.000000 04:04
6 36631.902344 35825.132812 0.733576 0.512209 1.000000 04:08
7 36634.308594 35847.406250 0.692677 0.503312 1.000000 04:06
8 37351.023438 35534.578125 0.689683 0.510992 1.000000 04:02
9 36165.355469 34997.855469 0.738407 0.500707 1.000000 04:02
10 33797.582031 33773.210938 0.770278 0.565218 1.000000 04:01
11 37077.937500 34007.226562 0.729761 0.544928 1.000000 04:05
12 33959.007812 32739.144531 0.767353 0.585421 1.000000 04:08
13 34267.722656 32678.582031 0.769639 0.577734 1.000000 04:09
14 34710.437500 31665.923828 0.782900 0.638368 1.000000 04:09
15 33882.789062 31908.816406 0.788911 0.613527 1.000000 04:07
16 33524.593750 31781.998047 0.815680 0.647756 1.000000 04:10
17 33885.957031 31086.320312 0.781484 0.635030 1.000000 04:08
18 33582.882812 31189.753906 0.795834 0.646655 1.000000 04:08
19 34172.046875 31026.679688 0.781517 0.631822 1.000000 04:07
20 32889.683594 30470.085938 0.793456 0.655204 1.000000 04:05
21 34317.046875 30454.828125 0.788285 0.650511 1.000000 04:07
22 34042.183594 30179.656250 0.795486 0.658711 1.000000 04:06
23 33061.484375 29962.724609 0.796448 0.663686 1.000000 04:11
24 31341.308594 29764.175781 0.799347 0.675103 1.000000 04:04
25 30202.384766 29713.958984 0.804643 0.675200 1.000000 03:59
26 31560.056641 29654.199219 0.801009 0.673363 1.000000 04:01
27 31049.183594 29626.998047 0.804344 0.673791 1.000000 04:02
28 32527.287109 29617.658203 0.800732 0.672378 1.000000 04:02
29 30877.294922 29602.923828 0.800952 0.672801 1.000000 04:01

Plot losses

In [19]:
model.plot_losses()

Visualize results

The code below will pick a few random samples and show us ground truth and respective model predictions side by side. This allows us to preview the results of your model in the notebook itself, once satisfied we can save the model and use it further in our workflow.

We have few parameters for visualization.

  • alpha: controls the opacity of predicted edges. Set to 1.
  • thinning: Its a post-processsing parameters, which thins or skeletonizes the predicted edges. We will be using our own pre-processing workflow build in ArcGIS pro hence, will set it to False. As per results, this could also be set to True/False during inferencing in pro.
In [20]:
model.show_results(alpha=1,thinning=False)