Skip To Content ArcGIS for Developers Sign In Dashboard

ArcGIS API for Python

Download the samples Try it live

Detecting settlements using supervised classification and deep learning

  • 🔬 Data Science
  • 🥠 Deep Learning and pixel-based classification

Introduction

Deep-Learning methods tend to perform well with high amounts of data as compared to machine learning methods which is one of the drawbacks of these models. These methods have also been used in geospatial domain to detect objects [1,2] and land use classification [3] which showed favourable results, but labelled input satellite data has always been an effortful task. In that regards, in this notebook we have attempted to use the supervised classification approach to generate the required volumes of data which after cleaning was used to come through the requirement of larger training data for Deep Learning model.

For this, we have considered detecting settlements for Saharanpur district in Uttar Pradesh, India. Settlements have their own importance to urban planners and monitoring them temporally can lay the foundation to design urban policies for any government.

Necessary imports

In [1]:
%matplotlib inline

import pandas as pd
from datetime import datetime
import matplotlib.pyplot as plt

import arcgis
from arcgis.gis import GIS
from arcgis.geocoding import geocode
from arcgis.learn import prepare_data, UnetClassifier
from arcgis.raster.functions import clip, apply, extract_band, colormap, mask, stretch
from arcgis.raster.analytics import train_classifier, classify, list_datastore_content

Connect to your GIS

In [5]:
gis = GIS(url='https://pythonapi.playground.esri.com/portal', username='arcgis_python', password='amazing_arcgis_123')

Get the data for analysis

Search for Multispectral Landsat layer in ArcGIS Online. We can search for content shared by users outside our organization by setting outside_org to True.

In [6]:
landsat_item = gis.content.search('title:Multispectral Landsat', 'Imagery Layer', outside_org=True)[0]
landsat = landsat_item.layers[0]
landsat_item
Out[6]:
Multispectral Landsat
Landsat 8 OLI, 30m multispectral and multitemporal 8-band imagery, with on-the-fly renderings and indices. This imagery layer is sourced from the Landsat on AWS collections and is updated daily with new imagery.Imagery Layer by esri_livingatlas
Last Modified: April 24, 2019
0 comments, 2 views

Search for India State Boundaries 2018 layer in ArcGIS Online. This layer has all the District boundaries for India at index - 2.

In [7]:
boundaries = gis.content.search('India District Boundaries 2018', 'Feature Layer', outside_org=True)[2]
boundaries
Out[7]:
India District Boundaries 2018
This layer shows the District level boundaries of India in 2018. The boundaries are optimized to improve Data Enrichment analysis performance.Feature Layer Collection by esri_livingatlas
Last Modified: April 24, 2019
0 comments, 0 views
In [8]:
district_boundaries = boundaries.layers[2]
district_boundaries
Out[8]:
<FeatureLayer url:"https://pythonapi.playground.esri.com/portal/sharing/servers/12fc8ff238e7420ea43906a9a33a9aab/rest/services/IND_Boundaries_2018/FeatureServer/2">
In [1]:
m = gis.map('Saharanpur, India')
m.add_layer(district_boundaries)
m.legend = True
Out[1]:

As this notebook is to detect settlements for Saharanpur district, you can filter the boundary for Saharanpur.

In [16]:
area = geocode("Saharanpur, India", out_sr=landsat.properties.spatialReference)[0]
landsat.extent = area['extent']

We want to detect settlements for Saharanpur district, so we will apply a query to the boundary layer, by setting "OBJECTID = 09132". The code below brings imagery and feature layer on the same extent.

In [9]:
saharanpur = district_boundaries.query(where='ID=09132')   # query for Saharanur district boundary
saharanpur_geom = saharanpur.features[0].geometry          # Extracting geometry of Saharanpur district boundary   
saharanpur_geom['spatialReference'] = {'wkid':4326}        # Set the Spatial Reference 
saharanpur.features[0].extent = area['extent']             # Set the extent

Get the training points for training the classifier. These are 212 points in total marked against 5 different classes namely Urban, Forest, Agricultural, Water and Wasteland. These points are marked using ArcGIS pro and pulished on the gis server.

In [10]:
data = gis.content.search('classificationPointsSaharanpur', 'Feature Layer')[0]
data
Out[10]:
classificationPointsSaharanpur
Feature Layer Collection by api_data_owner
Last Modified: January 21, 2020
0 comments, 3 views

Filter satellite Imagery based on cloud cover and time duration

In order to produce good results, it is important to select cloud free imagery from the image collection for a specified time duration. In this example we have selected all the images captured between 1 October 2018 to 31 December 2018 with cloud cover less than or equal to 5% for Saharanpur region.

In [23]:
selected = landsat.filter_by(where="(Category = 1) AND (cloudcover <=0.05)",
                             time=[datetime(2018, 10, 1), 
                                   datetime(2018, 12, 31)],
                             geometry=arcgis.geometry.filters.intersects(area))
df = selected.query(out_fields="AcquisitionDate, GroupName, CloudCover, DayOfYear", 
                    order_by_fields="AcquisitionDate").sdf
df['AcquisitionDate'] = pd.to_datetime(df['AcquisitionDate'], unit='ms')
df
Out[23]:
OBJECTID AcquisitionDate GroupName CloudCover DayOfYear Shape_Length Shape_Area SHAPE
0 2360748 2018-10-02 05:18:12 LC81460392018275LGN00_MTL 0.0411 275 863211.750925 4.652535e+10 {'rings': [[[8842440.179299999, 3627905.796800...
1 2360749 2018-10-02 05:18:36 LC81460402018275LGN00_MTL 0.0007 275 851821.808928 4.530870e+10 {'rings': [[[8799651.220800001, 3442773.486500...
2 2365229 2018-10-09 05:24:26 LC81470392018282LGN00_MTL 0.0235 282 864249.005456 4.664003e+10 {'rings': [[[8670522.289099999, 3627673.343800...
3 2365230 2018-10-09 05:24:50 LC81470402018282LGN00_MTL 0.0104 282 852038.709661 4.533114e+10 {'rings': [[[8627077.993299998, 3442759.749600...
4 2363765 2018-10-25 05:24:31 LC81470392018298LGN00_MTL 0.0034 298 864003.480054 4.661440e+10 {'rings': [[[8668674.6948, 3627854.217100002],...
5 2363766 2018-10-25 05:24:55 LC81470402018298LGN00_MTL 0.0084 298 852052.926372 4.533265e+10 {'rings': [[[8625255.243900001, 3442931.826300...
6 2374779 2018-10-27 05:12:34 LC81450402018300LGN00_MTL 0.0003 300 851500.296316 4.527314e+10 {'rings': [[[8969606.0125, 3442791.7321000025]...
7 2382362 2018-11-10 05:24:34 LC81470392018314LGN00_MTL 0.0009 314 863994.276519 4.661226e+10 {'rings': [[[8669263.2619, 3627728.654100001],...
8 2383968 2018-11-12 05:12:36 LC81450402018316LGN00_MTL 0.0001 316 851603.254073 4.528346e+10 {'rings': [[[8969738.007399999, 3442925.248800...
9 2391317 2018-11-26 05:24:34 LC81470392018330LGN00_MTL 0.0031 330 863991.415360 4.661195e+10 {'rings': [[[8669961.7306, 3627710.4842000008]...
10 2391319 2018-11-26 05:24:58 LC81470402018330LGN00_MTL 0.0106 330 851938.390626 4.532084e+10 {'rings': [[[8626659.587900002, 3442921.529799...
11 2392531 2018-12-05 05:18:45 LC81460402018339LGN00_MTL 0.0005 339 851973.199621 4.532425e+10 {'rings': [[[8799698.1228, 3442772.1602], [874...
12 2397733 2018-12-14 05:12:32 LC81450402018348LGN00_MTL 0.0100 348 851383.857661 4.526085e+10 {'rings': [[[8972363.073399998, 3442845.388999...
13 2401123 2018-12-21 05:18:44 LC81460402018355LGN00_MTL 0.0222 355 851916.741276 4.531834e+10 {'rings': [[[8800469.5092, 3442744.7710999995]...
14 2401177 2018-12-28 05:24:31 LC81470392018362LGN00_MTL 0.0019 362 863720.224906 4.658332e+10 {'rings': [[[8671754.256099999, 3627529.248599...
15 2403690 2018-12-30 05:12:33 LC81450402018364LGN00_MTL 0.0086 364 851598.768218 4.528268e+10 {'rings': [[[8972212.594099998, 3442761.256499...

We can now select the image dated 02 October 2018 from the collection using its OBJECTID which is "2360748"

In [24]:
saharanpur_image = landsat.filter_by('OBJECTID=2360748') # 2018-10-02 
In [27]:
m = gis.map('Saharanpur, India')
m.add_layer(apply(saharanpur_image, 'Natural Color with DRA'))
m.add_layer(saharanpur)
m.legend = True
m
Out[27]: