Spatial and temporal distribution of service calls using big data tools¶

Introduction
Necessary Imports
Connect to your ArcGIS Enterprise organization
Ensure your GIS supports GeoAnalytics
Prepare the data

Create a big data file share
Edit a big data file share

Get data for analysis

Search for big data file shares
Search for feature layers

Describe data
Extract features within New Orleans boundary
Summarize data
Analyze patterns
Find statistically significant hot and cold spots
Visualize other aspects of data
Conclusion

Introduction¶

The arcgis.geoanalytics module provides types and functions for distributed analysis of large datasets. These GeoAnalytics tools work with big data registered in the GIS datastores as well as with feature layers. In this notebook, we will go through the steps for setting up data to create a big data file share. We will also edit big data file share manifest to set spatial reference of the dataset. Once the data gets registered, we will demonstrate the utility of a number of tools including describe_dataset, aggregate_points, calculate_density, find_hot_spots, clip_layer, and run_python_script in order to better understand our data.

The sample aims to find answers to some fundamental questions:

What is the spatial relationship between 911 calls?
Which block groups have the highest number of 911 calls reporting?
What is the most common reason for 911 calls?
How many 911 calls occur each month?
How many 911 calls occur each hour?

The data that will be used in this sample was originally obtained from data.gov open data portal. You can obtain data by searching using keywords, for example: 'Calls for Service New Orleans'. This sample demonstrates ability of ArcGIS API for Python to perform big data analysis on your infrastructure.

Note:

The ability to perform big data analysis is only available on ArcGIS Enterprise licensed with a GeoAnalytics server and not yet available on ArcGIS Online.

Necessary Imports¶

In [1]:

%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
from datetime import datetime as dt

import arcgis
import arcgis.geoanalytics
from arcgis.gis import GIS

from arcgis.geoanalytics.summarize_data import aggregate_points, describe_dataset
from arcgis.geoanalytics.analyze_patterns import calculate_density, find_hot_spots
from arcgis.geoanalytics.manage_data import clip_layer, run_python_script

Connect to your ArcGIS Enterprise organization¶

In [2]:

gis = GIS('https://pythonapi.playground.esri.com/portal', 'arcgis_python', 'amazing_arcgis_123')

Ensure your GIS supports GeoAnalytics¶

After connecting to Enterprise portal, we need to ensure an ArcGIS Enterprise GIS is set up with a licensed GeoAnalytics server. To do so, we will call the is_supported() method.

In [3]:

arcgis.geoanalytics.is_supported()

Out[3]:

True

Prepare the data¶

To register a file share or an HDFS, we need to format datasets as subfolders within a single parent folder and register the parent folder. This parent folder becomes a datastore, and each subfolder becomes a dataset. Our folder hierarchy would look like below:

Learn more about preparing your big data file share datasets here

The get_datastores() method of the geoanalytics module returns a DatastoreManager object that lets you search for and manage the big data file share items as Python API Datastore objects on your GeoAnalytics server.

In [4]:

bigdata_datastore_manager = arcgis.geoanalytics.get_datastores()
bigdata_datastore_manager

Out[4]:

<DatastoreManager for https://pythonapi.playground.esri.com/ga/admin>

We will register service calls data as a big data file share using the add_bigdata() function on a DatastoreManager object.

When we register a directory, all subdirectories under the specified folder are also registered with the server. Always register the parent folder (for example, \machinename\mydatashare) that contains one or more individual dataset folders as the big data file share item. To learn more, see register a big data file share

Note: You cannot browse directories in ArcGIS Server Manager. You must provide the full path to the folder you want to register, for example, \myserver\share\bigdata. Avoid using local paths, such as C:\bigdata, unless the same data folder is available on all nodes of the server site.

In [5]:

# data_item = bigdata_datastore_manager.add_bigdata("ServiceCallsOrleans", r"\\machinename\datastore")

Created Big Data file share for ServiceCallsOrleans

In [5]:

bigdata_fileshares = bigdata_datastore_manager.search(id='cff51a1a-4f27-4955-a3ef-5fa23240ccf9')
bigdata_fileshares

Out[5]:

[<Datastore title:"/bigDataFileShares/ServiceCallsOrleans" type:"bigDataFileShare">]

In [6]:

file_share_folder = bigdata_fileshares[0]

Once a big data file share is created, the GeoAnalytics server samples the datasets to generate a manifest, which outlines the data schema and specifies any time and geometry fields. A query of the resulting manifest returns each dataset's schema.. This process can take a few minutes depending on the size of your data. Once processed, querying the manifest property returns the schema of the datasets in your big data file share.

In [7]:

manifest = file_share_folder.manifest
manifest

Out[7]:

{'datasets': [{'name': 'yearly_calls',
   'format': {'quoteChar': '"',
    'fieldDelimiter': ',',
    'hasHeaderRow': True,
    'encoding': 'UTF-8',
    'escapeChar': '"',
    'recordTerminator': '\n',
    'type': 'delimited',
    'extension': 'csv'},
   'schema': {'fields': [{'name': 'NOPD_Item', 'type': 'esriFieldTypeString'},
     {'name': 'Type_', 'type': 'esriFieldTypeString'},
     {'name': 'TypeText', 'type': 'esriFieldTypeString'},
     {'name': 'Priority', 'type': 'esriFieldTypeString'},
     {'name': 'MapX', 'type': 'esriFieldTypeDouble'},
     {'name': 'MapY', 'type': 'esriFieldTypeDouble'},
     {'name': 'TimeCreate', 'type': 'esriFieldTypeString'},
     {'name': 'TimeDispatch', 'type': 'esriFieldTypeString'},
     {'name': 'TimeArrive', 'type': 'esriFieldTypeString'},
     {'name': 'TimeClosed', 'type': 'esriFieldTypeString'},
     {'name': 'Disposition', 'type': 'esriFieldTypeString'},
     {'name': 'DispositionText', 'type': 'esriFieldTypeString'},
     {'name': 'BLOCK_ADDRESS', 'type': 'esriFieldTypeString'},
     {'name': 'Zip', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'PoliceDistrict', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'Location', 'type': 'esriFieldTypeString'}]},
   'geometry': {'geometryType': 'esriGeometryPoint',
    'spatialReference': {'wkid': 102682, 'latestWkid': 3452},
    'fields': [{'name': 'MapX', 'formats': ['x']},
     {'name': 'MapY', 'formats': ['y']}]},
   'time': {'timeType': 'instant',
    'timeReference': {'timeZone': 'UTC'},
    'fields': [{'name': 'TimeCreate',
      'formats': ['MM/dd/yyyy hh:mm:ss a']}]}}]}

The spatial reference of the dataset is set to 4326, but we know this data is from New Orleans, Louisiana, and is actually stored in the Louisiana State Plane Coordinate System. We need to edit the manifest with the correct spatial reference: {"wkid": 102682, "latestWkid": 3452}. Knowing the location where this data belongs to and the coordinate system which contains geospatial information of this dataset, we will edit our manifest. This will set the correct spatial reference.

In [8]:

manifest['datasets'][0]['geometry']['spatialReference'] = { "wkid": 102682, "latestWkid": 3452 }

In [9]:

file_share_folder.manifest = manifest

In [10]:

file_share_folder.manifest

Out[10]:

{'datasets': [{'name': 'yearly_calls',
   'format': {'quoteChar': '"',
    'fieldDelimiter': ',',
    'hasHeaderRow': True,
    'encoding': 'UTF-8',
    'escapeChar': '"',
    'recordTerminator': '\n',
    'type': 'delimited',
    'extension': 'csv'},
   'schema': {'fields': [{'name': 'NOPD_Item', 'type': 'esriFieldTypeString'},
     {'name': 'Type_', 'type': 'esriFieldTypeString'},
     {'name': 'TypeText', 'type': 'esriFieldTypeString'},
     {'name': 'Priority', 'type': 'esriFieldTypeString'},
     {'name': 'MapX', 'type': 'esriFieldTypeDouble'},
     {'name': 'MapY', 'type': 'esriFieldTypeDouble'},
     {'name': 'TimeCreate', 'type': 'esriFieldTypeString'},
     {'name': 'TimeDispatch', 'type': 'esriFieldTypeString'},
     {'name': 'TimeArrive', 'type': 'esriFieldTypeString'},
     {'name': 'TimeClosed', 'type': 'esriFieldTypeString'},
     {'name': 'Disposition', 'type': 'esriFieldTypeString'},
     {'name': 'DispositionText', 'type': 'esriFieldTypeString'},
     {'name': 'BLOCK_ADDRESS', 'type': 'esriFieldTypeString'},
     {'name': 'Zip', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'PoliceDistrict', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'Location', 'type': 'esriFieldTypeString'}]},
   'geometry': {'geometryType': 'esriGeometryPoint',
    'spatialReference': {'wkid': 102682, 'latestWkid': 3452},
    'fields': [{'name': 'MapX', 'formats': ['x']},
     {'name': 'MapY', 'formats': ['y']}]},
   'time': {'timeType': 'instant',
    'timeReference': {'timeZone': 'UTC'},
    'fields': [{'name': 'TimeCreate',
      'formats': ['MM/dd/yyyy hh:mm:ss a']}]}}]}

Get data for analysis¶

Search for big data file shares¶

Adding a big data file share to the Geoanalytics server adds a corresponding big data file share item on the portal. We can search for these types of items using the item_type parameter.

In [11]:

search_result = gis.content.search("bigDataFileShares_ServiceCallsOrleans", item_type = "big data file share", max_items=40)
search_result

Out[11]:

[<Item title:"bigDataFileShares_ServiceCallsOrleans" type:Big Data File Share owner:portaladmin>]

In [12]:

data_item = search_result[0]

In [13]:

data_item

Out[13]:

bigDataFileShares_ServiceCallsOrleans

Big Data File Share by portaladmin
Last Modified: October 05, 2019
0 comments, 0 views

Querying the layers property of the item returns a featureLayer representing the data. The object is actually an API Layer object.

In [14]:

data_item.layers

Out[14]:

[<Layer url:"https://pythonapi.playground.esri.com/ga/rest/services/DataStoreCatalogs/bigDataFileShares_ServiceCallsOrleans/BigDataCatalogServer/yearly_calls">]

In [15]:

calls = data_item.layers[0]

In [16]:

calls.properties

Out[16]:

{
  "dataStoreID": "cff51a1a-4f27-4955-a3ef-5fa23240ccf9",
  "fields": [
    {
      "name": "NOPD_Item",
      "type": "esriFieldTypeString"
    },
    {
      "name": "Type_",
      "type": "esriFieldTypeString"
    },
    {
      "name": "TypeText",
      "type": "esriFieldTypeString"
    },
    {
      "name": "Priority",
      "type": "esriFieldTypeString"
    },
    {
      "name": "MapX",
      "type": "esriFieldTypeDouble"
    },
    {
      "name": "MapY",
      "type": "esriFieldTypeDouble"
    },
    {
      "name": "TimeCreate",
      "type": "esriFieldTypeString"
    },
    {
      "name": "TimeDispatch",
      "type": "esriFieldTypeString"
    },
    {
      "name": "TimeArrive",
      "type": "esriFieldTypeString"
    },
    {
      "name": "TimeClosed",
      "type": "esriFieldTypeString"
    },
    {
      "name": "Disposition",
      "type": "esriFieldTypeString"
    },
    {
      "name": "DispositionText",
      "type": "esriFieldTypeString"
    },
    {
      "name": "BLOCK_ADDRESS",
      "type": "esriFieldTypeString"
    },
    {
      "name": "Zip",
      "type": "esriFieldTypeInteger"
    },
    {
      "name": "PoliceDistrict",
      "type": "esriFieldTypeInteger"
    },
    {
      "name": "Location",
      "type": "esriFieldTypeString"
    }
  ],
  "name": "yearly_calls",
  "geometryType": "esriGeometryPoint",
  "type": "featureClass",
  "spatialReference": {
    "wkid": 102682,
    "latestWkid": 3452
  },
  "geometry": {
    "fields": [
      {
        "name": "MapX",
        "formats": [
          "x"
        ]
      },
      {
        "name": "MapY",
        "formats": [
          "y"
        ]
      }
    ]
  },
  "time": {
    "timeType": "instant",
    "timeReference": {
      "timeZone": "UTC"
    },
    "fields": [
      {
        "name": "TimeCreate",
        "formats": [
          "MM/dd/yyyy hh:mm:ss a"
        ]
      }
    ]
  },
  "currentVersion": 10.81,
  "children": []
}

Search for feature layers¶

In [17]:

block_grp_item = gis.content.get('9975b4dd3ca24d4bbe6177b85f9da7bb')

In [18]:

block_grp_item

Out[18]:

New_Orleans_Block_Groups_2019
New_Orleans_Block_Groups_2019

Feature Layer Collection by api_data_owner
Last Modified: October 05, 2019
0 comments, 17 views

We will use the first item for our analysis. Since the item is a Feature Layer Collection, accessing the layers property will give us a list of Feature layer objects.

In [19]:

blk_grp_lyr = block_grp_item.layers[0]

Describe data¶

The describe_dataset method provides an overview of big data. By default, the tool outputs a table layer containing calculated field statistics and a dict outlining geometry and time settings for the input layer.

Optionally, the tool can output a feature layer representing a sample set of features using the sample_size parameter, or a single polygon feature layer representing the input feature layers' extent by setting the extent_output parameter to True.

In [20]:

description = describe_dataset(input_layer=calls,
                               extent_output=True,
                               sample_size=1000,
                               output_name="Description of service calls" + str(dt.now().microsecond),
                               return_tuple=True)

{"messageCode":"BD_101051","message":"Possible issues were found while reading 'inputLayer'.","params":{"paramName":"inputLayer"}}
{"messageCode":"BD_101052","message":"Some records have either missing or invalid time values."}
{"messageCode":"BD_101054","message":"Some records have either missing or invalid geometries."}

In [21]:

description.output_json

Out[21]:

{'datasetName': 'calls',
 'datasetSource': 'Big Data File Share - ServiceCallsOrleans',
 'recordCount': 3952898,
 'geometry': {'geometryType': 'Point',
  'sref': {'wkid': 102682, 'latestWkid': 3452},
  'countNonEmpty': 1763656,
  'countEmpty': 2189242,
  'spatialExtent': {'xmin': 0, 'ymin': 0, 'xmax': 37369000, 'ymax': 3513814}},
 'time': {'timeType': 'Instant',
  'countNonEmpty': 1763656,
  'countEmpty': 2189242,
  'temporalExtent': {'start': '2011-01-01 00:00:02.000',
   'end': '2019-08-03 23:59:09.000'}}}

In [22]:

description.sample_layer

Out[22]:

<FeatureLayer url:"https://deldevd014.esri.com/server/rest/services/Hosted/Description_of_service_calls223782/FeatureServer/2">

In [23]:

description.sample_layer.query().sdf

Out[23]:

	NOPD_Item	Type_	TypeText	Priority	MapX	MapY	TimeCreate	TimeDispatch	TimeArrive	TimeClosed	Disposition	DispositionText	BLOCK_ADDRESS	Zip	PoliceDistrict	Location	INSTANT_DATETIME	globalid	OBJECTID	SHAPE
0	A0247212	21	COMPLAINT OTHER	1H	37369000.0	3513814.0	01/02/2012 05:16:04 PM	01/02/2012 05:16:05 PM	01/02/2012 05:16:05 PM	01/02/2012 05:20:40 PM	NAT	NECESSARY ACTION TAKEN	052XX N Claiborne Ave	NaN	0.0	(9.235500543E-7, -0.0000022929847665)	2012-01-02 17:16:04	{A471E188-9E41-2AA9-C46D-00771D78C616}	12	{'x': 1.8008574168558748e-05, 'y': 3.003918417...
1	L3180018	62A	BURGLAR ALARM, SILENT	2E	NaN	NaN	2E	3706346	518681	12/25/2018 12:38:48 AM	12/25/2018 12:42:11 AM	12/25/2018 12:47:42 AM	12/25/2018 12:58:28 AM	NaN	NaN	N	NaT	{8E68BDAF-E483-6952-ED6F-220A07FAEC2D}	76	None
2	H0789414	21	COMPLAINT OTHER	1H	21.0	NaN	1H	3671292	536599	08/06/2014 07:52:05 PM	None	08/06/2014 07:52:05 PM	08/06/2014 08:04:40 PM	NaN	NaN	Y	NaT	{003975B6-5491-DDB0-A0D8-6158F8EA62E0}	133	None
3	E3777812	62A	BURGLAR ALARM, SILEN	2C	3668023.0	517302.0	05/24/2012 12:47:03 PM	05/24/2012 12:48:48 PM	05/24/2012 12:51:46 PM	05/24/2012 12:55:18 PM	NAT	NECESSARY ACTION TAKEN	005XX Dufossat St	70115.0	2.0	(29.916803057765, -90.11110486281)	2012-05-24 12:47:03	{5ABCF635-D42E-92D4-3E1F-88F8853DCEC6}	119	{'x': -90.11111048458989, 'y': 29.916808462433...
4	I0820718	62A	BURGLAR ALARM, SILENT	1A	NaN	NaN	2E	3704330	554832	09/07/2018 05:52:32 PM	09/08/2018 12:52:59 AM	None	09/08/2018 01:33:43 AM	NaN	NaN	N	NaT	{BC31F407-7C92-ADA5-5FFB-2B61E04604C2}	114	None
5	C1587515	21	COMPLAINT OTHER	1H	21.0	NaN	1H	3670550	539301	03/13/2015 09:42:20 PM	None	03/13/2015 09:42:20 PM	03/13/2015 09:53:24 PM	NaN	NaN	Y	NaT	{93EC1557-7D0A-539A-47A6-0EE7CEC9B645}	126	None
6	E0773819	107	SUSPICIOUS PERSON	2C	3669625.0	532902.0	05/06/2019 09:07:17 AM	05/06/2019 09:08:03 AM	05/06/2019 09:10:27 AM	05/06/2019 09:20:14 AM	GOA	GONE ON ARRIVAL	107	NaN	NaN	N	2019-05-06 09:07:17	{7BDE1FC8-733C-F08E-2624-A274A5E9AC61}	172	{'x': -90.10552626485547, 'y': 29.959656351679...
7	J3326816	21N	NOISE COMPLAINT	1E	NaN	NaN	1E	3693027	535071	10/30/2016 06:00:21 AM	10/30/2016 07:24:38 AM	10/30/2016 07:33:03 AM	10/30/2016 07:34:32 AM	NaN	NaN	N	NaT	{98B78BC8-4A35-0A0F-93E0-085E6490355C}	278	None
8	L4245411	62A	BURGLAR ALARM, SILEN	2C	3693800.0	525393.0	12/29/2011 04:47:02 PM	12/29/2011 04:48:31 PM	12/29/2011 04:56:36 PM	12/29/2011 05:26:41 PM	NAT	NECESSARY ACTION TAKEN	027XX Lawrence St	70114.0	4.0	(29.938269364881, -90.029451676433)	2011-12-29 16:47:02	{EC51B15E-98FD-B6A1-9905-412D09B336A6}	187	{'x': -90.02945727694797, 'y': 29.938274776870...
9	H1929011	103	DISTURBANCE (OTHER)	1C	3679384.0	545506.0	08/13/2011 07:13:55 AM	08/13/2011 07:13:57 AM	08/13/2011 07:14:02 AM	08/13/2011 07:17:31 AM	NAT	NECESSARY ACTION TAKEN	015XX Sere St	70122.0	3.0	(29.994016435986, -90.074266619976)	2011-08-13 07:13:55	{976D05C0-6D29-2618-439B-678E9EB46D37}	146	{'x': -90.0742722350555, 'y': 29.9940218586582...
10	H0834814	21	COMPLAINT OTHER	1H	21.0	NaN	1H	3672670	540915	08/07/2014 03:43:05 AM	08/07/2014 03:43:05 AM	08/07/2014 04:13:10 AM	08/07/2014 04:23:35 AM	NaN	NaN	N	NaT	{EBFC2284-8D11-B162-6801-CFB03ED6A7A9}	206	None
11	C2682218	103	DISTURBANCE (OTHER)	1A	103.0	NaN	1C	3687693	534738	03/22/2018 09:17:24 AM	03/22/2018 09:31:34 AM	03/22/2018 09:31:36 AM	03/22/2018 10:10:04 AM	NaN	NaN	N	NaT	{0A9ED09A-EA2F-F7DB-9756-10173D9E76D7}	305	None
12	H1150317	17M	MUNICIPAL ATTACHMENT	1F	17.0	NaN	1G	3670173	526285	08/09/2017 12:02:13 PM	None	08/09/2017 12:02:13 PM	08/09/2017 12:02:50 PM	NaN	NaN	Y	NaT	{ABED0474-C002-EE73-82AC-F48AE11AF6CF}	311	None
13	J3459717	21	COMPLAINT OTHER	1H	67.0	NaN	1E	3675387	534931	10/27/2017 07:26:01 PM	10/28/2017 08:36:33 AM	10/28/2017 08:36:49 AM	10/28/2017 08:49:46 AM	NaN	NaN	N	NaT	{7197EBC3-81CC-03A3-6BAA-E3868DA10AB6}	254	None
14	E0116112	18	TRAFFIC INCIDENT	1H	3697253.0	551601.0	05/01/2012 05:41:25 PM	05/01/2012 05:41:25 PM	05/01/2012 05:41:33 PM	05/01/2012 06:09:18 PM	NAT	NECESSARY ACTION TAKEN	Chef Menteur Hwy & Stemway Dr	70126.0	7.0	(30.010223436348, -90.017600200365)	2012-05-01 17:41:25	{0B02A1FF-4259-F0E5-ADEB-F46F44BBA987}	324	{'x': -90.01760580074226, 'y': 30.010228864391...
15	I0860111	62A	BURGLAR ALARM, SILEN	2C	3680347.0	524357.0	09/07/2011 03:14:10 AM	09/07/2011 03:16:26 AM	09/07/2011 03:22:28 AM	09/07/2011 03:27:41 AM	NAT	NECESSARY ACTION TAKEN	014XX Magazine St	70130.0	6.0	(29.935834785662, -90.07196006635)	2011-09-07 03:14:10	{0213BAA0-A677-4003-E125-D7BE1ADB47FA}	317	{'x': -90.07196567831078, 'y': 29.935840195744...
16	B0297112	21	COMPLAINT OTHER	2H	3692716.0	540389.0	02/02/2012 08:09:37 PM	02/02/2012 08:43:07 PM	None	02/02/2012 08:59:22 PM	NAT	NECESSARY ACTION TAKEN	024XX Alvar St	70117.0	5.0	(29.979536989582, -90.032336408255)	2012-02-02 20:09:37	{3EB84BEC-4A5C-337A-98FB-FE7982255EF0}	230	{'x': -90.03234201131997, 'y': 29.979542410466...
17	I2182114	107	SUSPICIOUS PERSON	2A	107.0	NaN	2A	3681704	531287	09/17/2014 04:07:07 PM	None	09/17/2014 04:07:07 PM	09/17/2014 04:31:57 PM	NaN	NaN	Y	NaT	{5A1B7046-0DD3-F55B-81CB-90B9D6E333D4}	258	None
18	I3863213	18	TRAFFIC INCIDENT	1H	3698925.0	557251.0	09/29/2013 04:01:05 AM	09/29/2013 04:01:06 AM	09/29/2013 04:01:05 AM	09/29/2013 05:05:33 AM	NAT	Necessary Action Taken	I-10 W (7D04) & Morrison Rd Exit	70126.0	7.0	(30.025705895352, -90.012111936158)	2013-09-29 04:01:05	{E6F85E86-8803-7E4E-C2A9-5DD81C2950EA}	525	{'x': -90.01211753570936, 'y': 30.025711326943...
19	B0139512	21	COMPLAINT OTHER	1B	3680623.0	525652.0	02/01/2012 08:53:09 PM	02/01/2012 09:13:18 PM	None	02/01/2012 09:32:21 PM	NAT	NECESSARY ACTION TAKEN	011XX Erato St	70130.0	6.0	(29.939387249037, -90.071043691518)	2012-02-01 20:53:09	{089C1D47-D66A-040D-9164-9AE1C7DF1AFC}	431	{'x': -90.07104930338194, 'y': 29.939392659922...
20	F2337812	65	SIMPLE ROBBERY	1B	3679940.0	529508.0	06/15/2012 11:20:15 PM	06/15/2012 11:24:43 PM	06/15/2012 11:41:09 PM	06/16/2012 02:03:22 AM	RTF	REPORT TO FOLLOW	005XX Baronne St	70113.0	8.0	(29.950010633457, -90.073066114888)	2012-06-15 23:20:15	{0AD4CBB9-E41A-8B1B-F89A-72387C6D4EDD}	586	{'x': -90.0730717277556, 'y': 29.9500160465899...
21	H1111512	21	COMPLAINT OTHER	1H	3667387.0	534128.0	08/08/2012 12:24:12 AM	08/08/2012 12:27:55 AM	08/08/2012 12:28:55 AM	08/08/2012 12:35:46 AM	NAT	NECESSARY ACTION TAKEN	034XX S Carrollton Ave	70125.0	2.0	(29.963087783259, -90.11254651347)	2012-08-08 00:24:12	{CA58ACFE-54ED-B2F2-2454-3FFB9CA69437}	563	{'x': -90.11255213762003, 'y': 29.963093197957...
22	G1868016	59	CRIMINAL MISCHIEF	1F	59.0	NaN	1F	3667711	522588	07/17/2016 04:45:33 PM	None	None	07/17/2016 06:10:49 PM	NaN	NaN	N	NaT	{B0995EEC-FE0D-6A05-92D8-DA61A3EC781F}	518	None
23	D3460315	20	AUTO ACCIDENT	1A	20.0	NaN	1A	3679962	526663	04/27/2015 06:25:27 PM	None	None	04/27/2015 07:40:24 PM	NaN	NaN	N	NaT	{299E2EF4-E60B-2C56-919F-CB7DD65CFF2D}	410	None
24	H4376912	62A	BURGLAR ALARM, SILEN	1C	3681788.0	532676.0	08/28/2012 06:48:22 PM	08/28/2012 07:19:29 PM	None	08/28/2012 07:20:30 PM	NAT	NECESSARY ACTION TAKEN	006XX Dauphine St	70112.0	8.0	(29.958665542329, -90.067120585039)	2012-08-28 18:48:22	{E1E36AC7-8FDD-0F08-23A6-C8D03FD7ECCE}	613	{'x': -90.0671261966614, 'y': 29.9586709575393...
25	A0451215	18	TRAFFIC INCIDENT	1H	18.0	NaN	1H	3697519	532902	01/05/2015 12:07:00 AM	01/05/2015 12:07:00 AM	01/05/2015 12:07:19 AM	01/05/2015 12:17:43 AM	NaN	NaN	N	NaT	{6532C606-9BF6-097C-6620-CFDBC1B8FE20}	514	None
26	E2080514	21	COMPLAINT OTHER	1H	21.0	NaN	1H	3699489	555988	05/15/2014 08:40:26 PM	05/15/2014 09:34:03 PM	None	05/15/2014 09:36:27 PM	NaN	NaN	N	NaT	{48BBAFC2-6F72-F619-2C1A-005EC1643B9F}	641	None
27	K1291714	18	TRAFFIC INCIDENT	1H	18.0	NaN	1H	3674226	527339	11/11/2014 02:54:57 AM	None	11/11/2014 02:54:57 AM	11/11/2014 03:29:14 AM	NaN	NaN	Y	NaT	{E1FF75DD-BCF4-C9F0-DFC8-F98E886F986D}	425	None
28	J0065712	21	COMPLAINT OTHER	1H	3681752.0	531382.0	10/01/2012 12:44:06 PM	10/01/2012 12:44:27 PM	10/01/2012 12:44:59 PM	10/01/2012 12:52:25 PM	DUP	DUPLICATE	003XX Royal St	70130.0	8.0	(29.955108575882, -90.067279411608)	2012-10-01 12:44:06	{77FCFBCA-49D8-A4B6-D8A5-822149E70D35}	660	{'x': -90.06728502312177, 'y': 29.955113990312...
29	E3786812	18	TRAFFIC INCIDENT	1H	3683352.0	538374.0	05/24/2012 01:54:53 PM	05/24/2012 01:54:53 PM	05/24/2012 02:05:54 PM	05/24/2012 02:24:43 PM	NAT	NECESSARY ACTION TAKEN	018XX N Derbigny St	70116.0	5.0	(29.97428546205, -90.061982249162)	2012-05-24 13:54:53	{C9476A29-8114-D2BD-6FA3-9491B1FD736C}	580	{'x': -90.0619878600577, 'y': 29.9742908808282...
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
970	L1621117	22A	AREA CHECK	1I	NaN	NaN	1I	3671435	553142	12/14/2017 09:14:14 PM	12/14/2017 09:14:14 PM	12/14/2017 09:14:19 PM	12/14/2017 09:38:10 PM	NaN	NaN	N	NaT	{C3A20D49-E44C-9717-CFA9-F2A5D47BCF9A}	179	None
971	H0552516	21	COMPLAINT OTHER	1H	21.0	NaN	1H	3692874	551918	08/05/2016 06:39:41 PM	None	08/05/2016 06:39:41 PM	08/05/2016 10:47:31 PM	NaN	NaN	Y	NaT	{5ECEE3CD-1712-9070-51DC-10790C55AE4A}	196	None
972	E0962512	17F	FUGITIVE ATTTACHMENT	1G	3684673.0	539982.0	05/06/2012 10:04:53 PM	05/06/2012 10:04:53 PM	None	05/06/2012 10:05:08 PM	RTF	REPORT TO FOLLOW	Elysian Fields Ave & N Galvez St	70117.0	5.0	(29.978666531583, -90.057753693797)	2012-05-06 22:04:53	{545A64C8-4872-3396-B99C-C524D5F56577}	301	{'x': -90.0577593037315, 'y': 29.9786719514519...
973	A3871913	18	TRAFFIC INCIDENT	1H	3701559.0	516958.0	01/29/2013 01:26:56 PM	01/29/2013 01:26:56 PM	01/29/2013 01:26:58 PM	01/29/2013 01:38:15 PM	NAT	NECESSARY ACTION TAKEN	General De Gaulle Dr & Huntlee D	70131.0	4.0	(29.914830976353, -90.005264597449)	2013-01-29 13:26:56	{EAD23CF8-07A5-AFB9-0643-8B8738F71A51}	271	{'x': -90.00527019039149, 'y': 29.914836384016...
974	I4080513	100	HIT & RUN	1E	3660531.0	526542.0	09/30/2013 11:18:52 PM	10/01/2013 02:04:05 AM	10/01/2013 02:12:05 AM	10/01/2013 02:16:20 AM	GOA	GONE ON ARRIVAL	005XX S Carrollton Ave	70118.0	2.0	(29.942427784731, -90.134447707047)	2013-09-30 23:18:52	{0A9D4CC7-04E4-8CF7-C8B9-434F1C0E740A}	272	{'x': -90.134453336259, 'y': 29.94243319422235...
975	H3203611	62A	BURGLAR ALARM, SILEN	2C	3672373.0	518941.0	08/21/2011 08:38:09 AM	08/21/2011 08:40:55 AM	08/21/2011 08:42:33 AM	08/21/2011 08:45:13 AM	NAT	NECESSARY ACTION TAKEN	040XX Magazine St	70115.0	6.0	(29.921181484798, -90.097318619706)	2011-08-21 08:38:09	{BA45AE5B-03C8-E746-BE09-27E515EF1CE4}	211	{'x': -90.0973242379294, 'y': 29.9211868908671...
976	J1301516	103D	DOMESTIC DISTURBANCE	2B	35.0	NaN	2B	3660930	531266	10/12/2016 04:28:05 PM	10/12/2016 04:28:41 PM	10/12/2016 04:37:15 PM	10/12/2016 06:26:56 PM	NaN	NaN	N	NaT	{1E51063B-B2A2-73E3-4BDB-222AC9C1DA04}	309	None
977	L3766312	62A	BURGLAR ALARM, SILEN	2C	3701991.0	518152.0	12/27/2012 04:11:56 PM	12/27/2012 04:15:28 PM	None	12/27/2012 04:18:05 PM	VOI	VOID	035XX Lang St	70131.0	4.0	(29.918100292854, -90.003857335737)	2012-12-27 16:11:56	{A0F02388-43F4-D44F-ECCE-28FC8EF2139E}	297	{'x': -90.00386292843632, 'y': 29.918105701276...
978	E0685114	18	TRAFFIC INCIDENT	1H	18.0	NaN	1H	3696210	551411	05/05/2014 11:30:29 PM	05/05/2014 11:31:39 PM	05/05/2014 11:30:29 PM	05/05/2014 11:37:51 PM	NaN	NaN	Y	NaT	{BC5C4182-CC3A-50A2-7B39-BFB88A1E9D4E}	372	None
979	B1491111	107	SUSPICIOUS PERSON	2B	3678129.0	533076.0	02/10/2011 12:45:04 PM	None	None	02/10/2011 12:50:15 PM	DUP	DUPLICATE	018XX Canal St	70112.0	1.0	(29.95987607933, -90.078660929549)	2011-02-10 12:45:04	{5A405527-7B44-78B0-3052-F3D52EB9C407}	489	{'x': -90.07866654435806, 'y': 29.959881494429...
980	I3344518	18	TRAFFIC INCIDENT	1J	21.0	NaN	1J	3698542	519147	09/27/2018 01:24:55 AM	09/27/2018 01:24:55 AM	09/27/2018 01:24:55 AM	09/27/2018 02:03:41 AM	NaN	NaN	Y	NaT	{AF7F4840-037C-876F-557E-44ABDF216CAB}	257	None
981	E1886814	103	DISTURBANCE (OTHER)	2A	103.0	NaN	2A	3672223	525508	05/14/2014 03:23:26 PM	05/14/2014 03:24:05 PM	05/14/2014 03:34:00 PM	05/14/2014 03:40:32 PM	NaN	NaN	N	NaT	{E585712D-C815-3258-5BB4-50FA9B3D93ED}	347	None
982	D1096317	62A	BURGLAR ALARM, SILENT	1A	NaN	NaN	2C	3703440	560261	04/09/2017 09:05:17 PM	04/10/2017 03:43:41 AM	None	04/10/2017 04:10:19 AM	NaN	NaN	N	NaT	{212DA271-B29E-FF3F-18EF-6B2E453BFBB1}	381	None
983	J2543015	21	COMPLAINT OTHER	1H	NaN	NaN	0E	3675672	525857	10/21/2015 01:05:16 PM	10/21/2015 01:10:01 PM	None	10/21/2015 02:04:53 PM	NaN	NaN	N	NaT	{DC45D2C4-34FC-2213-9429-AC20AF027237}	363	None
984	G1014312	67	THEFT	1E	3676106.0	530718.0	07/07/2012 03:21:21 PM	07/07/2012 03:23:28 PM	07/07/2012 03:23:33 PM	07/07/2012 04:26:56 PM	RTF	REPORT TO FOLLOW	Bertrand St & Lafayette St	70113.0	1.0	(29.953453123323, -90.085130233788)	2012-07-07 15:21:21	{26FFD6F7-F70D-08E2-F5E8-89BC738AD6D8}	416	{'x': -90.08513585007961, 'y': 29.953458536814...
985	K3910518	22A	AREA CHECK	1K	NaN	NaN	1K	3697581	515539	11/29/2018 08:07:55 PM	11/29/2018 08:07:55 PM	11/29/2018 08:07:55 PM	11/29/2018 09:48:13 PM	NaN	NaN	Y	NaT	{8C8963FB-6638-252A-DE32-5281D6C5BD3D}	403	None
986	K1279211	21	COMPLAINT OTHER	1H	37369000.0	3513814.0	11/09/2011 11:32:43 AM	11/09/2011 11:34:39 AM	None	11/09/2011 12:05:35 PM	NAT	NECESSARY ACTION TAKEN	019XX Joann	NaN	0.0	(9.235500543E-7, -0.0000022929847665)	2011-11-09 11:32:43	{77A76953-7458-EE16-CFEF-7E50C18CE3D0}	455	{'x': 1.8008574168558748e-05, 'y': 3.003918417...
987	J2756411	99	RECKLESS DRIVING	2D	3684064.0	547517.0	10/18/2011 10:30:47 PM	10/18/2011 10:43:30 PM	None	10/18/2011 10:47:35 PM	GOA	GONE ON ARRIVAL	Elysian Fields Ave & Gentilly Bl	70122.0	3.0	(29.999403737159, -90.059412555475)	2011-10-18 22:30:47	{41CB848F-2CD5-C099-F1E7-A24961015322}	432	{'x': -90.05941816675012, 'y': 29.999409161486...
988	I1844213	107	SUSPICIOUS PERSON	2A	3679397.0	535828.0	09/14/2013 02:59:38 AM	09/14/2013 03:03:08 AM	09/14/2013 03:09:04 AM	09/14/2013 03:15:08 AM	UNF	UNFOUNDED	009XX N Prieur St	70116.0	1.0	(29.967404904189, -90.074561399733)	2013-09-14 02:59:38	{C61167EA-14E3-6246-2EB2-B964B6316807}	521	{'x': -90.07456701375169, 'y': 29.967410321060...
989	H1058117	NOPD	INCIDENT REQUESTED BY ANOTHER AGENCY	2A	NaN	NaN	2A	3696876	525312	08/08/2017 06:33:18 PM	None	None	08/08/2017 06:33:59 PM	NaN	NaN	N	NaT	{D7847DCD-444D-AF11-8985-479601301C7D}	741	None
990	E2451818	58	RETURN FOR ADDITIONAL INFO	1I	58.0	NaN	1I	3672772	526264	05/19/2018 10:39:22 PM	05/19/2018 10:39:22 PM	05/19/2018 10:47:02 PM	05/19/2018 11:16:59 PM	NaN	NaN	N	NaT	{24F27B84-3F54-B20E-9AFE-61F10AD7B8C8}	543	None
991	E2650811	67	THEFT	1E	3677269.0	547605.0	05/17/2011 06:42:57 PM	05/17/2011 07:50:41 PM	05/17/2011 08:12:39 PM	05/17/2011 10:00:51 PM	RTF	REPORT TO FOLLOW	042XX Buchanan St	70122.0	3.0	(29.999851678873, -90.080875137137)	2011-05-17 18:42:57	{430BC094-3AA8-F5F3-F593-92697871071E}	726	{'x': -90.08088075426286, 'y': 29.999857102599...
992	H2357512	103	DISTURBANCE (OTHER)	1C	3715565.0	564186.0	08/15/2012 04:49:19 PM	None	None	08/15/2012 06:02:50 PM	VOI	VOID	060XX Bullard Ave	70128.0	7.0	(30.044236292957, -89.959269638889)	2012-08-15 16:49:19	{5A3CC2E0-D43B-28ED-8BB4-49E9503A7EF0}	630	{'x': -89.95927522486834, 'y': 30.044241730303...
993	B0811515	21	COMPLAINT OTHER	1H	NaN	NaN	0E	3682795	525927	02/07/2015 04:07:46 PM	02/07/2015 04:16:23 PM	None	02/07/2015 04:22:03 PM	NaN	NaN	N	NaT	{0AB3B844-7776-83EB-04AD-AAAF9CD1C344}	624	None
994	E3981912	21	COMPLAINT OTHER	1H	3682210.0	533199.0	05/25/2012 06:02:33 PM	05/25/2012 06:02:33 PM	None	05/25/2012 06:16:45 PM	NAT	NECESSARY ACTION TAKEN	Dauphine St & Saint Ann St	70116.0	8.0	(29.960090784609, -90.065769750523)	2012-05-25 18:02:33	{10D6A065-EE5B-A699-A0CB-136B7445DCC7}	639	{'x': -90.0657753618394, 'y': 29.9600962001742...
995	A2575219	22A	AREA CHECK	1K	3678685.0	536249.0	01/20/2019 08:44:04 AM	01/20/2019 08:44:04 AM	01/20/2019 08:44:04 AM	01/20/2019 09:46:30 AM	NAT	Necessary Action Taken	22A	NaN	NaN	Y	2019-01-20 08:44:04	{B4CF1985-73CB-398C-495C-EC752D62A62B}	686	{'x': -90.0768009430172, 'y': 29.9685894153133...
996	E2915114	17T	TRAFFIC ATTACHMENT	1F	17.0	NaN	1G	3688424	540633	05/21/2014 08:34:25 PM	05/21/2014 08:34:25 PM	None	05/21/2014 08:34:37 PM	NaN	NaN	N	NaT	{23366D2E-DD2D-9306-F16B-FB396E542F75}	651	None
997	A0811415	62A	BURGLAR ALARM, SILENT	2C	NaN	NaN	2C	3686861	541875	01/07/2015 10:39:22 PM	None	None	01/07/2015 10:43:47 PM	NaN	NaN	N	NaT	{CA91D5A7-B617-960F-140E-70FA0D96A912}	707	None
998	L1133615	62C	SIMPLE BURGLARY VEHICLE	1E	NaN	NaN	1E	3680642	537525	12/10/2015 01:21:07 PM	None	12/10/2015 01:21:07 PM	12/10/2015 01:52:47 PM	NaN	NaN	Y	NaT	{A1399702-B3FD-FB27-7D6F-649CCCD0F0F3}	798	None
999	B0108617	106	OBSCENITY, EXPOSING	2A	106.0	NaN	2A	3681317	530762	02/01/2017 08:14:41 PM	02/01/2017 08:15:47 PM	02/01/2017 08:15:53 PM	02/02/2017 01:03:07 AM	NaN	NaN	N	NaT	{E13C6440-2B7D-A8AA-B7D4-A4923A84B8B5}	764	None

1000 rows Ãƒâ€” 20 columns

We can see some records have missing or invalid attribute values, including in the fields the manifests defines as the time and geometry values. We will visualize a sample layer on the map to understand it better.

In [24]:

m1 = gis.map()
m1

Out[24]:

In [25]:

m1.add_layer(description.sample_layer)

The map shows that some data points have been located outside New Orleans because of missing or invalid geometries. We want to explore data points within New Orleans city limits. We will use clip_layer tool to extract only point features within the New Orleans boundary. This will remove data with missing or invalid geometries.

Extract features within New Orleans boundary¶

The clip_layer method extracts input point, line, or polygon features from an input_layer that fall within the boundaries of features in a clip_layer. The output layer contains a subset of features from the input layer. We will clip our input call feature layer to the New Orleans blk_grp_lyr features.

In [26]:

clip_result = clip_layer(calls, blk_grp_lyr, output_name="service calls in new Orleans" + str(dt.now().microsecond))

{"messageCode":"BD_101051","message":"Possible issues were found while reading 'inputLayer'.","params":{"paramName":"inputLayer"}}
{"messageCode":"BD_101052","message":"Some records have either missing or invalid time values."}
{"messageCode":"BD_101054","message":"Some records have either missing or invalid geometries."}

In [27]:

clip_result

Out[27]:

service_calls_in_new_Orleans630022
service_calls_in_new_Orleans630022

Feature Layer Collection by admin
Last Modified: November 12, 2019
0 comments, 0 views

In [28]:

orleans_calls = clip_result.layers[0]

In [29]:

m2 = gis.map("New Orleans")
m2

Out[29]:

ArcGIS API for Python Sample Notebooks