Skip To Content ArcGIS for Developers Sign In Dashboard

ArcGIS API for Python

Download the samples Try it live

Spatial and temporal distribution of service calls using big data tools

Introduction

The arcgis.geoanalytics module provides types and functions for distributed analysis of large datasets. These GeoAnalytics tools work with big data registered in the GIS datastores as well as with feature layers. In this notebook, we will go through the steps for setting up data to create a big data file share. We will also edit big data file share manifest to set spatial reference of the dataset. Once the data gets registered, we will demonstrate the utility of a number of tools including describe_dataset, aggregate_points, calculate_density, find_hot_spots, clip_layer, and run_python_script in order to better understand our data.

The sample aims to find answers to some fundamental questions:

  • What is the spatial relationship between 911 calls?
  • Which block groups have the highest number of 911 calls reporting?
  • What is the most common reason for 911 calls?
  • How many 911 calls occur each month?
  • How many 911 calls occur each hour?

The data that will be used in this sample was originally obtained from data.gov open data portal. You can obtain data by searching using keywords, for example: 'Calls for Service New Orleans'. This sample demonstrates ability of ArcGIS API for Python to perform big data analysis on your infrastructure.

Note:

The ability to perform big data analysis is only available on ArcGIS Enterprise licensed with a GeoAnalytics server and not yet available on ArcGIS Online.

Necessary Imports

In [1]:
%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
from datetime import datetime as dt

import arcgis
import arcgis.geoanalytics
from arcgis.gis import GIS

from arcgis.geoanalytics.summarize_data import aggregate_points, describe_dataset
from arcgis.geoanalytics.analyze_patterns import calculate_density, find_hot_spots
from arcgis.geoanalytics.manage_data import clip_layer, run_python_script

Connect to your ArcGIS Enterprise organization

In [2]:
gis = GIS(profile="your_enterprise_portal")

Ensure your GIS supports GeoAnalytics

After connecting to Enterprise portal, we need to ensure an ArcGIS Enterprise GIS is set up with a licensed GeoAnalytics server. To do so, we will call the is_supported() method.

In [3]:
arcgis.geoanalytics.is_supported()
Out[3]:
True

Prepare the data

To register a file share or an HDFS, we need to format datasets as subfolders within a single parent folder and register the parent folder. This parent folder becomes a datastore, and each subfolder becomes a dataset. Our folder hierarchy would look like below:

|---FileShareFolder < -- The top-level folder is what is registered as a big data file share |---Calls < -- A folder "calls", composed of sub folders |---2011 < -- A folder "2011", composed of 1 csv |---Calls_for_Service_2011.csv |---2012 |---Calls_for_Service_2012.csv |---2013 |---Calls_for_Service_2013.csv |---2014 |---Calls_for_Service_2014.csv |---2015 |---Calls_for_Service_2015.csv |---2016 |---Calls_for_Service_2016.csv |---2017 |---Calls_for_Service_2017.csv |---2018 |---Calls_for_Service_2018.csv |---2019 |---Calls_for_Service_2019.csv

Learn more about preparing your big data file share datasets here

Create a big data file share

The get_datastores() method of the geoanalytics module returns a DatastoreManager object that lets you search for and manage the big data file share items as Python API Datastore objects on your GeoAnalytics server.

In [4]:
bigdata_datastore_manager = arcgis.geoanalytics.get_datastores()
bigdata_datastore_manager

We will register service calls data as a big data file share using the add_bigdata() function on a DatastoreManager object.

When we register a directory, all subdirectories under the specified folder are also registered with the server. Always register the parent folder (for example, \machinename\mydatashare) that contains one or more individual dataset folders as the big data file share item. To learn more, see register a big data file share

Note: You cannot browse directories in ArcGIS Server Manager. You must provide the full path to the folder you want to register, for example, \myserver\share\bigdata. Avoid using local paths, such as C:\bigdata, unless the same data folder is available on all nodes of the server site.

In [5]:
data_item = bigdata_datastore_manager.add_bigdata("ServiceCallsOrleans", r"\\machinename\datastore")
Created Big Data file share for ServiceCallsOrleans
In [6]:
bigdata_fileshares = bigdata_datastore_manager.search()
bigdata_fileshares
Out[6]:
[<Datastore title:"/bigDataFileShares/ServiceCalls" type:"bigDataFileShare">,
 <Datastore title:"/bigDataFileShares/ServiceCallsOrleans" type:"bigDataFileShare">,
 <Datastore title:"/enterpriseDatabases/AGSDataStore_ds_2m7srouk" type:"egdb">,
 <Datastore title:"/nosqlDatabases/AGSDataStore_bigdata_bds_ekbvqahj" type:"nosql">,
 <Datastore title:"/nosqlDatabases/AGSDataStore_nosqldb_tcs_6fkw3ovx" type:"nosql">]
In [7]:
file_share_folder = bigdata_fileshares[1]

Once a big data file share is created, the GeoAnalytics server samples the datasets to generate a manifest, which outlines the data schema and specifies any time and geometry fields. A query of the resulting manifest returns each dataset's schema.. This process can take a few minutes depending on the size of your data. Once processed, querying the manifest property returns the schema of the datasets in your big data file share.

In [8]:
manifest = file_share_folder.manifest
manifest
Out[8]:
{'datasets': [{'name': 'calls',
   'format': {'quoteChar': '"',
    'fieldDelimiter': ',',
    'hasHeaderRow': True,
    'encoding': 'UTF-8',
    'escapeChar': '"',
    'recordTerminator': '\n',
    'type': 'delimited',
    'extension': 'csv'},
   'schema': {'fields': [{'name': 'NOPD_Item', 'type': 'esriFieldTypeString'},
     {'name': 'Type_', 'type': 'esriFieldTypeString'},
     {'name': 'TypeText', 'type': 'esriFieldTypeString'},
     {'name': 'Priority', 'type': 'esriFieldTypeString'},
     {'name': 'MapX', 'type': 'esriFieldTypeDouble'},
     {'name': 'MapY', 'type': 'esriFieldTypeDouble'},
     {'name': 'TimeCreate', 'type': 'esriFieldTypeString'},
     {'name': 'TimeDispatch', 'type': 'esriFieldTypeString'},
     {'name': 'TimeArrive', 'type': 'esriFieldTypeString'},
     {'name': 'TimeClosed', 'type': 'esriFieldTypeString'},
     {'name': 'Disposition', 'type': 'esriFieldTypeString'},
     {'name': 'DispositionText', 'type': 'esriFieldTypeString'},
     {'name': 'BLOCK_ADDRESS', 'type': 'esriFieldTypeString'},
     {'name': 'Zip', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'PoliceDistrict', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'Location', 'type': 'esriFieldTypeString'}]},
   'geometry': {'geometryType': 'esriGeometryPoint',
    'spatialReference': {'wkid': 4326},
    'fields': [{'name': 'MapX', 'formats': ['x']},
     {'name': 'MapY', 'formats': ['y']}]},
   'time': {'timeType': 'instant',
    'timeReference': {'timeZone': 'UTC'},
    'fields': [{'name': 'TimeCreate',
      'formats': ['MM/dd/yyyy hh:mm:ss a']}]}}]}

Edit a big data file share

The spatial reference of the dataset is set to 4326, but we know this data is from New Orleans, Louisiana, and is actually stored in the Louisiana State Plane Coordinate System. We need to edit the manifest with the correct spatial reference: {"wkid": 102682, "latestWkid": 3452}. Knowing the location where this data belongs to and the coordinate system which contains geospatial information of this dataset, we will edit our manifest. This will set the correct spatial reference.

In [9]:
manifest['datasets'][0]['geometry']['spatialReference'] = { "wkid": 102682, "latestWkid": 3452 }
In [10]:
file_share_folder.manifest = manifest

Get data for analysis

Search for big data file shares

Adding a big data file share to the Geoanalytics server adds a corresponding big data file share item on the portal. We can search for these types of items using the item_type parameter.

In [11]:
search_result = gis.content.search("", item_type = "big data file share", max_items=40)
search_result
Out[11]:
[<Item title:"bigDataFileShares_ServiceCallsOrleans" type:Big Data File Share owner:admin>]
In [12]:
data_item = search_result[0]
In [13]:
data_item
Out[13]:
bigDataFileShares_ServiceCallsOrleans
Big Data File Share by admin
Last Modified: November 11, 2019
0 comments, 0 views

Querying the layers property of the item returns a featureLayer representing the data. The object is actually an API Layer object.

In [14]:
data_item.layers
Out[14]:
[<Layer url:"https://dev0006300.esri.com/gax/rest/services/DataStoreCatalogs/bigDataFileShares_ServiceCallsOrleans/BigDataCatalogServer/calls">]
In [15]:
calls = data_item.layers[0]
In [16]:
calls.properties
Out[16]:
{
  "dataStoreID": "bc99afe0-5859-4a2c-8ecf-688872b9a75f",
  "fields": [
    {
      "name": "NOPD_Item",
      "type": "esriFieldTypeString"
    },
    {
      "name": "Type_",
      "type": "esriFieldTypeString"
    },
    {
      "name": "TypeText",
      "type": "esriFieldTypeString"
    },
    {
      "name": "Priority",
      "type": "esriFieldTypeString"
    },
    {
      "name": "MapX",
      "type": "esriFieldTypeDouble"
    },
    {
      "name": "MapY",
      "type": "esriFieldTypeDouble"
    },
    {
      "name": "TimeCreate",
      "type": "esriFieldTypeString"
    },
    {
      "name": "TimeDispatch",
      "type": "esriFieldTypeString"
    },
    {
      "name": "TimeArrive",
      "type": "esriFieldTypeString"
    },
    {
      "name": "TimeClosed",
      "type": "esriFieldTypeString"
    },
    {
      "name": "Disposition",
      "type": "esriFieldTypeString"
    },
    {
      "name": "DispositionText",
      "type": "esriFieldTypeString"
    },
    {
      "name": "BLOCK_ADDRESS",
      "type": "esriFieldTypeString"
    },
    {
      "name": "Zip",
      "type": "esriFieldTypeInteger"
    },
    {
      "name": "PoliceDistrict",
      "type": "esriFieldTypeInteger"
    },
    {
      "name": "Location",
      "type": "esriFieldTypeString"
    }
  ],
  "name": "calls",
  "geometryType": "esriGeometryPoint",
  "type": "featureClass",
  "spatialReference": {
    "wkid": 102682,
    "latestWkid": 3452
  },
  "geometry": {
    "fields": [
      {
        "name": "MapX",
        "formats": [
          "x"
        ]
      },
      {
        "name": "MapY",
        "formats": [
          "y"
        ]
      }
    ]
  },
  "time": {
    "timeType": "instant",
    "timeReference": {
      "timeZone": "UTC"
    },
    "fields": [
      {
        "name": "TimeCreate",
        "formats": [
          "MM/dd/yyyy hh:mm:ss a"
        ]
      }
    ]
  },
  "currentVersion": 10.8,
  "children": []
}

Search for feature layers

In [17]:
block_grp_item = gis.content.search('New_Orleans_Block_Goups', 'feature layer')[0]
In [18]:
block_grp_item
Out[18]:
New_Orleans_Block_Goups
Feature Layer Collection by admin
Last Modified: November 12, 2019
0 comments, 2 views

We will use the first item for our analysis. Since the item is a Feature Layer Collection, accessing the layers property will give us a list of Feature layer objects.

In [19]:
blk_grp_lyr = block_grp_item.layers[0]

Describe data

The describe_dataset method provides an overview of big data. By default, the tool outputs a table layer containing calculated field statistics and a dict outlining geometry and time settings for the input layer.

Optionally, the tool can output a feature layer representing a sample set of features using the sample_size parameter, or a single polygon feature layer representing the input feature layers' extent by setting the extent_output parameter to True.

In [20]:
description = describe_dataset(input_layer=calls,
                               extent_output=True,
                               sample_size=1000,
                               output_name="Description of service calls" + str(dt.now().microsecond),
                               return_tuple=True)
{"messageCode":"BD_101051","message":"Possible issues were found while reading 'inputLayer'.","params":{"paramName":"inputLayer"}}
{"messageCode":"BD_101052","message":"Some records have either missing or invalid time values."}
{"messageCode":"BD_101054","message":"Some records have either missing or invalid geometries."}
In [21]:
description.output_json
Out[21]:
{'datasetName': 'calls',
 'datasetSource': 'Big Data File Share - ServiceCallsOrleans',
 'recordCount': 3952898,
 'geometry': {'geometryType': 'Point',
  'sref': {'wkid': 102682, 'latestWkid': 3452},
  'countNonEmpty': 1763656,
  'countEmpty': 2189242,
  'spatialExtent': {'xmin': 0, 'ymin': 0, 'xmax': 37369000, 'ymax': 3513814}},
 'time': {'timeType': 'Instant',
  'countNonEmpty': 1763656,
  'countEmpty': 2189242,
  'temporalExtent': {'start': '2011-01-01 00:00:02.000',
   'end': '2019-08-03 23:59:09.000'}}}
In [22]:
description.sample_layer
Out[22]:
<FeatureLayer url:"https://deldevd014.esri.com/server/rest/services/Hosted/Description_of_service_calls223782/FeatureServer/2">
In [23]:
description.sample_layer.query().sdf
Out[23]:
NOPD_Item Type_ TypeText Priority MapX MapY TimeCreate TimeDispatch TimeArrive TimeClosed Disposition DispositionText BLOCK_ADDRESS Zip PoliceDistrict Location INSTANT_DATETIME globalid OBJECTID SHAPE
0 A0247212 21 COMPLAINT OTHER 1H 37369000.0 3513814.0 01/02/2012 05:16:04 PM 01/02/2012 05:16:05 PM 01/02/2012 05:16:05 PM 01/02/2012 05:20:40 PM NAT NECESSARY ACTION TAKEN 052XX N Claiborne Ave NaN 0.0 (9.235500543E-7, -0.0000022929847665) 2012-01-02 17:16:04 {A471E188-9E41-2AA9-C46D-00771D78C616} 12 {'x': 1.8008574168558748e-05, 'y': 3.003918417...
1 L3180018 62A BURGLAR ALARM, SILENT 2E NaN NaN 2E 3706346 518681 12/25/2018 12:38:48 AM 12/25/2018 12:42:11 AM 12/25/2018 12:47:42 AM 12/25/2018 12:58:28 AM NaN NaN N NaT {8E68BDAF-E483-6952-ED6F-220A07FAEC2D} 76 None
2 H0789414 21 COMPLAINT OTHER 1H 21.0 NaN 1H 3671292 536599 08/06/2014 07:52:05 PM None 08/06/2014 07:52:05 PM 08/06/2014 08:04:40 PM NaN NaN Y NaT {003975B6-5491-DDB0-A0D8-6158F8EA62E0} 133 None
3 E3777812 62A BURGLAR ALARM, SILEN 2C 3668023.0 517302.0 05/24/2012 12:47:03 PM 05/24/2012 12:48:48 PM 05/24/2012 12:51:46 PM 05/24/2012 12:55:18 PM NAT NECESSARY ACTION TAKEN 005XX Dufossat St 70115.0 2.0 (29.916803057765, -90.11110486281) 2012-05-24 12:47:03 {5ABCF635-D42E-92D4-3E1F-88F8853DCEC6} 119 {'x': -90.11111048458989, 'y': 29.916808462433...
4 I0820718 62A BURGLAR ALARM, SILENT 1A NaN NaN 2E 3704330 554832 09/07/2018 05:52:32 PM 09/08/2018 12:52:59 AM None 09/08/2018 01:33:43 AM NaN NaN N NaT {BC31F407-7C92-ADA5-5FFB-2B61E04604C2} 114 None
5 C1587515 21 COMPLAINT OTHER 1H 21.0 NaN 1H 3670550 539301 03/13/2015 09:42:20 PM None 03/13/2015 09:42:20 PM 03/13/2015 09:53:24 PM NaN NaN Y NaT {93EC1557-7D0A-539A-47A6-0EE7CEC9B645} 126 None
6 E0773819 107 SUSPICIOUS PERSON 2C 3669625.0 532902.0 05/06/2019 09:07:17 AM 05/06/2019 09:08:03 AM 05/06/2019 09:10:27 AM 05/06/2019 09:20:14 AM GOA GONE ON ARRIVAL 107 NaN NaN N 2019-05-06 09:07:17 {7BDE1FC8-733C-F08E-2624-A274A5E9AC61} 172 {'x': -90.10552626485547, 'y': 29.959656351679...
7 J3326816 21N NOISE COMPLAINT 1E NaN NaN 1E 3693027 535071 10/30/2016 06:00:21 AM 10/30/2016 07:24:38 AM 10/30/2016 07:33:03 AM 10/30/2016 07:34:32 AM NaN NaN N NaT {98B78BC8-4A35-0A0F-93E0-085E6490355C} 278 None
8 L4245411 62A BURGLAR ALARM, SILEN 2C 3693800.0 525393.0 12/29/2011 04:47:02 PM 12/29/2011 04:48:31 PM 12/29/2011 04:56:36 PM 12/29/2011 05:26:41 PM NAT NECESSARY ACTION TAKEN 027XX Lawrence St 70114.0 4.0 (29.938269364881, -90.029451676433) 2011-12-29 16:47:02 {EC51B15E-98FD-B6A1-9905-412D09B336A6} 187 {'x': -90.02945727694797, 'y': 29.938274776870...
9 H1929011 103 DISTURBANCE (OTHER) 1C 3679384.0 545506.0 08/13/2011 07:13:55 AM 08/13/2011 07:13:57 AM 08/13/2011 07:14:02 AM 08/13/2011 07:17:31 AM NAT NECESSARY ACTION TAKEN 015XX Sere St 70122.0 3.0 (29.994016435986, -90.074266619976) 2011-08-13 07:13:55 {976D05C0-6D29-2618-439B-678E9EB46D37} 146 {'x': -90.0742722350555, 'y': 29.9940218586582...
10 H0834814 21 COMPLAINT OTHER 1H 21.0 NaN 1H 3672670 540915 08/07/2014 03:43:05 AM 08/07/2014 03:43:05 AM 08/07/2014 04:13:10 AM 08/07/2014 04:23:35 AM NaN NaN N NaT {EBFC2284-8D11-B162-6801-CFB03ED6A7A9} 206 None
11 C2682218 103 DISTURBANCE (OTHER) 1A 103.0 NaN 1C 3687693 534738 03/22/2018 09:17:24 AM 03/22/2018 09:31:34 AM 03/22/2018 09:31:36 AM 03/22/2018 10:10:04 AM NaN NaN N NaT {0A9ED09A-EA2F-F7DB-9756-10173D9E76D7} 305 None
12 H1150317 17M MUNICIPAL ATTACHMENT 1F 17.0 NaN 1G 3670173 526285 08/09/2017 12:02:13 PM None 08/09/2017 12:02:13 PM 08/09/2017 12:02:50 PM NaN NaN Y NaT {ABED0474-C002-EE73-82AC-F48AE11AF6CF} 311 None
13 J3459717 21 COMPLAINT OTHER 1H 67.0 NaN 1E 3675387 534931 10/27/2017 07:26:01 PM 10/28/2017 08:36:33 AM 10/28/2017 08:36:49 AM 10/28/2017 08:49:46 AM NaN NaN N NaT {7197EBC3-81CC-03A3-6BAA-E3868DA10AB6} 254 None
14 E0116112 18 TRAFFIC INCIDENT 1H 3697253.0 551601.0 05/01/2012 05:41:25 PM 05/01/2012 05:41:25 PM 05/01/2012 05:41:33 PM 05/01/2012 06:09:18 PM NAT NECESSARY ACTION TAKEN Chef Menteur Hwy & Stemway Dr 70126.0 7.0 (30.010223436348, -90.017600200365) 2012-05-01 17:41:25 {0B02A1FF-4259-F0E5-ADEB-F46F44BBA987} 324 {'x': -90.01760580074226, 'y': 30.010228864391...
15 I0860111 62A BURGLAR ALARM, SILEN 2C 3680347.0 524357.0 09/07/2011 03:14:10 AM 09/07/2011 03:16:26 AM 09/07/2011 03:22:28 AM 09/07/2011 03:27:41 AM NAT NECESSARY ACTION TAKEN 014XX Magazine St 70130.0 6.0 (29.935834785662, -90.07196006635) 2011-09-07 03:14:10 {0213BAA0-A677-4003-E125-D7BE1ADB47FA} 317 {'x': -90.07196567831078, 'y': 29.935840195744...
16 B0297112 21 COMPLAINT OTHER 2H 3692716.0 540389.0 02/02/2012 08:09:37 PM 02/02/2012 08:43:07 PM None 02/02/2012 08:59:22 PM NAT NECESSARY ACTION TAKEN 024XX Alvar St 70117.0 5.0 (29.979536989582, -90.032336408255) 2012-02-02 20:09:37 {3EB84BEC-4A5C-337A-98FB-FE7982255EF0} 230 {'x': -90.03234201131997, 'y': 29.979542410466...
17 I2182114 107 SUSPICIOUS PERSON 2A 107.0 NaN 2A 3681704 531287 09/17/2014 04:07:07 PM None 09/17/2014 04:07:07 PM 09/17/2014 04:31:57 PM NaN NaN Y NaT {5A1B7046-0DD3-F55B-81CB-90B9D6E333D4} 258 None
18 I3863213 18 TRAFFIC INCIDENT 1H 3698925.0 557251.0 09/29/2013 04:01:05 AM 09/29/2013 04:01:06 AM 09/29/2013 04:01:05 AM 09/29/2013 05:05:33 AM NAT Necessary Action Taken I-10 W (7D04) & Morrison Rd Exit 70126.0 7.0 (30.025705895352, -90.012111936158) 2013-09-29 04:01:05 {E6F85E86-8803-7E4E-C2A9-5DD81C2950EA} 525 {'x': -90.01211753570936, 'y': 30.025711326943...
19 B0139512 21 COMPLAINT OTHER 1B 3680623.0 525652.0 02/01/2012 08:53:09 PM 02/01/2012 09:13:18 PM None 02/01/2012 09:32:21 PM NAT NECESSARY ACTION TAKEN 011XX Erato St 70130.0 6.0 (29.939387249037, -90.071043691518) 2012-02-01 20:53:09 {089C1D47-D66A-040D-9164-9AE1C7DF1AFC} 431 {'x': -90.07104930338194, 'y': 29.939392659922...
20 F2337812 65 SIMPLE ROBBERY 1B 3679940.0 529508.0 06/15/2012 11:20:15 PM 06/15/2012 11:24:43 PM 06/15/2012 11:41:09 PM 06/16/2012 02:03:22 AM RTF REPORT TO FOLLOW 005XX Baronne St 70113.0 8.0 (29.950010633457, -90.073066114888) 2012-06-15 23:20:15 {0AD4CBB9-E41A-8B1B-F89A-72387C6D4EDD} 586 {'x': -90.0730717277556, 'y': 29.9500160465899...
21 H1111512 21 COMPLAINT OTHER 1H 3667387.0 534128.0 08/08/2012 12:24:12 AM 08/08/2012 12:27:55 AM 08/08/2012 12:28:55 AM 08/08/2012 12:35:46 AM NAT NECESSARY ACTION TAKEN 034XX S Carrollton Ave 70125.0 2.0 (29.963087783259, -90.11254651347) 2012-08-08 00:24:12 {CA58ACFE-54ED-B2F2-2454-3FFB9CA69437} 563 {'x': -90.11255213762003, 'y': 29.963093197957...
22 G1868016 59 CRIMINAL MISCHIEF 1F 59.0 NaN 1F 3667711 522588 07/17/2016 04:45:33 PM None None 07/17/2016 06:10:49 PM NaN NaN N NaT {B0995EEC-FE0D-6A05-92D8-DA61A3EC781F} 518 None
23 D3460315 20 AUTO ACCIDENT 1A 20.0 NaN 1A 3679962 526663 04/27/2015 06:25:27 PM None None 04/27/2015 07:40:24 PM NaN NaN N NaT {299E2EF4-E60B-2C56-919F-CB7DD65CFF2D} 410 None
24 H4376912 62A BURGLAR ALARM, SILEN 1C 3681788.0 532676.0 08/28/2012 06:48:22 PM 08/28/2012 07:19:29 PM None 08/28/2012 07:20:30 PM NAT NECESSARY ACTION TAKEN 006XX Dauphine St 70112.0 8.0 (29.958665542329, -90.067120585039) 2012-08-28 18:48:22 {E1E36AC7-8FDD-0F08-23A6-C8D03FD7ECCE} 613 {'x': -90.0671261966614, 'y': 29.9586709575393...
25 A0451215 18 TRAFFIC INCIDENT 1H 18.0 NaN 1H 3697519 532902 01/05/2015 12:07:00 AM 01/05/2015 12:07:00 AM 01/05/2015 12:07:19 AM 01/05/2015 12:17:43 AM NaN NaN N NaT {6532C606-9BF6-097C-6620-CFDBC1B8FE20} 514 None
26 E2080514 21 COMPLAINT OTHER 1H 21.0 NaN 1H 3699489 555988 05/15/2014 08:40:26 PM 05/15/2014 09:34:03 PM None 05/15/2014 09:36:27 PM NaN NaN N NaT {48BBAFC2-6F72-F619-2C1A-005EC1643B9F} 641 None
27 K1291714 18 TRAFFIC INCIDENT 1H 18.0 NaN 1H 3674226 527339 11/11/2014 02:54:57 AM None 11/11/2014 02:54:57 AM 11/11/2014 03:29:14 AM NaN NaN Y NaT {E1FF75DD-BCF4-C9F0-DFC8-F98E886F986D} 425 None
28 J0065712 21 COMPLAINT OTHER 1H 3681752.0 531382.0 10/01/2012 12:44:06 PM 10/01/2012 12:44:27 PM 10/01/2012 12:44:59 PM 10/01/2012 12:52:25 PM DUP DUPLICATE 003XX Royal St 70130.0 8.0 (29.955108575882, -90.067279411608) 2012-10-01 12:44:06 {77FCFBCA-49D8-A4B6-D8A5-822149E70D35} 660 {'x': -90.06728502312177, 'y': 29.955113990312...
29 E3786812 18 TRAFFIC INCIDENT 1H 3683352.0 538374.0 05/24/2012 01:54:53 PM 05/24/2012 01:54:53 PM 05/24/2012 02:05:54 PM 05/24/2012 02:24:43 PM NAT NECESSARY ACTION TAKEN 018XX N Derbigny St 70116.0 5.0 (29.97428546205, -90.061982249162) 2012-05-24 13:54:53 {C9476A29-8114-D2BD-6FA3-9491B1FD736C} 580 {'x': -90.0619878600577, 'y': 29.9742908808282...
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
970 L1621117 22A AREA CHECK 1I NaN NaN 1I 3671435 553142 12/14/2017 09:14:14 PM 12/14/2017 09:14:14 PM 12/14/2017 09:14:19 PM 12/14/2017 09:38:10 PM NaN NaN N NaT {C3A20D49-E44C-9717-CFA9-F2A5D47BCF9A} 179 None
971 H0552516 21 COMPLAINT OTHER 1H 21.0 NaN 1H 3692874 551918 08/05/2016 06:39:41 PM None 08/05/2016 06:39:41 PM 08/05/2016 10:47:31 PM NaN NaN Y NaT {5ECEE3CD-1712-9070-51DC-10790C55AE4A} 196 None
972 E0962512 17F FUGITIVE ATTTACHMENT 1G 3684673.0 539982.0 05/06/2012 10:04:53 PM 05/06/2012 10:04:53 PM None 05/06/2012 10:05:08 PM RTF REPORT TO FOLLOW Elysian Fields Ave & N Galvez St 70117.0 5.0 (29.978666531583, -90.057753693797) 2012-05-06 22:04:53 {545A64C8-4872-3396-B99C-C524D5F56577} 301 {'x': -90.0577593037315, 'y': 29.9786719514519...
973 A3871913 18 TRAFFIC INCIDENT 1H 3701559.0 516958.0 01/29/2013 01:26:56 PM 01/29/2013 01:26:56 PM 01/29/2013 01:26:58 PM 01/29/2013 01:38:15 PM NAT NECESSARY ACTION TAKEN General De Gaulle Dr & Huntlee D 70131.0 4.0 (29.914830976353, -90.005264597449) 2013-01-29 13:26:56 {EAD23CF8-07A5-AFB9-0643-8B8738F71A51} 271 {'x': -90.00527019039149, 'y': 29.914836384016...
974 I4080513 100 HIT & RUN 1E 3660531.0 526542.0 09/30/2013 11:18:52 PM 10/01/2013 02:04:05 AM 10/01/2013 02:12:05 AM 10/01/2013 02:16:20 AM GOA GONE ON ARRIVAL 005XX S Carrollton Ave 70118.0 2.0 (29.942427784731, -90.134447707047) 2013-09-30 23:18:52 {0A9D4CC7-04E4-8CF7-C8B9-434F1C0E740A} 272 {'x': -90.134453336259, 'y': 29.94243319422235...
975 H3203611 62A BURGLAR ALARM, SILEN 2C 3672373.0 518941.0 08/21/2011 08:38:09 AM 08/21/2011 08:40:55 AM 08/21/2011 08:42:33 AM 08/21/2011 08:45:13 AM NAT NECESSARY ACTION TAKEN 040XX Magazine St 70115.0 6.0 (29.921181484798, -90.097318619706) 2011-08-21 08:38:09 {BA45AE5B-03C8-E746-BE09-27E515EF1CE4} 211 {'x': -90.0973242379294, 'y': 29.9211868908671...
976 J1301516 103D DOMESTIC DISTURBANCE 2B 35.0 NaN 2B 3660930 531266 10/12/2016 04:28:05 PM 10/12/2016 04:28:41 PM 10/12/2016 04:37:15 PM 10/12/2016 06:26:56 PM NaN NaN N NaT {1E51063B-B2A2-73E3-4BDB-222AC9C1DA04} 309 None
977 L3766312 62A BURGLAR ALARM, SILEN 2C 3701991.0 518152.0 12/27/2012 04:11:56 PM 12/27/2012 04:15:28 PM None 12/27/2012 04:18:05 PM VOI VOID 035XX Lang St 70131.0 4.0 (29.918100292854, -90.003857335737) 2012-12-27 16:11:56 {A0F02388-43F4-D44F-ECCE-28FC8EF2139E} 297 {'x': -90.00386292843632, 'y': 29.918105701276...
978 E0685114 18 TRAFFIC INCIDENT 1H 18.0 NaN 1H 3696210 551411 05/05/2014 11:30:29 PM 05/05/2014 11:31:39 PM 05/05/2014 11:30:29 PM 05/05/2014 11:37:51 PM NaN NaN Y NaT {BC5C4182-CC3A-50A2-7B39-BFB88A1E9D4E} 372 None
979 B1491111 107 SUSPICIOUS PERSON 2B 3678129.0 533076.0 02/10/2011 12:45:04 PM None None 02/10/2011 12:50:15 PM DUP DUPLICATE 018XX Canal St 70112.0 1.0 (29.95987607933, -90.078660929549) 2011-02-10 12:45:04 {5A405527-7B44-78B0-3052-F3D52EB9C407} 489 {'x': -90.07866654435806, 'y': 29.959881494429...
980 I3344518 18 TRAFFIC INCIDENT 1J 21.0 NaN 1J 3698542 519147 09/27/2018 01:24:55 AM 09/27/2018 01:24:55 AM 09/27/2018 01:24:55 AM 09/27/2018 02:03:41 AM NaN NaN Y NaT {AF7F4840-037C-876F-557E-44ABDF216CAB} 257 None
981 E1886814 103 DISTURBANCE (OTHER) 2A 103.0 NaN 2A 3672223 525508 05/14/2014 03:23:26 PM 05/14/2014 03:24:05 PM 05/14/2014 03:34:00 PM 05/14/2014 03:40:32 PM NaN NaN N NaT {E585712D-C815-3258-5BB4-50FA9B3D93ED} 347 None
982 D1096317 62A BURGLAR ALARM, SILENT 1A NaN NaN 2C 3703440 560261 04/09/2017 09:05:17 PM 04/10/2017 03:43:41 AM None 04/10/2017 04:10:19 AM NaN NaN N NaT {212DA271-B29E-FF3F-18EF-6B2E453BFBB1} 381 None
983 J2543015 21 COMPLAINT OTHER 1H NaN NaN 0E 3675672 525857 10/21/2015 01:05:16 PM 10/21/2015 01:10:01 PM None 10/21/2015 02:04:53 PM NaN NaN N NaT {DC45D2C4-34FC-2213-9429-AC20AF027237} 363 None
984 G1014312 67 THEFT 1E 3676106.0 530718.0 07/07/2012 03:21:21 PM 07/07/2012 03:23:28 PM 07/07/2012 03:23:33 PM 07/07/2012 04:26:56 PM RTF REPORT TO FOLLOW Bertrand St & Lafayette St 70113.0 1.0 (29.953453123323, -90.085130233788) 2012-07-07 15:21:21 {26FFD6F7-F70D-08E2-F5E8-89BC738AD6D8} 416 {'x': -90.08513585007961, 'y': 29.953458536814...
985 K3910518 22A AREA CHECK 1K NaN NaN 1K 3697581 515539 11/29/2018 08:07:55 PM 11/29/2018 08:07:55 PM 11/29/2018 08:07:55 PM 11/29/2018 09:48:13 PM NaN NaN Y NaT {8C8963FB-6638-252A-DE32-5281D6C5BD3D} 403 None
986 K1279211 21 COMPLAINT OTHER 1H 37369000.0 3513814.0 11/09/2011 11:32:43 AM 11/09/2011 11:34:39 AM None 11/09/2011 12:05:35 PM NAT NECESSARY ACTION TAKEN 019XX Joann NaN 0.0 (9.235500543E-7, -0.0000022929847665) 2011-11-09 11:32:43 {77A76953-7458-EE16-CFEF-7E50C18CE3D0} 455 {'x': 1.8008574168558748e-05, 'y': 3.003918417...
987 J2756411 99 RECKLESS DRIVING 2D 3684064.0 547517.0 10/18/2011 10:30:47 PM 10/18/2011 10:43:30 PM None 10/18/2011 10:47:35 PM GOA GONE ON ARRIVAL Elysian Fields Ave & Gentilly Bl 70122.0 3.0 (29.999403737159, -90.059412555475) 2011-10-18 22:30:47 {41CB848F-2CD5-C099-F1E7-A24961015322} 432 {'x': -90.05941816675012, 'y': 29.999409161486...
988 I1844213 107 SUSPICIOUS PERSON 2A 3679397.0 535828.0 09/14/2013 02:59:38 AM 09/14/2013 03:03:08 AM 09/14/2013 03:09:04 AM 09/14/2013 03:15:08 AM UNF UNFOUNDED 009XX N Prieur St 70116.0 1.0 (29.967404904189, -90.074561399733) 2013-09-14 02:59:38 {C61167EA-14E3-6246-2EB2-B964B6316807} 521 {'x': -90.07456701375169, 'y': 29.967410321060...
989 H1058117 NOPD INCIDENT REQUESTED BY ANOTHER AGENCY 2A NaN NaN 2A 3696876 525312 08/08/2017 06:33:18 PM None None 08/08/2017 06:33:59 PM NaN NaN N NaT {D7847DCD-444D-AF11-8985-479601301C7D} 741 None
990 E2451818 58 RETURN FOR ADDITIONAL INFO 1I 58.0 NaN 1I 3672772 526264 05/19/2018 10:39:22 PM 05/19/2018 10:39:22 PM 05/19/2018 10:47:02 PM 05/19/2018 11:16:59 PM NaN NaN N NaT {24F27B84-3F54-B20E-9AFE-61F10AD7B8C8} 543 None
991 E2650811 67 THEFT 1E 3677269.0 547605.0 05/17/2011 06:42:57 PM 05/17/2011 07:50:41 PM 05/17/2011 08:12:39 PM 05/17/2011 10:00:51 PM RTF REPORT TO FOLLOW 042XX Buchanan St 70122.0 3.0 (29.999851678873, -90.080875137137) 2011-05-17 18:42:57 {430BC094-3AA8-F5F3-F593-92697871071E} 726 {'x': -90.08088075426286, 'y': 29.999857102599...
992 H2357512 103 DISTURBANCE (OTHER) 1C 3715565.0 564186.0 08/15/2012 04:49:19 PM None None 08/15/2012 06:02:50 PM VOI VOID 060XX Bullard Ave 70128.0 7.0 (30.044236292957, -89.959269638889) 2012-08-15 16:49:19 {5A3CC2E0-D43B-28ED-8BB4-49E9503A7EF0} 630 {'x': -89.95927522486834, 'y': 30.044241730303...
993 B0811515 21 COMPLAINT OTHER 1H NaN NaN 0E 3682795 525927 02/07/2015 04:07:46 PM 02/07/2015 04:16:23 PM None 02/07/2015 04:22:03 PM NaN NaN N NaT {0AB3B844-7776-83EB-04AD-AAAF9CD1C344} 624 None
994 E3981912 21 COMPLAINT OTHER 1H 3682210.0 533199.0 05/25/2012 06:02:33 PM 05/25/2012 06:02:33 PM None 05/25/2012 06:16:45 PM NAT NECESSARY ACTION TAKEN Dauphine St & Saint Ann St 70116.0 8.0 (29.960090784609, -90.065769750523) 2012-05-25 18:02:33 {10D6A065-EE5B-A699-A0CB-136B7445DCC7} 639 {'x': -90.0657753618394, 'y': 29.9600962001742...
995 A2575219 22A AREA CHECK 1K 3678685.0 536249.0 01/20/2019 08:44:04 AM 01/20/2019 08:44:04 AM 01/20/2019 08:44:04 AM 01/20/2019 09:46:30 AM NAT Necessary Action Taken 22A NaN NaN Y 2019-01-20 08:44:04 {B4CF1985-73CB-398C-495C-EC752D62A62B} 686 {'x': -90.0768009430172, 'y': 29.9685894153133...
996 E2915114 17T TRAFFIC ATTACHMENT 1F 17.0 NaN 1G 3688424 540633 05/21/2014 08:34:25 PM 05/21/2014 08:34:25 PM None 05/21/2014 08:34:37 PM NaN NaN N NaT {23366D2E-DD2D-9306-F16B-FB396E542F75} 651 None
997 A0811415 62A BURGLAR ALARM, SILENT 2C NaN NaN 2C 3686861 541875 01/07/2015 10:39:22 PM None None 01/07/2015 10:43:47 PM NaN NaN N NaT {CA91D5A7-B617-960F-140E-70FA0D96A912} 707 None
998 L1133615 62C SIMPLE BURGLARY VEHICLE 1E NaN NaN 1E 3680642 537525 12/10/2015 01:21:07 PM None 12/10/2015 01:21:07 PM 12/10/2015 01:52:47 PM NaN NaN Y NaT {A1399702-B3FD-FB27-7D6F-649CCCD0F0F3} 798 None
999 B0108617 106 OBSCENITY, EXPOSING 2A 106.0 NaN 2A 3681317 530762 02/01/2017 08:14:41 PM 02/01/2017 08:15:47 PM 02/01/2017 08:15:53 PM 02/02/2017 01:03:07 AM NaN NaN N NaT {E13C6440-2B7D-A8AA-B7D4-A4923A84B8B5} 764 None

1000 rows × 20 columns

We can see some records have missing or invalid attribute values, including in the fields the manifests defines as the time and geometry values. We will visualize a sample layer on the map to understand it better.

In [24]:
m1 = gis.map()
m1
Out[24]:
In [25]:
m1.add_layer(description.sample_layer)

The map shows that some data points have been located outside New Orleans because of missing or invalid geometries. We want to explore data points within New Orleans city limits. We will use clip_layer tool to extract only point features within the New Orleans boundary. This will remove data with missing or invalid geometries.

Extract features within New Orleans boundary

The clip_layer method extracts input point, line, or polygon features from an input_layer that fall within the boundaries of features in a clip_layer. The output layer contains a subset of features from the input layer. We will clip our input call feature layer to the New Orleans blk_grp_lyr features.

In [26]:
clip_result = clip_layer(calls, blk_grp_lyr, output_name="service calls in new Orleans" + str(dt.now().microsecond))
{"messageCode":"BD_101051","message":"Possible issues were found while reading 'inputLayer'.","params":{"paramName":"inputLayer"}}
{"messageCode":"BD_101052","message":"Some records have either missing or invalid time values."}
{"messageCode":"BD_101054","message":"Some records have either missing or invalid geometries."}
In [27]:
clip_result
Out[27]:
service_calls_in_new_Orleans630022
service_calls_in_new_Orleans630022Feature Layer Collection by admin
Last Modified: November 12, 2019
0 comments, 0 views
In [28]:
orleans_calls = clip_result.layers[0]
In [29]:
m2 = gis.map("New Orleans")
m2
Out[29]: