Spatial and temporal distribution of service calls using big data tools

Introduction

The arcgis.geoanalytics module provides types and functions for distributed analysis of large datasets. These GeoAnalytics tools work with big data registered in the GIS datastores as well as with feature layers. In this notebook, we will go through the steps for setting up data to create a big data file share. We will also edit big data file share manifest to set spatial reference of the dataset. Once the data gets registered, we will demonstrate the utility of a number of tools including describe_dataset, aggregate_points, calculate_density, find_hot_spots, clip_layer, and run_python_script in order to better understand our data.

The sample aims to find answers to some fundamental questions:

  • What is the spatial relationship between 911 calls?
  • Which block groups have the highest number of 911 calls reporting?
  • What is the most common reason for 911 calls?
  • How many 911 calls occur each month?
  • How many 911 calls occur each hour?

The data that will be used in this sample was originally obtained from data.gov open data portal. You can obtain data by searching using keywords, for example: 'Calls for Service New Orleans'. This sample demonstrates ability of ArcGIS API for Python to perform big data analysis on your infrastructure.

Note:

The ability to perform big data analysis is only available on ArcGIS Enterprise licensed with a GeoAnalytics server and not yet available on ArcGIS Online.

Necessary Imports

%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
from datetime import datetime as dt

import arcgis
import arcgis.geoanalytics
from arcgis.gis import GIS

from arcgis.geoanalytics.summarize_data import aggregate_points, describe_dataset
from arcgis.geoanalytics.analyze_patterns import calculate_density, find_hot_spots
from arcgis.geoanalytics.manage_data import clip_layer, run_python_script

Connect to your ArcGIS Enterprise organization

gis = GIS('https://pythonapi.playground.esri.com/portal', 'arcgis_python', 'amazing_arcgis_123')

Ensure your GIS supports GeoAnalytics

After connecting to Enterprise portal, we need to ensure an ArcGIS Enterprise GIS is set up with a licensed GeoAnalytics server. To do so, we will call the is_supported() method.

arcgis.geoanalytics.is_supported()
True

Prepare the data

To register a file share or an HDFS, we need to format datasets as subfolders within a single parent folder and register the parent folder. This parent folder becomes a datastore, and each subfolder becomes a dataset. Our folder hierarchy would look like below:

|---FileShareFolder                 < -- The top-level folder is what is registered as a big data file share
   |---Calls                        < -- A folder "calls", composed of sub folders
      |---2011                      < -- A folder "2011", composed of 1 csv 
         |---Calls_for_Service_2011.csv                                                                                         
      |---2012                                                   
         |---Calls_for_Service_2012.csv
      |---2013
         |---Calls_for_Service_2013.csv
      |---2014
         |---Calls_for_Service_2014.csv
      |---2015
         |---Calls_for_Service_2015.csv
      |---2016
         |---Calls_for_Service_2016.csv     
      |---2017
         |---Calls_for_Service_2017.csv
      |---2018
         |---Calls_for_Service_2018.csv     
      |---2019
         |---Calls_for_Service_2019.csv        

Learn more about preparing your big data file share datasets here

Create a big data file share

The get_datastores() method of the geoanalytics module returns a DatastoreManager object that lets you search for and manage the big data file share items as Python API Datastore objects on your GeoAnalytics server.

bigdata_datastore_manager = arcgis.geoanalytics.get_datastores()
bigdata_datastore_manager
<DatastoreManager for https://pythonapi.playground.esri.com/ga/admin>

We will register service calls data as a big data file share using the add_bigdata() function on a DatastoreManager object.

When we register a directory, all subdirectories under the specified folder are also registered with the server. Always register the parent folder (for example, \machinename\mydatashare) that contains one or more individual dataset folders as the big data file share item. To learn more, see register a big data file share

Note: You cannot browse directories in ArcGIS Server Manager. You must provide the full path to the folder you want to register, for example, \myserver\share\bigdata. Avoid using local paths, such as C:\bigdata, unless the same data folder is available on all nodes of the server site.

# data_item = bigdata_datastore_manager.add_bigdata("ServiceCallsOrleans", r"\\machinename\datastore")
Created Big Data file share for ServiceCallsOrleans
bigdata_fileshares = bigdata_datastore_manager.search(id='cff51a1a-4f27-4955-a3ef-5fa23240ccf9')
bigdata_fileshares
[<Datastore title:"/bigDataFileShares/ServiceCallsOrleans" type:"bigDataFileShare">]
file_share_folder = bigdata_fileshares[0]

Once a big data file share is created, the GeoAnalytics server samples the datasets to generate a manifest, which outlines the data schema and specifies any time and geometry fields. A query of the resulting manifest returns each dataset's schema.. This process can take a few minutes depending on the size of your data. Once processed, querying the manifest property returns the schema of the datasets in your big data file share.

manifest = file_share_folder.manifest
manifest
{'datasets': [{'name': 'yearly_calls',
   'format': {'quoteChar': '"',
    'fieldDelimiter': ',',
    'hasHeaderRow': True,
    'encoding': 'UTF-8',
    'escapeChar': '"',
    'recordTerminator': '\n',
    'type': 'delimited',
    'extension': 'csv'},
   'schema': {'fields': [{'name': 'NOPD_Item', 'type': 'esriFieldTypeString'},
     {'name': 'Type_', 'type': 'esriFieldTypeString'},
     {'name': 'TypeText', 'type': 'esriFieldTypeString'},
     {'name': 'Priority', 'type': 'esriFieldTypeString'},
     {'name': 'MapX', 'type': 'esriFieldTypeDouble'},
     {'name': 'MapY', 'type': 'esriFieldTypeDouble'},
     {'name': 'TimeCreate', 'type': 'esriFieldTypeString'},
     {'name': 'TimeDispatch', 'type': 'esriFieldTypeString'},
     {'name': 'TimeArrive', 'type': 'esriFieldTypeString'},
     {'name': 'TimeClosed', 'type': 'esriFieldTypeString'},
     {'name': 'Disposition', 'type': 'esriFieldTypeString'},
     {'name': 'DispositionText', 'type': 'esriFieldTypeString'},
     {'name': 'BLOCK_ADDRESS', 'type': 'esriFieldTypeString'},
     {'name': 'Zip', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'PoliceDistrict', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'Location', 'type': 'esriFieldTypeString'}]},
   'geometry': {'geometryType': 'esriGeometryPoint',
    'spatialReference': {'wkid': 102682, 'latestWkid': 3452},
    'fields': [{'name': 'MapX', 'formats': ['x']},
     {'name': 'MapY', 'formats': ['y']}]},
   'time': {'timeType': 'instant',
    'timeReference': {'timeZone': 'UTC'},
    'fields': [{'name': 'TimeCreate',
      'formats': ['MM/dd/yyyy hh:mm:ss a']}]}}]}

Edit a big data file share

The spatial reference of the dataset is set to 4326, but we know this data is from New Orleans, Louisiana, and is actually stored in the Louisiana State Plane Coordinate System. We need to edit the manifest with the correct spatial reference: {"wkid": 102682, "latestWkid": 3452}. Knowing the location where this data belongs to and the coordinate system which contains geospatial information of this dataset, we will edit our manifest. This will set the correct spatial reference.

manifest['datasets'][0]['geometry']['spatialReference'] = { "wkid": 102682, "latestWkid": 3452 }
file_share_folder.manifest = manifest
file_share_folder.manifest
{'datasets': [{'name': 'yearly_calls',
   'format': {'quoteChar': '"',
    'fieldDelimiter': ',',
    'hasHeaderRow': True,
    'encoding': 'UTF-8',
    'escapeChar': '"',
    'recordTerminator': '\n',
    'type': 'delimited',
    'extension': 'csv'},
   'schema': {'fields': [{'name': 'NOPD_Item', 'type': 'esriFieldTypeString'},
     {'name': 'Type_', 'type': 'esriFieldTypeString'},
     {'name': 'TypeText', 'type': 'esriFieldTypeString'},
     {'name': 'Priority', 'type': 'esriFieldTypeString'},
     {'name': 'MapX', 'type': 'esriFieldTypeDouble'},
     {'name': 'MapY', 'type': 'esriFieldTypeDouble'},
     {'name': 'TimeCreate', 'type': 'esriFieldTypeString'},
     {'name': 'TimeDispatch', 'type': 'esriFieldTypeString'},
     {'name': 'TimeArrive', 'type': 'esriFieldTypeString'},
     {'name': 'TimeClosed', 'type': 'esriFieldTypeString'},
     {'name': 'Disposition', 'type': 'esriFieldTypeString'},
     {'name': 'DispositionText', 'type': 'esriFieldTypeString'},
     {'name': 'BLOCK_ADDRESS', 'type': 'esriFieldTypeString'},
     {'name': 'Zip', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'PoliceDistrict', 'type': 'esriFieldTypeBigInteger'},
     {'name': 'Location', 'type': 'esriFieldTypeString'}]},
   'geometry': {'geometryType': 'esriGeometryPoint',
    'spatialReference': {'wkid': 102682, 'latestWkid': 3452},
    'fields': [{'name': 'MapX', 'formats': ['x']},
     {'name': 'MapY', 'formats': ['y']}]},
   'time': {'timeType': 'instant',
    'timeReference': {'timeZone': 'UTC'},
    'fields': [{'name': 'TimeCreate',
      'formats': ['MM/dd/yyyy hh:mm:ss a']}]}}]}

Get data for analysis

Search for big data file shares

Adding a big data file share to the Geoanalytics server adds a corresponding big data file share item on the portal. We can search for these types of items using the item_type parameter.

search_result = gis.content.search("bigDataFileShares_ServiceCallsOrleans", item_type = "big data file share", max_items=40)
search_result
[<Item title:"bigDataFileShares_ServiceCallsOrleans" type:Big Data File Share owner:portaladmin>]
data_item = search_result[0]
data_item
bigDataFileShares_ServiceCallsOrleans
Big Data File Share by portaladmin
Last Modified: October 05, 2019
0 comments, 0 views

Querying the layers property of the item returns a featureLayer representing the data. The object is actually an API Layer object.

data_item.layers
[<Layer url:"https://pythonapi.playground.esri.com/ga/rest/services/DataStoreCatalogs/bigDataFileShares_ServiceCallsOrleans/BigDataCatalogServer/yearly_calls">]
calls = data_item.layers[0]
calls.properties
{
  "dataStoreID": "cff51a1a-4f27-4955-a3ef-5fa23240ccf9",
  "fields": [
    {
      "name": "NOPD_Item",
      "type": "esriFieldTypeString"
    },
    {
      "name": "Type_",
      "type": "esriFieldTypeString"
    },
    {
      "name": "TypeText",
      "type": "esriFieldTypeString"
    },
    {
      "name": "Priority",
      "type": "esriFieldTypeString"
    },
    {
      "name": "MapX",
      "type": "esriFieldTypeDouble"
    },
    {
      "name": "MapY",
      "type": "esriFieldTypeDouble"
    },
    {
      "name": "TimeCreate",
      "type": "esriFieldTypeString"
    },
    {
      "name": "TimeDispatch",
      "type": "esriFieldTypeString"
    },
    {
      "name": "TimeArrive",
      "type": "esriFieldTypeString"
    },
    {
      "name": "TimeClosed",
      "type": "esriFieldTypeString"
    },
    {
      "name": "Disposition",
      "type": "esriFieldTypeString"
    },
    {
      "name": "DispositionText",
      "type": "esriFieldTypeString"
    },
    {
      "name": "BLOCK_ADDRESS",
      "type": "esriFieldTypeString"
    },
    {
      "name": "Zip",
      "type": "esriFieldTypeInteger"
    },
    {
      "name": "PoliceDistrict",
      "type": "esriFieldTypeInteger"
    },
    {
      "name": "Location",
      "type": "esriFieldTypeString"
    }
  ],
  "name": "yearly_calls",
  "geometryType": "esriGeometryPoint",
  "type": "featureClass",
  "spatialReference": {
    "wkid": 102682,
    "latestWkid": 3452
  },
  "geometry": {
    "fields": [
      {
        "name": "MapX",
        "formats": [
          "x"
        ]
      },
      {
        "name": "MapY",
        "formats": [
          "y"
        ]
      }
    ]
  },
  "time": {
    "timeType": "instant",
    "timeReference": {
      "timeZone": "UTC"
    },
    "fields": [
      {
        "name": "TimeCreate",
        "formats": [
          "MM/dd/yyyy hh:mm:ss a"
        ]
      }
    ]
  },
  "currentVersion": 10.81,
  "children": []
}

Search for feature layers

block_grp_item = gis.content.get('9975b4dd3ca24d4bbe6177b85f9da7bb')
block_grp_item
New_Orleans_Block_Groups_2019
New_Orleans_Block_Groups_2019Feature Layer Collection by api_data_owner
Last Modified: October 05, 2019
0 comments, 17 views

We will use the first item for our analysis. Since the item is a Feature Layer Collection, accessing the layers property will give us a list of Feature layer objects.

blk_grp_lyr = block_grp_item.layers[0]

Describe data

The describe_dataset method provides an overview of big data. By default, the tool outputs a table layer containing calculated field statistics and a dict outlining geometry and time settings for the input layer.

Optionally, the tool can output a feature layer representing a sample set of features using the sample_size parameter, or a single polygon feature layer representing the input feature layers' extent by setting the extent_output parameter to True.

description = describe_dataset(input_layer=calls,
                               extent_output=True,
                               sample_size=1000,
                               output_name="Description of service calls" + str(dt.now().microsecond),
                               return_tuple=True)
{"messageCode":"BD_101051","message":"Possible issues were found while reading 'inputLayer'.","params":{"paramName":"inputLayer"}}
{"messageCode":"BD_101052","message":"Some records have either missing or invalid time values."}
{"messageCode":"BD_101054","message":"Some records have either missing or invalid geometries."}
description.output_json
{'datasetName': 'calls',
 'datasetSource': 'Big Data File Share - ServiceCallsOrleans',
 'recordCount': 3952898,
 'geometry': {'geometryType': 'Point',
  'sref': {'wkid': 102682, 'latestWkid': 3452},
  'countNonEmpty': 1763656,
  'countEmpty': 2189242,
  'spatialExtent': {'xmin': 0, 'ymin': 0, 'xmax': 37369000, 'ymax': 3513814}},
 'time': {'timeType': 'Instant',
  'countNonEmpty': 1763656,
  'countEmpty': 2189242,
  'temporalExtent': {'start': '2011-01-01 00:00:02.000',
   'end': '2019-08-03 23:59:09.000'}}}
description.sample_layer
<FeatureLayer url:"https://deldevd014.esri.com/server/rest/services/Hosted/Description_of_service_calls223782/FeatureServer/2">
description.sample_layer.query().sdf
NOPD_ItemType_TypeTextPriorityMapXMapYTimeCreateTimeDispatchTimeArriveTimeClosedDispositionDispositionTextBLOCK_ADDRESSZipPoliceDistrictLocationINSTANT_DATETIMEglobalidOBJECTIDSHAPE
0A024721221COMPLAINT OTHER1H37369000.03513814.001/02/2012 05:16:04 PM01/02/2012 05:16:05 PM01/02/2012 05:16:05 PM01/02/2012 05:20:40 PMNATNECESSARY ACTION TAKEN052XX N Claiborne AveNaN0.0(9.235500543E-7, -0.0000022929847665)2012-01-02 17:16:04{A471E188-9E41-2AA9-C46D-00771D78C616}12{'x': 1.8008574168558748e-05, 'y': 3.003918417...
1L318001862ABURGLAR ALARM, SILENT2ENaNNaN2E370634651868112/25/2018 12:38:48 AM12/25/2018 12:42:11 AM12/25/2018 12:47:42 AM12/25/2018 12:58:28 AMNaNNaNNNaT{8E68BDAF-E483-6952-ED6F-220A07FAEC2D}76None
2H078941421COMPLAINT OTHER1H21.0NaN1H367129253659908/06/2014 07:52:05 PMNone08/06/2014 07:52:05 PM08/06/2014 08:04:40 PMNaNNaNYNaT{003975B6-5491-DDB0-A0D8-6158F8EA62E0}133None
3E377781262ABURGLAR ALARM, SILEN2C3668023.0517302.005/24/2012 12:47:03 PM05/24/2012 12:48:48 PM05/24/2012 12:51:46 PM05/24/2012 12:55:18 PMNATNECESSARY ACTION TAKEN005XX Dufossat St70115.02.0(29.916803057765, -90.11110486281)2012-05-24 12:47:03{5ABCF635-D42E-92D4-3E1F-88F8853DCEC6}119{'x': -90.11111048458989, 'y': 29.916808462433...
4I082071862ABURGLAR ALARM, SILENT1ANaNNaN2E370433055483209/07/2018 05:52:32 PM09/08/2018 12:52:59 AMNone09/08/2018 01:33:43 AMNaNNaNNNaT{BC31F407-7C92-ADA5-5FFB-2B61E04604C2}114None
5C158751521COMPLAINT OTHER1H21.0NaN1H367055053930103/13/2015 09:42:20 PMNone03/13/2015 09:42:20 PM03/13/2015 09:53:24 PMNaNNaNYNaT{93EC1557-7D0A-539A-47A6-0EE7CEC9B645}126None
6E0773819107SUSPICIOUS PERSON2C3669625.0532902.005/06/2019 09:07:17 AM05/06/2019 09:08:03 AM05/06/2019 09:10:27 AM05/06/2019 09:20:14 AMGOAGONE ON ARRIVAL107NaNNaNN2019-05-06 09:07:17{7BDE1FC8-733C-F08E-2624-A274A5E9AC61}172{'x': -90.10552626485547, 'y': 29.959656351679...
7J332681621NNOISE COMPLAINT1ENaNNaN1E369302753507110/30/2016 06:00:21 AM10/30/2016 07:24:38 AM10/30/2016 07:33:03 AM10/30/2016 07:34:32 AMNaNNaNNNaT{98B78BC8-4A35-0A0F-93E0-085E6490355C}278None
8L424541162ABURGLAR ALARM, SILEN2C3693800.0525393.012/29/2011 04:47:02 PM12/29/2011 04:48:31 PM12/29/2011 04:56:36 PM12/29/2011 05:26:41 PMNATNECESSARY ACTION TAKEN027XX Lawrence St70114.04.0(29.938269364881, -90.029451676433)2011-12-29 16:47:02{EC51B15E-98FD-B6A1-9905-412D09B336A6}187{'x': -90.02945727694797, 'y': 29.938274776870...
9H1929011103DISTURBANCE (OTHER)1C3679384.0545506.008/13/2011 07:13:55 AM08/13/2011 07:13:57 AM08/13/2011 07:14:02 AM08/13/2011 07:17:31 AMNATNECESSARY ACTION TAKEN015XX Sere St70122.03.0(29.994016435986, -90.074266619976)2011-08-13 07:13:55{976D05C0-6D29-2618-439B-678E9EB46D37}146{'x': -90.0742722350555, 'y': 29.9940218586582...
10H083481421COMPLAINT OTHER1H21.0NaN1H367267054091508/07/2014 03:43:05 AM08/07/2014 03:43:05 AM08/07/2014 04:13:10 AM08/07/2014 04:23:35 AMNaNNaNNNaT{EBFC2284-8D11-B162-6801-CFB03ED6A7A9}206None
11C2682218103DISTURBANCE (OTHER)1A103.0NaN1C368769353473803/22/2018 09:17:24 AM03/22/2018 09:31:34 AM03/22/2018 09:31:36 AM03/22/2018 10:10:04 AMNaNNaNNNaT{0A9ED09A-EA2F-F7DB-9756-10173D9E76D7}305None
12H115031717MMUNICIPAL ATTACHMENT1F17.0NaN1G367017352628508/09/2017 12:02:13 PMNone08/09/2017 12:02:13 PM08/09/2017 12:02:50 PMNaNNaNYNaT{ABED0474-C002-EE73-82AC-F48AE11AF6CF}311None
13J345971721COMPLAINT OTHER1H67.0NaN1E367538753493110/27/2017 07:26:01 PM10/28/2017 08:36:33 AM10/28/2017 08:36:49 AM10/28/2017 08:49:46 AMNaNNaNNNaT{7197EBC3-81CC-03A3-6BAA-E3868DA10AB6}254None
14E011611218TRAFFIC INCIDENT1H3697253.0551601.005/01/2012 05:41:25 PM05/01/2012 05:41:25 PM05/01/2012 05:41:33 PM05/01/2012 06:09:18 PMNATNECESSARY ACTION TAKENChef Menteur Hwy & Stemway Dr70126.07.0(30.010223436348, -90.017600200365)2012-05-01 17:41:25{0B02A1FF-4259-F0E5-ADEB-F46F44BBA987}324{'x': -90.01760580074226, 'y': 30.010228864391...
15I086011162ABURGLAR ALARM, SILEN2C3680347.0524357.009/07/2011 03:14:10 AM09/07/2011 03:16:26 AM09/07/2011 03:22:28 AM09/07/2011 03:27:41 AMNATNECESSARY ACTION TAKEN014XX Magazine St70130.06.0(29.935834785662, -90.07196006635)2011-09-07 03:14:10{0213BAA0-A677-4003-E125-D7BE1ADB47FA}317{'x': -90.07196567831078, 'y': 29.935840195744...
16B029711221COMPLAINT OTHER2H3692716.0540389.002/02/2012 08:09:37 PM02/02/2012 08:43:07 PMNone02/02/2012 08:59:22 PMNATNECESSARY ACTION TAKEN024XX Alvar St70117.05.0(29.979536989582, -90.032336408255)2012-02-02 20:09:37{3EB84BEC-4A5C-337A-98FB-FE7982255EF0}230{'x': -90.03234201131997, 'y': 29.979542410466...
17I2182114107SUSPICIOUS PERSON2A107.0NaN2A368170453128709/17/2014 04:07:07 PMNone09/17/2014 04:07:07 PM09/17/2014 04:31:57 PMNaNNaNYNaT{5A1B7046-0DD3-F55B-81CB-90B9D6E333D4}258None
18I386321318TRAFFIC INCIDENT1H3698925.0557251.009/29/2013 04:01:05 AM09/29/2013 04:01:06 AM09/29/2013 04:01:05 AM09/29/2013 05:05:33 AMNATNecessary Action TakenI-10 W (7D04) & Morrison Rd Exit70126.07.0(30.025705895352, -90.012111936158)2013-09-29 04:01:05{E6F85E86-8803-7E4E-C2A9-5DD81C2950EA}525{'x': -90.01211753570936, 'y': 30.025711326943...
19B013951221COMPLAINT OTHER1B3680623.0525652.002/01/2012 08:53:09 PM02/01/2012 09:13:18 PMNone02/01/2012 09:32:21 PMNATNECESSARY ACTION TAKEN011XX Erato St70130.06.0(29.939387249037, -90.071043691518)2012-02-01 20:53:09{089C1D47-D66A-040D-9164-9AE1C7DF1AFC}431{'x': -90.07104930338194, 'y': 29.939392659922...
20F233781265SIMPLE ROBBERY1B3679940.0529508.006/15/2012 11:20:15 PM06/15/2012 11:24:43 PM06/15/2012 11:41:09 PM06/16/2012 02:03:22 AMRTFREPORT TO FOLLOW005XX Baronne St70113.08.0(29.950010633457, -90.073066114888)2012-06-15 23:20:15{0AD4CBB9-E41A-8B1B-F89A-72387C6D4EDD}586{'x': -90.0730717277556, 'y': 29.9500160465899...
21H111151221COMPLAINT OTHER1H3667387.0534128.008/08/2012 12:24:12 AM08/08/2012 12:27:55 AM08/08/2012 12:28:55 AM08/08/2012 12:35:46 AMNATNECESSARY ACTION TAKEN034XX S Carrollton Ave70125.02.0(29.963087783259, -90.11254651347)2012-08-08 00:24:12{CA58ACFE-54ED-B2F2-2454-3FFB9CA69437}563{'x': -90.11255213762003, 'y': 29.963093197957...
22G186801659CRIMINAL MISCHIEF1F59.0NaN1F366771152258807/17/2016 04:45:33 PMNoneNone07/17/2016 06:10:49 PMNaNNaNNNaT{B0995EEC-FE0D-6A05-92D8-DA61A3EC781F}518None
23D346031520AUTO ACCIDENT1A20.0NaN1A367996252666304/27/2015 06:25:27 PMNoneNone04/27/2015 07:40:24 PMNaNNaNNNaT{299E2EF4-E60B-2C56-919F-CB7DD65CFF2D}410None
24H437691262ABURGLAR ALARM, SILEN1C3681788.0532676.008/28/2012 06:48:22 PM08/28/2012 07:19:29 PMNone08/28/2012 07:20:30 PMNATNECESSARY ACTION TAKEN006XX Dauphine St70112.08.0(29.958665542329, -90.067120585039)2012-08-28 18:48:22{E1E36AC7-8FDD-0F08-23A6-C8D03FD7ECCE}613{'x': -90.0671261966614, 'y': 29.9586709575393...
25A045121518TRAFFIC INCIDENT1H18.0NaN1H369751953290201/05/2015 12:07:00 AM01/05/2015 12:07:00 AM01/05/2015 12:07:19 AM01/05/2015 12:17:43 AMNaNNaNNNaT{6532C606-9BF6-097C-6620-CFDBC1B8FE20}514None
26E208051421COMPLAINT OTHER1H21.0NaN1H369948955598805/15/2014 08:40:26 PM05/15/2014 09:34:03 PMNone05/15/2014 09:36:27 PMNaNNaNNNaT{48BBAFC2-6F72-F619-2C1A-005EC1643B9F}641None
27K129171418TRAFFIC INCIDENT1H18.0NaN1H367422652733911/11/2014 02:54:57 AMNone11/11/2014 02:54:57 AM11/11/2014 03:29:14 AMNaNNaNYNaT{E1FF75DD-BCF4-C9F0-DFC8-F98E886F986D}425None
28J006571221COMPLAINT OTHER1H3681752.0531382.010/01/2012 12:44:06 PM10/01/2012 12:44:27 PM10/01/2012 12:44:59 PM10/01/2012 12:52:25 PMDUPDUPLICATE003XX Royal St70130.08.0(29.955108575882, -90.067279411608)2012-10-01 12:44:06{77FCFBCA-49D8-A4B6-D8A5-822149E70D35}660{'x': -90.06728502312177, 'y': 29.955113990312...
29E378681218TRAFFIC INCIDENT1H3683352.0538374.005/24/2012 01:54:53 PM05/24/2012 01:54:53 PM05/24/2012 02:05:54 PM05/24/2012 02:24:43 PMNATNECESSARY ACTION TAKEN018XX N Derbigny St70116.05.0(29.97428546205, -90.061982249162)2012-05-24 13:54:53{C9476A29-8114-D2BD-6FA3-9491B1FD736C}580{'x': -90.0619878600577, 'y': 29.9742908808282...
...............................................................
970L162111722AAREA CHECK1INaNNaN1I367143555314212/14/2017 09:14:14 PM12/14/2017 09:14:14 PM12/14/2017 09:14:19 PM12/14/2017 09:38:10 PMNaNNaNNNaT{C3A20D49-E44C-9717-CFA9-F2A5D47BCF9A}179None
971H055251621COMPLAINT OTHER1H21.0NaN1H369287455191808/05/2016 06:39:41 PMNone08/05/2016 06:39:41 PM08/05/2016 10:47:31 PMNaNNaNYNaT{5ECEE3CD-1712-9070-51DC-10790C55AE4A}196None
972E096251217FFUGITIVE ATTTACHMENT1G3684673.0539982.005/06/2012 10:04:53 PM05/06/2012 10:04:53 PMNone05/06/2012 10:05:08 PMRTFREPORT TO FOLLOWElysian Fields Ave & N Galvez St70117.05.0(29.978666531583, -90.057753693797)2012-05-06 22:04:53{545A64C8-4872-3396-B99C-C524D5F56577}301{'x': -90.0577593037315, 'y': 29.9786719514519...
973A387191318TRAFFIC INCIDENT1H3701559.0516958.001/29/2013 01:26:56 PM01/29/2013 01:26:56 PM01/29/2013 01:26:58 PM01/29/2013 01:38:15 PMNATNECESSARY ACTION TAKENGeneral De Gaulle Dr & Huntlee D70131.04.0(29.914830976353, -90.005264597449)2013-01-29 13:26:56{EAD23CF8-07A5-AFB9-0643-8B8738F71A51}271{'x': -90.00527019039149, 'y': 29.914836384016...
974I4080513100HIT & RUN1E3660531.0526542.009/30/2013 11:18:52 PM10/01/2013 02:04:05 AM10/01/2013 02:12:05 AM10/01/2013 02:16:20 AMGOAGONE ON ARRIVAL005XX S Carrollton Ave70118.02.0(29.942427784731, -90.134447707047)2013-09-30 23:18:52{0A9D4CC7-04E4-8CF7-C8B9-434F1C0E740A}272{'x': -90.134453336259, 'y': 29.94243319422235...
975H320361162ABURGLAR ALARM, SILEN2C3672373.0518941.008/21/2011 08:38:09 AM08/21/2011 08:40:55 AM08/21/2011 08:42:33 AM08/21/2011 08:45:13 AMNATNECESSARY ACTION TAKEN040XX Magazine St70115.06.0(29.921181484798, -90.097318619706)2011-08-21 08:38:09{BA45AE5B-03C8-E746-BE09-27E515EF1CE4}211{'x': -90.0973242379294, 'y': 29.9211868908671...
976J1301516103DDOMESTIC DISTURBANCE2B35.0NaN2B366093053126610/12/2016 04:28:05 PM10/12/2016 04:28:41 PM10/12/2016 04:37:15 PM10/12/2016 06:26:56 PMNaNNaNNNaT{1E51063B-B2A2-73E3-4BDB-222AC9C1DA04}309None
977L376631262ABURGLAR ALARM, SILEN2C3701991.0518152.012/27/2012 04:11:56 PM12/27/2012 04:15:28 PMNone12/27/2012 04:18:05 PMVOIVOID035XX Lang St70131.04.0(29.918100292854, -90.003857335737)2012-12-27 16:11:56{A0F02388-43F4-D44F-ECCE-28FC8EF2139E}297{'x': -90.00386292843632, 'y': 29.918105701276...
978E068511418TRAFFIC INCIDENT1H18.0NaN1H369621055141105/05/2014 11:30:29 PM05/05/2014 11:31:39 PM05/05/2014 11:30:29 PM05/05/2014 11:37:51 PMNaNNaNYNaT{BC5C4182-CC3A-50A2-7B39-BFB88A1E9D4E}372None
979B1491111107SUSPICIOUS PERSON2B3678129.0533076.002/10/2011 12:45:04 PMNoneNone02/10/2011 12:50:15 PMDUPDUPLICATE018XX Canal St70112.01.0(29.95987607933, -90.078660929549)2011-02-10 12:45:04{5A405527-7B44-78B0-3052-F3D52EB9C407}489{'x': -90.07866654435806, 'y': 29.959881494429...
980I334451818TRAFFIC INCIDENT1J21.0NaN1J369854251914709/27/2018 01:24:55 AM09/27/2018 01:24:55 AM09/27/2018 01:24:55 AM09/27/2018 02:03:41 AMNaNNaNYNaT{AF7F4840-037C-876F-557E-44ABDF216CAB}257None
981E1886814103DISTURBANCE (OTHER)2A103.0NaN2A367222352550805/14/2014 03:23:26 PM05/14/2014 03:24:05 PM05/14/2014 03:34:00 PM05/14/2014 03:40:32 PMNaNNaNNNaT{E585712D-C815-3258-5BB4-50FA9B3D93ED}347None
982D109631762ABURGLAR ALARM, SILENT1ANaNNaN2C370344056026104/09/2017 09:05:17 PM04/10/2017 03:43:41 AMNone04/10/2017 04:10:19 AMNaNNaNNNaT{212DA271-B29E-FF3F-18EF-6B2E453BFBB1}381None
983J254301521COMPLAINT OTHER1HNaNNaN0E367567252585710/21/2015 01:05:16 PM10/21/2015 01:10:01 PMNone10/21/2015 02:04:53 PMNaNNaNNNaT{DC45D2C4-34FC-2213-9429-AC20AF027237}363None
984G101431267THEFT1E3676106.0530718.007/07/2012 03:21:21 PM07/07/2012 03:23:28 PM07/07/2012 03:23:33 PM07/07/2012 04:26:56 PMRTFREPORT TO FOLLOWBertrand St & Lafayette St70113.01.0(29.953453123323, -90.085130233788)2012-07-07 15:21:21{26FFD6F7-F70D-08E2-F5E8-89BC738AD6D8}416{'x': -90.08513585007961, 'y': 29.953458536814...
985K391051822AAREA CHECK1KNaNNaN1K369758151553911/29/2018 08:07:55 PM11/29/2018 08:07:55 PM11/29/2018 08:07:55 PM11/29/2018 09:48:13 PMNaNNaNYNaT{8C8963FB-6638-252A-DE32-5281D6C5BD3D}403None
986K127921121COMPLAINT OTHER1H37369000.03513814.011/09/2011 11:32:43 AM11/09/2011 11:34:39 AMNone11/09/2011 12:05:35 PMNATNECESSARY ACTION TAKEN019XX JoannNaN0.0(9.235500543E-7, -0.0000022929847665)2011-11-09 11:32:43{77A76953-7458-EE16-CFEF-7E50C18CE3D0}455{'x': 1.8008574168558748e-05, 'y': 3.003918417...
987J275641199RECKLESS DRIVING2D3684064.0547517.010/18/2011 10:30:47 PM10/18/2011 10:43:30 PMNone10/18/2011 10:47:35 PMGOAGONE ON ARRIVALElysian Fields Ave & Gentilly Bl70122.03.0(29.999403737159, -90.059412555475)2011-10-18 22:30:47{41CB848F-2CD5-C099-F1E7-A24961015322}432{'x': -90.05941816675012, 'y': 29.999409161486...
988I1844213107SUSPICIOUS PERSON2A3679397.0535828.009/14/2013 02:59:38 AM09/14/2013 03:03:08 AM09/14/2013 03:09:04 AM09/14/2013 03:15:08 AMUNFUNFOUNDED009XX N Prieur St70116.01.0(29.967404904189, -90.074561399733)2013-09-14 02:59:38{C61167EA-14E3-6246-2EB2-B964B6316807}521{'x': -90.07456701375169, 'y': 29.967410321060...
989H1058117NOPDINCIDENT REQUESTED BY ANOTHER AGENCY2ANaNNaN2A369687652531208/08/2017 06:33:18 PMNoneNone08/08/2017 06:33:59 PMNaNNaNNNaT{D7847DCD-444D-AF11-8985-479601301C7D}741None
990E245181858RETURN FOR ADDITIONAL INFO1I58.0NaN1I367277252626405/19/2018 10:39:22 PM05/19/2018 10:39:22 PM05/19/2018 10:47:02 PM05/19/2018 11:16:59 PMNaNNaNNNaT{24F27B84-3F54-B20E-9AFE-61F10AD7B8C8}543None
991E265081167THEFT1E3677269.0547605.005/17/2011 06:42:57 PM05/17/2011 07:50:41 PM05/17/2011 08:12:39 PM05/17/2011 10:00:51 PMRTFREPORT TO FOLLOW042XX Buchanan St70122.03.0(29.999851678873, -90.080875137137)2011-05-17 18:42:57{430BC094-3AA8-F5F3-F593-92697871071E}726{'x': -90.08088075426286, 'y': 29.999857102599...
992H2357512103DISTURBANCE (OTHER)1C3715565.0564186.008/15/2012 04:49:19 PMNoneNone08/15/2012 06:02:50 PMVOIVOID060XX Bullard Ave70128.07.0(30.044236292957, -89.959269638889)2012-08-15 16:49:19{5A3CC2E0-D43B-28ED-8BB4-49E9503A7EF0}630{'x': -89.95927522486834, 'y': 30.044241730303...
993B081151521COMPLAINT OTHER1HNaNNaN0E368279552592702/07/2015 04:07:46 PM02/07/2015 04:16:23 PMNone02/07/2015 04:22:03 PMNaNNaNNNaT{0AB3B844-7776-83EB-04AD-AAAF9CD1C344}624None
994E398191221COMPLAINT OTHER1H3682210.0533199.005/25/2012 06:02:33 PM05/25/2012 06:02:33 PMNone05/25/2012 06:16:45 PMNATNECESSARY ACTION TAKENDauphine St & Saint Ann St70116.08.0(29.960090784609, -90.065769750523)2012-05-25 18:02:33{10D6A065-EE5B-A699-A0CB-136B7445DCC7}639{'x': -90.0657753618394, 'y': 29.9600962001742...
995A257521922AAREA CHECK1K3678685.0536249.001/20/2019 08:44:04 AM01/20/2019 08:44:04 AM01/20/2019 08:44:04 AM01/20/2019 09:46:30 AMNATNecessary Action Taken22ANaNNaNY2019-01-20 08:44:04{B4CF1985-73CB-398C-495C-EC752D62A62B}686{'x': -90.0768009430172, 'y': 29.9685894153133...
996E291511417TTRAFFIC ATTACHMENT1F17.0NaN1G368842454063305/21/2014 08:34:25 PM05/21/2014 08:34:25 PMNone05/21/2014 08:34:37 PMNaNNaNNNaT{23366D2E-DD2D-9306-F16B-FB396E542F75}651None
997A081141562ABURGLAR ALARM, SILENT2CNaNNaN2C368686154187501/07/2015 10:39:22 PMNoneNone01/07/2015 10:43:47 PMNaNNaNNNaT{CA91D5A7-B617-960F-140E-70FA0D96A912}707None
998L113361562CSIMPLE BURGLARY VEHICLE1ENaNNaN1E368064253752512/10/2015 01:21:07 PMNone12/10/2015 01:21:07 PM12/10/2015 01:52:47 PMNaNNaNYNaT{A1399702-B3FD-FB27-7D6F-649CCCD0F0F3}798None
999B0108617106OBSCENITY, EXPOSING2A106.0NaN2A368131753076202/01/2017 08:14:41 PM02/01/2017 08:15:47 PM02/01/2017 08:15:53 PM02/02/2017 01:03:07 AMNaNNaNNNaT{E13C6440-2B7D-A8AA-B7D4-A4923A84B8B5}764None

1000 rows × 20 columns

We can see some records have missing or invalid attribute values, including in the fields the manifests defines as the time and geometry values. We will visualize a sample layer on the map to understand it better.

m1 = gis.map()
m1
m1.add_layer(description.sample_layer)

The map shows that some data points have been located outside New Orleans because of missing or invalid geometries. We want to explore data points within New Orleans city limits. We will use clip_layer tool to extract only point features within the New Orleans boundary. This will remove data with missing or invalid geometries.

Extract features within New Orleans boundary

The clip_layer method extracts input point, line, or polygon features from an input_layer that fall within the boundaries of features in a clip_layer. The output layer contains a subset of features from the input layer. We will clip our input call feature layer to the New Orleans blk_grp_lyr features.

clip_result = clip_layer(calls, blk_grp_lyr, output_name="service calls in new Orleans" + str(dt.now().microsecond))
{"messageCode":"BD_101051","message":"Possible issues were found while reading 'inputLayer'.","params":{"paramName":"inputLayer"}}
{"messageCode":"BD_101052","message":"Some records have either missing or invalid time values."}
{"messageCode":"BD_101054","message":"Some records have either missing or invalid geometries."}
clip_result
service_calls_in_new_Orleans630022
service_calls_in_new_Orleans630022Feature Layer Collection by admin
Last Modified: November 12, 2019
0 comments, 0 views
orleans_calls = clip_result.layers[0]
m2 = gis.map("New Orleans")
m2
m2.add_layer(orleans_calls)

Summarize data

We can use the aggregate_points method in the arcgis.geoanalytics.summarize_data submodule to group call features into individual block group features. The output polygon feature layer summarizes attribute information for all calls that fall within each block group. If no calls fall within a block group, that block group will not appear in the output.

The GeoAnalytics Tools use a process spatial reference during execution. Analyses with square or hexagon bins require a projected coordinate system. We'll use the World Cylindrical Equal Area projection (WKID 54034) below. All results are stored in the spatiotemporal datastore of the Enterprise in the WGS 84 Spatial Reference.

See the GeoAnalytics Documentation for a full explanation of analysis environment settings.

arcgis.env.process_spatial_reference = 54034
agg_result = aggregate_points(orleans_calls, 
                              polygon_layer=blk_grp_lyr,
                              output_name="aggregate results of call" + str(dt.now().microsecond))
agg_result
aggregate_results_of_call666755
aggregate_results_of_call666755Feature Layer Collection by admin
Last Modified: November 12, 2019
0 comments, 0 views
m3 = gis.map("New Orleans")
m3
m3.add_layer(agg_result)
m3.legend = True

Analyze patterns

The calculate_density method creates a density map from point features by spreading known quantities of some phenomenon (represented as attributes of the points) across the map. The result is a layer of areas classified from least dense to most dense. In this example, we will create density map by aggregating points within a bin of 1 kilometers. To learn more. please see here

cal_density = calculate_density(orleans_calls,
                                weight='Uniform',
                                bin_type='Square',
                                bin_size=1,
                                bin_size_unit="Kilometers",
                                time_step_interval=1,
                                time_step_interval_unit="Years",
                                time_step_repeat_interval=1,
                                time_step_repeat_interval_unit="Months",
                                time_step_reference=dt(2011, 1, 1),
                                radius=1000,
                                radius_unit="Meters",
                                area_units='SquareKilometers',
                                output_name="calculate density of call" + str(dt.now().microsecond))
cal_density
calculate_density_of_call124117
calculate_density_of_call124117Feature Layer Collection by admin
Last Modified: November 12, 2019
0 comments, 0 views
m4 = gis.map("New Orleans")
m4
m4.add_layer(cal_density)
m4.legend = True

Find statistically significant hot and cold spots

The find_hot_spots tool analyzes point data and finds statistically significant spatial clustering of high (hot spots) and low (cold spots) numbers of incidents relative to the overall distribution of the data.

hot_spots = find_hot_spots(orleans_calls, 
                           bin_size=100,
                           bin_size_unit='Meters',
                           neighborhood_distance=250,
                           neighborhood_distance_unit='Meters',
                           output_name="get hot spot areas" + str(dt.now().microsecond))
hot_spots
get_hot_spot_areas185860
get_hot_spot_areas185860Feature Layer Collection by admin
Last Modified: November 12, 2019
0 comments, 0 views
m5 = gis.map("New Orleans")
m5
m5.add_layer(hot_spots)
m5.legend = True

The darkest red features indicate areas where you can state with 99 percent confidence that the clustering of 911 call features is not the result of random chance but rather of some other variable that might be worth investigating. Similarly, the darkest blue features indicate that the lack of 911 calls is most likely not just random, but with 90% certainty you can state it is because of some variable in those locations. Features that are beige do not represent statistically significant clustering; the number of 911 calls could very likely be the result of random processes and random chance in those areas.

Visualize other aspects of data

The run_python_script method executes a Python script directly in an ArcGIS GeoAnalytics server site . The script can create an analysis pipeline by chaining together multiple GeoAnalytics tools without writing intermediate results to a data store. The tool can also distribute Python functionality across the GeoAnalytics server site.

Geoanalytics Server installs a Python 3.6 environment that this tool uses. The environment includes Spark 2.2.0, the compute platform that distributes analysis across multiple cores of one or more machines in your GeoAnalytics Server site. The environment includes the pyspark module which provides a collection of distributed analysis tools for data management, clustering, regression, and more. The run_python_script task automatically imports the pyspark module so you can directly interact with it.

When using the geoanalytics and pyspark packages, most functions return analysis results as Spark DataFrame memory structures. You can write these data frames to a data store or process them in a script. This lets you chain multiple geoanalytics and pyspark tools while only writing out the final result, eliminating the need to create any bulky intermediate result layers.

The TextType field contains the reason for the initial 911 call. Let's investigate the most frequent purposes for 911 calls in the New Orleans area by writing our own function:

def groupby_texttype():
    from datetime import datetime as dt
    # Calls data is stored in a feature service and accessed as a DataFrame via the layers object
    df = layers[0]
    # group the dataframe by TextType field and count the number of calls for each call type. 
    out = df.groupBy('TypeText').count()
    # Write the final result to our datastore.
    out.write.format("webgis").save("groupby_texttype" + str(dt.now().microsecond))

In the fnction above, calls layer containing 911 call points is converted to a DataFrame. pyspark can used to group the dataframe by TypeText field to find the count of each category of call. The result can be saved as a feature service or other ArcGIS Enterprise layer type.

run_python_script(code=groupby_texttype, layers=[calls])
[{'type': 'esriJobMessageTypeInformative',
  'description': 'Executing (RunPythonScript): RunPythonScript "def groupby_texttype():\\n    from datetime import datetime as dt\\n    # Load the big data file share layer into a DataFrame.\\n    df = layers[0]\\n    # group the dataframe by TextType field and count the number of calls for each call type. \\n    out = df.groupBy(\'TypeText\').count()\\n    # Write the final result to our datastore.\\n    output_name = "groupby_texttype" + str(dt.now().microsecond)\\n    out.write.format("webgis").save(output_name)\\n\\ngroupby_texttype()" https://deldevd014.esri.com/server/rest/services/DataStoreCatalogs/bigDataFileShares_ServiceCallsOrleans/BigDataCatalogServer/calls "{"defaultAggregationStyles": false, "processSR": {"wkid": 54034}}"'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Start Time: Mon Aug 26 15:03:57 2019'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Using URL based GPRecordSet param: https://deldevd014.esri.com/server/rest/services/DataStoreCatalogs/bigDataFileShares_ServiceCallsOrleans/BigDataCatalogServer/calls'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101028","message":"Starting new distributed job with 234 tasks.","params":{"totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"0/234 distributed tasks completed.","params":{"completedTasks":"0","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"1/234 distributed tasks completed.","params":{"completedTasks":"1","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"5/234 distributed tasks completed.","params":{"completedTasks":"5","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"63/234 distributed tasks completed.","params":{"completedTasks":"63","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"234/234 distributed tasks completed.","params":{"completedTasks":"234","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101081","message":"Finished writing results:"}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101082","message":"* Count of features = 283","params":{"resultCount":"283"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101083","message":"* Spatial extent = None","params":{"extent":"None"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101084","message":"* Temporal extent = None","params":{"extent":"None"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101226","message":"Feature service layer created: https://deldevd014.esri.com/server/rest/services/Hosted/groupby_texttype44974/FeatureServer/0","params":{"serviceUrl":"https://deldevd014.esri.com/server/rest/services/Hosted/groupby_texttype44974/FeatureServer/0"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Succeeded at Mon Aug 26 15:04:29 2019 (Elapsed Time: 31.45 seconds)'}]

The result is saved as a feature layer. We can Search for the saved item using the search() method. Providing the search keyword same as the name we used for writing the result will retrieve the layer.

grp_cat = gis.content.search('groupby_texttype')[0]

Accessing the tables property of the item will give us the tables object. We will then use query() method to read the table as spatially enabled dataframe.

grp_cat_df = grp_cat.tables[0].query().sdf

Sort the values in the decreasing order of the count field.

grp_cat_df.sort_values(by='count', ascending=False, inplace=True)
grp_cat_df.head(10).plot(x='TypeText', y='count', kind='barh')
<matplotlib.axes._subplots.AxesSubplot at 0x19fd0292898>
<Figure size 432x288 with 1 Axes>

We can see that COMPLAINT OTHER is the most common category of call followed by TRAFFIC INCIDENTS.

Now we'll investigate 911 calls to investigate the most common reasons for calls within block addresses. We will define a new function to input to the run_python_script tool, this time grouping data by the TextType and BLOCK_ADDRESS attributes.

def grpby_cat_blkadd():
    from datetime import datetime as dt
    # Load the big data file share layer into a DataFrame
    df = layers[0]
    out = df.groupBy('TypeText', 'BLOCK_ADDRESS').count()
    out.write.format("webgis").save("grpby_cat_blkadd" + str(dt.now().microsecond))
run_python_script(code=grpby_cat_blkadd, layers=[calls])
[{'type': 'esriJobMessageTypeInformative',
  'description': 'Executing (RunPythonScript): RunPythonScript "def grpby_cat_blkadd():\\n    from datetime import datetime as dt\\n    # Load the big data file share layer into a DataFrame\\n    df = layers[0]\\n    out = df.groupBy(\'TypeText\', \'BLOCK_ADDRESS\').count()\\n    out.write.format("webgis").save("grpby_cat_blkadd" + str(dt.now().microsecond))\\n\\ngrpby_cat_blkadd()" https://deldevd014.esri.com/server/rest/services/DataStoreCatalogs/bigDataFileShares_ServiceCallsOrleans/BigDataCatalogServer/calls "{"defaultAggregationStyles": false, "processSR": {"wkid": 54034}}"'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Start Time: Mon Aug 26 15:17:49 2019'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Using URL based GPRecordSet param: https://deldevd014.esri.com/server/rest/services/DataStoreCatalogs/bigDataFileShares_ServiceCallsOrleans/BigDataCatalogServer/calls'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101028","message":"Starting new distributed job with 234 tasks.","params":{"totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"0/234 distributed tasks completed.","params":{"completedTasks":"0","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"1/234 distributed tasks completed.","params":{"completedTasks":"1","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"7/234 distributed tasks completed.","params":{"completedTasks":"7","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"35/234 distributed tasks completed.","params":{"completedTasks":"35","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"82/234 distributed tasks completed.","params":{"completedTasks":"82","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"111/234 distributed tasks completed.","params":{"completedTasks":"111","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"161/234 distributed tasks completed.","params":{"completedTasks":"161","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"186/234 distributed tasks completed.","params":{"completedTasks":"186","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"234/234 distributed tasks completed.","params":{"completedTasks":"234","totalTasks":"234"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101081","message":"Finished writing results:"}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101082","message":"* Count of features = 2518651","params":{"resultCount":"2518651"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101083","message":"* Spatial extent = None","params":{"extent":"None"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101084","message":"* Temporal extent = None","params":{"extent":"None"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101226","message":"Feature service layer created: https://deldevd014.esri.com/server/rest/services/Hosted/grpby_cat_blkadd961138/FeatureServer/0","params":{"serviceUrl":"https://deldevd014.esri.com/server/rest/services/Hosted/grpby_cat_blkadd961138/FeatureServer/0"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Succeeded at Mon Aug 26 15:18:45 2019 (Elapsed Time: 56.25 seconds)'}]
grp_cat_addr = gis.content.search('grpby_cat_blkadd')[0]
grp_cat_addr_df = grp_cat_addr.tables[0].query().sdf
grp_cat_addr_df.sort_values(by='count', ascending=False, inplace=True)
grp_cat_addr_df.head(10).plot(x='BLOCK_ADDRESS', y='count', kind='barh')
<matplotlib.axes._subplots.AxesSubplot at 0x19fcff5f908>
<Figure size 432x288 with 1 Axes>

The chart shows that 22A block address has the highest number of incidents reported. So we want to further investigate the reason why this area had the most number of calls.

blk_addr_high = grp_cat_addr_df[grp_cat_addr_df['BLOCK_ADDRESS'] == '22A']
blk_addr_high.head()
TypeTextBLOCK_ADDRESScountglobalidOBJECTID
2516001AREA CHECK22A43842{A703737A-EEF0-64D0-43A7-53BC43E0EEA1}2536366
2477875COMPLAINT OTHER22A147{4CE9300A-9C88-84F6-C19B-E63965844BAE}2478122
2477683BUSINESS CHECK22A119{DB15C4C7-1B57-4CB3-F3A5-2BA31DFE2AF1}2477922
2511367TRAFFIC INCIDENT22A60{9BA080BD-EE95-6A32-0483-82016B6FAE90}2521217
2513961DIRECTED PATROL22A34{0FAB1EB8-4524-9ECE-7AA6-3861770DEF10}2528186
blk_addr_high.TypeText.sort_values(ascending=False).head()
1986885         WARR STOP WITH RELEASE
676774                    WALKING BEAT
2036113    UNAUTHORIZED USE OF VEHICLE
649816                TRAFFIC INCIDENT
2226650              SUSPICIOUS PERSON
Name: TypeText, dtype: object

The result indicates the most common reason for a 911 call in the 22A Block in New Orleans is defined as WARR STOP WITH RELEASE.

Now let's investigate the 911 call data for temporal trends. We saw in the manifest that the TimeCreate field holds specific time information for one instant (in UTC time zone) using a string format: MM/dd/yyyy hh:mm:ss a. We can parse these strings using Python's datetime module to extract year, month, day, hour, minute, and second to perform time analyses.

Let's define a helper function to convert the TimeCreate attribute field string types into a date type.

def calls_with_datetime():
    from datetime import datetime as dt
    # Load the big data file share layer into a DataFrame
    from pyspark.sql import functions as F
    df = layers[0]
    out = df.withColumn('datetime', F.unix_timestamp('TimeCreate', 'MM/dd/yyyy hh:mm:ss a').cast('timestamp'))
    out.write.format("webgis").save("calls_with_datetime" + str(dt.now().microsecond))
run_python_script(code=calls_with_datetime, layers=[calls])
[{'type': 'esriJobMessageTypeInformative',
  'description': 'Executing (RunPythonScript): RunPythonScript "def calls_with_datetime():\\n    from datetime import datetime as dt\\n    # Load the big data file share layer into a DataFrame\\n    from pyspark.sql import functions as F\\n    df = layers[0]\\n    out = df.withColumn(\'datetime\', F.unix_timestamp(\'TimeCreate\', \'MM/dd/yyyy hh:mm:ss a\').cast(\'timestamp\'))\\n    out.write.format("webgis").save("calls_with_datetime" + str(dt.now().microsecond))\\n\\ncalls_with_datetime()" https://deldevd014.esri.com/server/rest/services/DataStoreCatalogs/bigDataFileShares_ServiceCallsOrleans/BigDataCatalogServer/calls "{"defaultAggregationStyles": false, "processSR": {"wkid": 54034}}"'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Start Time: Mon Aug 26 16:19:49 2019'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Using URL based GPRecordSet param: https://deldevd014.esri.com/server/rest/services/DataStoreCatalogs/bigDataFileShares_ServiceCallsOrleans/BigDataCatalogServer/calls'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101028","message":"Starting new distributed job with 34 tasks.","params":{"totalTasks":"34"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"0/34 distributed tasks completed.","params":{"completedTasks":"0","totalTasks":"34"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"1/34 distributed tasks completed.","params":{"completedTasks":"1","totalTasks":"34"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"4/34 distributed tasks completed.","params":{"completedTasks":"4","totalTasks":"34"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"5/34 distributed tasks completed.","params":{"completedTasks":"5","totalTasks":"34"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"6/34 distributed tasks completed.","params":{"completedTasks":"6","totalTasks":"34"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"9/34 distributed tasks completed.","params":{"completedTasks":"9","totalTasks":"34"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"13/34 distributed tasks completed.","params":{"completedTasks":"13","totalTasks":"34"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"24/34 distributed tasks completed.","params":{"completedTasks":"24","totalTasks":"34"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"33/34 distributed tasks completed.","params":{"completedTasks":"33","totalTasks":"34"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"34/34 distributed tasks completed.","params":{"completedTasks":"34","totalTasks":"34"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101081","message":"Finished writing results:"}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101082","message":"* Count of features = 3952898","params":{"resultCount":"3952898"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101083","message":"* Spatial extent = {\\"xmin\\":-101.51780476781194,\\"ymin\\":3.0039184174589957e-06,\\"xmax\\":1.8008574168558748e-05,\\"ymax\\":30.167660889419377}","params":{"extent":"{\\"xmin\\":-101.51780476781194,\\"ymin\\":3.0039184174589957e-06,\\"xmax\\":1.8008574168558748e-05,\\"ymax\\":30.167660889419377}"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101084","message":"* Temporal extent = Interval(MutableInstant(2011-01-01 00:00:02.000),MutableInstant(2019-08-03 23:59:09.000))","params":{"extent":"Interval(MutableInstant(2011-01-01 00:00:02.000),MutableInstant(2019-08-03 23:59:09.000))"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101226","message":"Feature service layer created: https://deldevd014.esri.com/server/rest/services/Hosted/calls_with_datetime470961/FeatureServer/0","params":{"serviceUrl":"https://deldevd014.esri.com/server/rest/services/Hosted/calls_with_datetime470961/FeatureServer/0"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Succeeded at Mon Aug 26 16:21:19 2019 (Elapsed Time: 1 minutes 29 seconds)'}]
calls_with_datetime = gis.content.search('calls_with_datetime')[0]
calls_with_datetime
calls_with_datetime470961
Feature Layer Collection by admin
Last Modified: August 26, 2019
0 comments, 0 views
calls_with_datetime_lyr = calls_with_datetime.layers[0]

We will now split the date field into year, month and hour for studying temporal trends.

def call_with_added_date_time_cols():
    from datetime import datetime as dt
    # Load the big data file share layer into a DataFrame
    from pyspark.sql.functions import year, month, hour
    df = layers[0]
    df = df.withColumn('year', year(df['datetime']))
    df = df.withColumn('month', month(df['datetime']))
    out = df.withColumn('hour', hour(df['datetime']))
    out.write.format("webgis").save("call_with_added_date_time_cols" + str(dt.now().microsecond))
run_python_script(code=call_with_added_date_time_cols, layers=[calls_with_datetime_lyr])
[{'type': 'esriJobMessageTypeInformative',
  'description': 'Executing (RunPythonScript): RunPythonScript "def call_with_added_date_time_cols():\\n    from datetime import datetime as dt\\n    # Load the big data file share layer into a DataFrame\\n    from pyspark.sql.functions import year, month, hour\\n    df = layers[0]\\n    df = df.withColumn(\'year\', year(df[\'datetime\']))\\n    df = df.withColumn(\'month\', month(df[\'datetime\']))\\n    out = df.withColumn(\'hour\', hour(df[\'datetime\']))\\n    out.write.format("webgis").save("call_with_added_date_time_cols" + str(dt.now().microsecond))\\n\\ncall_with_added_date_time_cols()" https://deldevd014.esri.com/server/rest/services/Hosted/calls_with_datetime470961/FeatureServer/0 "{"defaultAggregationStyles": false, "processSR": {"wkid": 54034}}"'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Start Time: Mon Aug 26 16:36:53 2019'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Using URL based GPRecordSet param: https://deldevd014.esri.com/server/rest/services/Hosted/calls_with_datetime470961/FeatureServer/0'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101028","message":"Starting new distributed job with 39 tasks.","params":{"totalTasks":"39"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"0/39 distributed tasks completed.","params":{"completedTasks":"0","totalTasks":"39"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"1/39 distributed tasks completed.","params":{"completedTasks":"1","totalTasks":"39"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"5/39 distributed tasks completed.","params":{"completedTasks":"5","totalTasks":"39"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"7/39 distributed tasks completed.","params":{"completedTasks":"7","totalTasks":"39"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"14/39 distributed tasks completed.","params":{"completedTasks":"14","totalTasks":"39"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"15/39 distributed tasks completed.","params":{"completedTasks":"15","totalTasks":"39"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"20/39 distributed tasks completed.","params":{"completedTasks":"20","totalTasks":"39"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"28/39 distributed tasks completed.","params":{"completedTasks":"28","totalTasks":"39"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"35/39 distributed tasks completed.","params":{"completedTasks":"35","totalTasks":"39"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"39/39 distributed tasks completed.","params":{"completedTasks":"39","totalTasks":"39"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101081","message":"Finished writing results:"}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101082","message":"* Count of features = 3952898","params":{"resultCount":"3952898"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101083","message":"* Spatial extent = {\\"xmin\\":-101.51780476781194,\\"ymin\\":3.0039184174589957e-06,\\"xmax\\":1.8008574168558748e-05,\\"ymax\\":30.167660889419377}","params":{"extent":"{\\"xmin\\":-101.51780476781194,\\"ymin\\":3.0039184174589957e-06,\\"xmax\\":1.8008574168558748e-05,\\"ymax\\":30.167660889419377}"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101084","message":"* Temporal extent = Interval(MutableInstant(2011-01-01 00:00:02.000),MutableInstant(2019-08-03 23:59:09.000))","params":{"extent":"Interval(MutableInstant(2011-01-01 00:00:02.000),MutableInstant(2019-08-03 23:59:09.000))"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101226","message":"Feature service layer created: https://deldevd014.esri.com/server/rest/services/Hosted/call_with_added_date_time_cols193397/FeatureServer/0","params":{"serviceUrl":"https://deldevd014.esri.com/server/rest/services/Hosted/call_with_added_date_time_cols193397/FeatureServer/0"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Succeeded at Mon Aug 26 16:40:14 2019 (Elapsed Time: 3 minutes 20 seconds)'}]
date_time_added_item = gis.content.search('call_with_added_date_time_cols')
date_time_added_item[0]
call_with_added_date_time_cols193397
Feature Layer Collection by admin
Last Modified: August 26, 2019
0 comments, 0 views
date_time_added_lyr = date_time_added_item[0].layers[0]
def grp_calls_by_month():
    from datetime import datetime as dt
    # Load the big data file share layer into a DataFrame
    df = layers[0]
    out = df.groupBy('month').count()
    out.write.format("webgis").save("grp_calls_by_month" + str(dt.now().microsecond))
run_python_script(code=grp_calls_by_month, layers=[date_time_added_lyr])
[{'type': 'esriJobMessageTypeInformative',
  'description': 'Executing (RunPythonScript): RunPythonScript "def grp_calls_by_month():\\n    from datetime import datetime as dt\\n    # Load the big data file share layer into a DataFrame\\n    df = layers[0]\\n    out = df.groupBy(\'month\').count()\\n    out.write.format("webgis").save("grp_calls_by_month" + str(dt.now().microsecond))\\n\\ngrp_calls_by_month()" https://deldevd014.esri.com/server/rest/services/Hosted/call_with_added_date_time_cols193397/FeatureServer/0 "{"defaultAggregationStyles": false, "processSR": {"wkid": 54034}}"'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Start Time: Mon Aug 26 16:40:16 2019'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Using URL based GPRecordSet param: https://deldevd014.esri.com/server/rest/services/Hosted/call_with_added_date_time_cols193397/FeatureServer/0'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101028","message":"Starting new distributed job with 239 tasks.","params":{"totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"0/239 distributed tasks completed.","params":{"completedTasks":"0","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"1/239 distributed tasks completed.","params":{"completedTasks":"1","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"8/239 distributed tasks completed.","params":{"completedTasks":"8","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"9/239 distributed tasks completed.","params":{"completedTasks":"9","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"10/239 distributed tasks completed.","params":{"completedTasks":"10","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"12/239 distributed tasks completed.","params":{"completedTasks":"12","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"14/239 distributed tasks completed.","params":{"completedTasks":"14","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"16/239 distributed tasks completed.","params":{"completedTasks":"16","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"17/239 distributed tasks completed.","params":{"completedTasks":"17","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"19/239 distributed tasks completed.","params":{"completedTasks":"19","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"21/239 distributed tasks completed.","params":{"completedTasks":"21","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"23/239 distributed tasks completed.","params":{"completedTasks":"23","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"26/239 distributed tasks completed.","params":{"completedTasks":"26","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"35/239 distributed tasks completed.","params":{"completedTasks":"35","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"38/239 distributed tasks completed.","params":{"completedTasks":"38","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"239/239 distributed tasks completed.","params":{"completedTasks":"239","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101081","message":"Finished writing results:"}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101082","message":"* Count of features = 13","params":{"resultCount":"13"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101083","message":"* Spatial extent = None","params":{"extent":"None"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101084","message":"* Temporal extent = None","params":{"extent":"None"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101226","message":"Feature service layer created: https://deldevd014.esri.com/server/rest/services/Hosted/grp_calls_by_month385803/FeatureServer/0","params":{"serviceUrl":"https://deldevd014.esri.com/server/rest/services/Hosted/grp_calls_by_month385803/FeatureServer/0"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Succeeded at Mon Aug 26 16:43:52 2019 (Elapsed Time: 3 minutes 36 seconds)'}]
month = gis.content.search('grp_calls_by_month')[0]
grp_month = month.tables[0]
df_month = grp_month.query().sdf
df_month
monthcountglobalidOBJECTID
04.0164835{B85CF9B6-A0B7-0CDB-2BD4-EE75CC1D8B7D}103
17.0169707{9EDBD2EF-DC5E-FA4C-36B9-886AE09BEEB6}108
212.0120340{F55F4B76-040A-B393-157B-84A13AA90D34}25
32.0150248{1985235D-E1C1-F681-BDFC-4551A55A476A}175
49.0120086{9F6CB958-8506-C6CD-13DC-7E39B13E1D02}90
56.0165339{F51867CF-B370-11D7-106F-3BEA57712EAE}50
61.0161162{667184E7-5446-EFC4-DF05-642A5F93B1C9}44
73.0166684{24E0D747-1DF5-FB24-C5B5-4CACC874DB3B}52
85.0171744{3BEDD3BF-726B-D044-5B75-FDF1D088FFFD}67
9NaN2189242{143D6439-ACBD-FD65-A324-1C26B8F7E75F}43
108.0134547{B31419A6-0889-F609-E45E-B3506EDB07B7}104
1110.0124362{2E75CF98-AD0F-8F70-9F0C-83E89A5446F0}123
1211.0114602{077602D9-D33E-D51E-5657-3E0A823E1ABE}164
df_month.dropna().sort_values(by='month').plot(x='month', y='count', kind='bar')
<matplotlib.axes._subplots.AxesSubplot at 0x19fd05de2e8>
<Figure size 432x288 with 1 Axes>

It shows that calls are most frequent in the earlier months of the year.

def grp_calls_by_hour():
    from datetime import datetime as dt
    # Load the big data file share layer into a DataFrame
    df = layers[0]
    out = df.groupBy('hour').count()
    out.write.format("webgis").save("grp_calls_by_hour" + str(dt.now().microsecond))
run_python_script(code=grp_calls_by_hour, layers=[date_time_added_lyr])
[{'type': 'esriJobMessageTypeInformative',
  'description': 'Executing (RunPythonScript): RunPythonScript "def grp_calls_by_hour():\\n    from datetime import datetime as dt\\n    # Load the big data file share layer into a DataFrame\\n    df = layers[0]\\n    out = df.groupBy(\'hour\').count()\\n    out.write.format("webgis").save("grp_calls_by_hour" + str(dt.now().microsecond))\\n\\ngrp_calls_by_hour()" https://deldevd014.esri.com/server/rest/services/Hosted/call_with_added_date_time_cols193397/FeatureServer/0 "{"defaultAggregationStyles": false, "processSR": {"wkid": 54034}}"'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Start Time: Mon Aug 26 16:49:37 2019'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Using URL based GPRecordSet param: https://deldevd014.esri.com/server/rest/services/Hosted/call_with_added_date_time_cols193397/FeatureServer/0'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101028","message":"Starting new distributed job with 239 tasks.","params":{"totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"0/239 distributed tasks completed.","params":{"completedTasks":"0","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"1/239 distributed tasks completed.","params":{"completedTasks":"1","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"29/239 distributed tasks completed.","params":{"completedTasks":"29","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101029","message":"239/239 distributed tasks completed.","params":{"completedTasks":"239","totalTasks":"239"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101081","message":"Finished writing results:"}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101082","message":"* Count of features = 25","params":{"resultCount":"25"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101083","message":"* Spatial extent = None","params":{"extent":"None"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101084","message":"* Temporal extent = None","params":{"extent":"None"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': '{"messageCode":"BD_101226","message":"Feature service layer created: https://deldevd014.esri.com/server/rest/services/Hosted/grp_calls_by_hour873509/FeatureServer/0","params":{"serviceUrl":"https://deldevd014.esri.com/server/rest/services/Hosted/grp_calls_by_hour873509/FeatureServer/0"}}'},
 {'type': 'esriJobMessageTypeInformative',
  'description': 'Succeeded at Mon Aug 26 16:50:55 2019 (Elapsed Time: 1 minutes 18 seconds)'}]
hour = gis.content.search('grp_calls_by_hour')[0]
grp_hour = hour.tables[0]
df_hour = grp_hour.query().sdf
df_hour.dropna().sort_values(by='hour').plot(x='hour', y='count', kind='bar')
<matplotlib.axes._subplots.AxesSubplot at 0x19fd0ab4438>
<Figure size 432x288 with 1 Axes>

The chart above shows frequency of calls per hour of the day. We can see that there is a high frequency of calls during 4-5 p.m

Conclusion

We have shown spatial patterns in distribuition of emergency calls and the need of including temporal trends in the analysis. There is an obvious variation in the monthly, weekly, daily and even hourly distribution of service calls and this should be accounted for in further analysis or allocation of resources.

Your browser is no longer supported. Please upgrade your browser for the best experience. See our browser deprecation post for more details.