Part 7 - Utility Functions for Geocoding

Introduction

AnalyzeGeocodeInput, BatchGeocode, GeocodeEnterpriseTable, GeocodeFile and Suggest are counted as utilities for geocoding tools in an ArcGIS Enterprise environment, and they can be used to fine tune, improve accuracy, or investigate geocoding results, and are intended for advanced users.

In the following sections, we will be looking at how the methods analyze_geocode_input() and suggest() are called and how the results can be understood.

from arcgis.gis import GIS
from arcgis.geocoding import Geocoder, get_geocoders, geocode, analyze_geocode_input, suggest

gis = GIS(profile="your_enterprise_profile")

my_geocoder = get_geocoders(gis)[0]
my_geocoder

<Geocoder url:"https://geocode.arcgis.com/arcgis/rest/services/World/GeocodeServer">

my_geocoder.properties.capabilities

'Geocode,ReverseGeocode,Suggest'

When the capability needed is not present in the above output, to continue performing the utility function will result in an error message(s) such as, AttributeError: 'Toolbox' object has no attribute 'analyze_geocode_input'

Utility function of `analyze_geocode_input()`

The utility function analyze_geocode_input() analyzes a file and auto-generates a field mapping for the input table or file provided, and then outputs GeocodeParameters JSON, which is a required parameter for the batch_geocode tool and can be used to investigate/improve/fine tune batch_geocode operation.

The analyze_geocode_input function takes in a geocode input (either a table or file of addresses) and returns an output dictionary that includes a suggested field mapping.

It supports CSV, XLS, or table input. The table can be from a big data file share or from a feature service.
The task generates a suggested field mapping based on the input fields and the geocoding service candidate fields and returns it in a geocode_parameters dictionary. This geocode_parameters dictionary output is the an input to the Batch Geocode tool. The output geocode_parameters dictionary also includes field info (name, length, and type) as well as additional information that helps the geocode tool parse the input file or table.
Required parameter - geocode_service_url: The REST URL of the geocode service that you want to geocode your addresses and places against. The URL must end in geocodeServer. This can be a utility service or a locator that is anonymously accessible. The geocode service must be configured to allow batch geocoding.
Optional parameter - input_table_or_item: Item Id or Url pointing to the CSV, Excel file, or the table item.

For other ways to perform analyze actions, please check:

Note: This method only exists in ArcGIS Enterprise. This API is not supported for ArcGIS Online.

Input type as Item

>>> analyze_geocode_input(input_table_or_item={"itemid":"<CSV Item Id>"},
                          geocode_service_url="<Your custom geocode service url>",
                          input_file_parameters={"fileType":"csv",
                                                 "headerRowExists":"true",
                                                 "columnDelimiter":"",
                                                 "textQualifier":""})

{
  "paramName": "geocodeParameters",
  "dataType": "GPString",
  "value": {
    "field_info": "[("ObjectID", "TEXT", 255), ("Address", "TEXT", 255)]",
    "column_names": "",
    "file_type": "csv",
    "column_delimiter": "",
    "text_qualifier": "",
    "field_mapping": "[["ObjectID", "OBJECTID"], ["Address", "Address"]]",
    "header_row_exists": true
  }
}

Input type as URL

>>> analyze_geocode_input(input_table_or_item={"url":"<CSV Item url>",
                                               "serviceToken":"<Your service token>"},
                          geocode_service_url="<Your custom geocode service url>",
                          input_file_parameters={"fileType":"csv",
                                                 "headerRowExists":"true",
                                                 "columnDelimiter":"",
                                                 "textQualifier":""})

{
  "paramName": "geocodeParameters",
  "dataType": "GPString",
  "value": {
    "field_info": "[("ObjectID", "TEXT", 255), ("Address", "TEXT", 255)]",
    "column_names": "",
    "file_type": "csv",
    "column_delimiter": "",
    "text_qualifier": "",
    "field_mapping": "[["ObjectID", "OBJECTID"], ["Address", "Address"]]",
    "header_row_exists": true
  }
}

Input type as Table

>>> from arcgis.features import Table
    table_lyr = Table(url="<Your Hosted Services Directory>/missoula_coffee_shops1/FeatureServer/0", gis=gis)
>>> res = analyze_geocode_input(input_table_or_item = table_lyr,
                                geocode_service_url="<Your custom geocode service url>",
                                column_names=None,
                                input_file_parameters={  "fileType":"csv",
                                                         "headerRowExists":"true",
                                                         "columnDelimiter":"",
                                                         "textQualifier":""},
                                locale="en",
                                context=None,
                                gis=gis)

{
  "paramName": "geocodeParameters",
  "dataType": "GPString",
  "value": {
    "field_info": "[("ObjectID", "TEXT", 255), ("Address", "TEXT", 255)]",
    "column_names": "",
    "file_type": "csv",
    "column_delimiter": "",
    "text_qualifier": "",
    "field_mapping": "[["ObjectID", "OBJECTID"], ["Address", "Address"]]",
    "header_row_exists": true
  }
}

Utility function of `suggest()`

suggest() can be used to get search suggestions. For example suggest('Millennium Park',{'x': -87.62245002553522, 'y': 41.88425998326565}) will return a list of similarly named POIs within 'n' distance from the Point. However, suggest() would not return actual geocoded result, instead suggested search strings.

The result of this operation is a resource representing a list of suggested matches for the input text. This resource provides the matching text, as well as a unique ID value, which links a suggestion to a specific place or address. A geocoder must meet the following requirements to support the suggest operation:

The address locator from which the geocoder was published must support suggestions, and the geocoder must have the Suggest capability enabled. Note that, only geocoders published using ArcGIS 10.3 for Server or later support the Suggest capability.
The suggest operation allows character-by-character auto-complete suggestions to be generated for user input in a client application. This capability facilitates the interactive search user experience by reducing the number of characters that need to be typed before a suggested match is obtained. A client application can provide a list of suggestions that is updated with each character typed by a user until the address they are looking for appears in the list.

For example, if a user wants to find 27488 Stanford Ave, Bowden, North Dakota, they need to type only 27488 stanfo, and the address they are looking for is returned as an item in the suggestion list. Or if they are looking for a Disney park but don't know the proper name or address, they can find it using suggest.

Point of interests (POI) by name

suggest(text="Disney Park")

{'suggestions': [{'text': 'Disney Park, Elk Grove Village, IL, USA',
   'magicKey': 'dHA9MCNsb2M9MTA2MzMyNDkjbG5nPTM0I3BsPTE2ODg3NjEjbGJzPTE0OjExNzM1MzIz',
   'isCollection': False},
  {'text': 'Disney Park, Wanowrie, Pune, Maharashtra, 411022, IND',
   'magicKey': 'dHA9MCNsb2M9MzkxNDY5MjcjbG5nPTM0I3BsPTM2MTc4MjE5I2xicz0xNDoxMTczNTMyMw==',
   'isCollection': False},
  {'text': 'Disney Park, Keytesville, MO, 65261, USA',
   'magicKey': 'dHA9MCNsb2M9MTA3ODAwNCNsbmc9MzQjbGJzPTEwOTo0NzMzNzYzNA==',
   'isCollection': False},
  {'text': 'Disney Store, 1645 Parkway, Sevierville, TN, 37862, USA',
   'magicKey': 'dHA9MCNsb2M9MTQ1NjMxNDMjbG5nPTM0I3BsPTY0NzcyMDkjbGJzPTE0OjExNzM1MzcxOzEwOTo1MDYwOTU5OQ==',
   'isCollection': False},
  {'text': 'Disney Store, 8405 Park Meadows Center Dr, Lone Tree, CO, 80124, USA',
   'magicKey': 'dHA9MCNsb2M9MTM5MDg4ODcjbG5nPTM0I3BsPTMxNzE1NTEjbGJzPTE0OjExNzM1MzcxOzEwOTo1MDYwMzA1MA==',
   'isCollection': False}]}

Points of interest (POI) by name with restrictions

suggest(text="Starbucks",
        location={'x': -87.62245002553522, 'y': 41.88425998326565},
        distance=50)

{'suggestions': [{'text': 'Starbucks',
   'magicKey': 'dHA9MSNubT1TdGFyYnVja3Mjc3o9LTg3LjYyMjQ1MDAyNTUzNTIxNTo0MS44ODQyNTk5ODMyNjU2NSNjcz03MA==',
   'isCollection': True},
  {'text': 'Starbucks, 200 E Randolph St Aon Ctr, Chicago, IL, 60601, USA',
   'magicKey': 'dHA9MCNsb2M9MTQzNDAxODMjbG5nPTM0I3BsPTU0NDM3MTEjbGJzPTE0OjM0ODY4OTQ0',
   'isCollection': False},
  {'text': 'Starbucks, 151 N Michigan Ave, Chicago, IL, 60601, USA',
   'magicKey': 'dHA9MCNsb2M9MTQzNDAxNTMjbG5nPTM0I3BsPTU0NDMyNTgjbGJzPTE0OjM0ODY4OTQ0',
   'isCollection': False},
  {'text': 'Starbucks, 300 E Randolph St, Chicago, IL, 60601, USA',
   'magicKey': 'dHA9MCNsb2M9MTQzNDAxNTEjbG5nPTM0I3BsPTU0NDMwODgjbGJzPTE0OjM0ODY4OTQ0',
   'isCollection': False},
  {'text': 'Starbucks, 200 N Michigan Ave, Chicago, IL, 60601, USA',
   'magicKey': 'dHA9MCNsb2M9MTQzNDAxNTMjbG5nPTM0I3BsPTU0NDMyNDYjbGJzPTE0OjM0ODY4OTQ0',
   'isCollection': False}]}

suggest(text="Millennium Park",
        location={'x': -87.62245002553522, 'y': 41.88425998326565},
        distance=50)

{'suggestions': [{'text': 'Millennium Park, 201 E Randolph St, Chicago, IL, 60601, USA',
   'magicKey': 'dHA9MCNsb2M9MzgyNzgzMyNsbmc9MzQjcGw9NDU4MjAwI2xicz0xNDoyMzk2NDQwMQ==',
   'isCollection': False},
  {'text': 'Millennium Park Plaza, 155 N Michigan Ave, Chicago, IL, 60601, USA',
   'magicKey': 'dHA9MCNsb2M9MTQzNDAxNTMjbG5nPTM0I3BsPTU0NDMyMDcjbGJzPTE0OjIzOTY0NDE2',
   'isCollection': False},
  {'text': 'Millennium Park Eye Center, 30 N Michigan Ave, Chicago, IL, 60602, USA',
   'magicKey': 'dHA9MCNsb2M9MTQzNDAyMTcjbG5nPTM0I3BsPTU0NDQ2MTAjbGJzPTE0OjIzOTY0NDA5',
   'isCollection': False},
  {'text': 'Millennium Park Oral Surgery, 25 E Washington St, Chicago, IL, 60602, USA',
   'magicKey': 'dHA9MCNsb2M9MTQzNDAyMTYjbG5nPTM0I3BsPTU0NDQ1MDUjbGJzPTE0OjIzOTY0NDE1',
   'isCollection': False},
  {'text': 'Millennium Parking Garage, 5 S Columbus Dr, Chicago, IL, 60603, USA',
   'magicKey': 'dHA9MCNsb2M9MzgyNzczMSNsbmc9MzQjcGw9NDU3OTAxI2xicz0xNDoyMzk2NDQzMQ==',
   'isCollection': False}]}

Street addresses (including street name) with restrictions

suggest(text="27488 stanfo", 
        location={'y':47.47022800176198, 'x':-99.70795415025731},
        distance=50,
        max_suggestions=4)

{'suggestions': [{'text': '27488 Stanford Ave, Madera, CA, 93637, USA',
   'magicKey': 'dHA9MCNsb2M9NDYwODc3I2xuZz0zNCNobj0yNzQ4OCNsYnM9MTA5OjUxODAwMjQ3',
   'isCollection': False},
  {'text': '27488 Stanford St, Hemet, CA, 92544, USA',
   'magicKey': 'dHA9MCNsb2M9NDgxMjEzI2xuZz0zNCNobj0yNzQ4OCNsYnM9MTA5OjUxODAwMzIw',
   'isCollection': False},
  {'text': '27488 Stanford Dr, Temecula, CA, 92591, USA',
   'magicKey': 'dHA9MCNsb2M9MTE5NDM2MTAjbG5nPTM0I2huPTI3NDg4I2xicz0xMDk6NTE4MDAyODE=',
   'isCollection': False},
  {'text': 'Stanford Ave E, Bowdon, ND, 58418, USA',
   'magicKey': 'dHA9MCNsb2M9Mzk5MTMxNCNsbmc9MzQjbGJzPTEwOTo1MTgwMDI0OA==',
   'isCollection': False}]}

Understanding the output of `suggest()`

`text`

The suggestion text can be used in a client application to populate a list of suggestions as a user enters characters in a search text box.

It can also be included with magic_key in a geocode request to quickly retrieve a geosearch candidate.

Example:

"text": Starbucks, 1265 Alabama St, Redlands, California

`magicKey`

An ID attribute value that, along with the text attribute, links a suggestion to an address or place.

After you make a suggest request, the typical workflow is to pass the text (as the single_line_address parameter value) and magic_key values in a geocode request, which retrieves the result in less time than passing in a single_line_address value by itself.

Note that, the magicKey value is not permanent across versions of the ArcGIS World Geocoding Service and should not be stored by a client application. It is only intended to be used in geosearch operations as input for the geocode's magic_key parameter.

`isCollection`

A Boolean parameter that indicates if the suggestion item represents a collection of places, as opposed to a specific place.

If isCollection = True for a suggestion item, it means the item represents a search term for a common place-name or POI category; suggestion items such as Starbucks, McDonald's, Gas Station, and Airport will have isCollection = True.

If isCollection = False, the suggestion item represents a specific place-name or address; suggestion items such as Disneyland or 380 New York St, Redlands, CA will have isCollection = false.

This flag can be used by application developers to apply different behavior to cases where isCollection is True versus cases where isCollection is False.

Example (for POI categories and common place-names):

"isCollection": True

Example (for street addresses, postal codes, admin places, and POI's):

"isCollection": False

The `magic_key` parameter

More about `magic_key`

The geocode operation retrieves results quicker when you pass in valid single_line_address and magic_key values than when you don't pass in magic_key. However, to get these advantages, you need to make a prior request to suggest, which provides a magic_key. This may or may not be relevant to your workflow.

The suggest operation is often called on to improve the user experience of search boxes by analyzing partial text and providing complete names of places, addresses, points of interest, and so on. For instance, typing Mbu into a search box offers Mbuji-Mayi, Democratic Republic of the Congo as a suggestion, so the user doesn't need to type the complete name.

Looking at the suggestion process from another perspective, as the user types, the suggest operation performs a text search, which is a redundant part of the overall search that the geocode operation can also perform. The user chooses a place-name or type—narrowing the results to a specific record. The results from suggest include text and magic_key values that contain the information the user chose; passing these values from suggest into geocode results in faster and more accurate operations.

In summary, using the magic_key parameter in geocode is a two-step process:

Make a request to suggest. The response includes text and magic_key properties.
Make a request to geocode and pass in the text and magic_key values returned from suggest as the single_line_address and magic_key input parameters, respectively. The magic_key parameter will not function correctly if passed alone. Both magic_key and single_line_address must be included in a geocode request so the output matches the selected suggestion, e.g. geocode(magic_key=...).

no_magic_key = geocode("Mbu")
len(no_magic_key)

suggest("Mbu")

{'suggestions': [{'text': 'Mbuji-Mayi, Kasaï-Oriental, COD',
   'magicKey': 'dHA9MCNsb2M9MzIzNDIwNDAjbG5nPTQzI3BsPTIxODQ3NTMzI2xicz0xNDoyMzUwMDM4Nw==',
   'isCollection': False},
  {'text': 'Mburundi',
   'magicKey': 'dHA9MCNsb2M9Mjg1NjQ1ODAjbG5nPTQzI3BsPTE3OTIxNzQ1I2xicz0xNDoyMzUwMTAzMw==',
   'isCollection': False},
  {'text': 'Mbuye, Kibilizi, Nyanza, Amajyepfo, RWA',
   'magicKey': 'dHA9MCNsb2M9NDgxODMwODgjbG5nPTM0I3BsPTUxMTMyMjk3I2xicz0xNDoyMzUwMTIyNA==',
   'isCollection': False},
  {'text': 'Mbulamuti, Buzaaya, Kamuli, UGA',
   'magicKey': 'dHA9MCNsb2M9NTE2MDc2NzEjbG5nPTM0I3BsPTU4MTkzNjk2I2xicz0xNDoyMzUwMDYzMQ==',
   'isCollection': False},
  {'text': 'Mbuji-Mayi, Kasaï Oriental, COD',
   'magicKey': 'dHA9MCNsb2M9MzIzNDAxNTMjbG5nPTQzI3BsPTIxODQ1MTg1I2xicz0xNDoyMzUwMDM4Nw==',
   'isCollection': False}]}

with_magic_key = geocode("Mbu",
                         magic_key="dHA9MCNsb2M9MzIzNDIwNDAjbG5nPTQzI3BsPTIxODQ3NTMzI2xicz0xNDoyMzUwMDM4Nw==")
display(with_magic_key)
len(with_magic_key)

[{'address': 'Mbuji-Mayi, Kasaï-Oriental',
  'location': {'x': 23.58979000000005, 'y': -6.136029999999948},
  'score': 100,
  'attributes': {'Loc_name': 'World',
   'Status': 'M',
   'Score': 100,
   'Match_addr': 'Mbuji-Mayi, Kasaï-Oriental',
   'LongLabel': 'Mbuji-Mayi, Kasaï-Oriental, COD',
   'ShortLabel': 'Mbuji-Mayi',
   'Addr_type': 'Locality',
   'Type': 'City',
   'PlaceName': 'Mbuji-Mayi',
   'Place_addr': 'Mbuji-Mayi, Kasaï-Oriental',
   'Phone': '',
   'URL': '',
   'Rank': 3,
   'AddBldg': '',
   'AddNum': '',
   'AddNumFrom': '',
   'AddNumTo': '',
   'AddRange': '',
   'Side': '',
   'StPreDir': '',
   'StPreType': '',
   'StName': '',
   'StType': '',
   'StDir': '',
   'BldgType': '',
   'BldgName': '',
   'LevelType': '',
   'LevelName': '',
   'UnitType': '',
   'UnitName': '',
   'SubAddr': '',
   'StAddr': '',
   'Block': '',
   'Sector': '',
   'Nbrhd': '',
   'District': '',
   'City': 'Mbuji-Mayi',
   'MetroArea': '',
   'Subregion': '',
   'Region': 'Kasaï-Oriental',
   'RegionAbbr': '',
   'Territory': '',
   'Zone': '',
   'Postal': '',
   'PostalExt': '',
   'Country': 'COD',
   'LangCode': 'FRE',
   'Distance': 0,
   'X': 23.58979000000005,
   'Y': -6.136029999999948,
   'DisplayX': 23.58979000000005,
   'DisplayY': -6.136029999999948,
   'Xmin': 23.51779000000005,
   'Xmax': 23.66179000000005,
   'Ymin': -6.208029999999948,
   'Ymax': -6.064029999999948,
   'ExInfo': ''},
  'extent': {'xmin': 23.51779000000005,
   'ymin': -6.208029999999948,
   'xmax': 23.66179000000005,
   'ymax': -6.064029999999948}}]

Note: For best results, when the `search_extent` and `location` parameters are included in a `suggest` request, the same parameters should be included in the corresponding `geocode` request.

Conclusions

In Part 7, we have explored how the utilities of geocoding tools can make geocoding operations faster, more reliable and more accurate.

Part 7 - Utility Functions for Geocoding

Introduction

Utility function of analyze_geocode_input()

Input type as Item

Input type as URL

Input type as Table

Utility function of suggest()

Point of interests (POI) by name

Points of interest (POI) by name with restrictions

Street addresses (including street name) with restrictions

Understanding the output of suggest()

text

magicKey

isCollection

The magic_key parameter

More about magic_key