Introduction to AI-Powered services in ArcGIS Online
ArcGIS Online comes equipped with AI-powered services, supported in the ArcGIS API for Python via the arcgis.ai module. It is designed to extract valuable information and help automate tasks involving unstructured content, such as text, images, and documents.
The API supports three services or methods:
analyze_image()- Analyzes images based on the user provided instructions.analyze_text()- Analyzes provided text against the specified context.translate()- Translates language strings.
We will explore these methods through a generic example, as well as examples involving ArcGIS items to see potential uses of these AI services applicable to our ArcGIS data.
Note: Using these services consumes credits:
analyze_image()- 5 images per creditanalyze_text()- 5,000 characters per credittranslate()- 1,000 characters per credit
from arcgis.gis import GIS
from arcgis.ai import analyze_image, analyze_text, translate, AIUtilsResponsegis = GIS(profile='your_online_profile')Analyze Image
This tool helps analyze images using AI-powered models for object detection, text extraction, image description, and segmentation. Currently supported image formats are: JPEG, PNG and WEBP.
Note: Input image could be provided as an URL or Base64. Only images hosted on *.arcgis.com domains are supported via url input.
Let us start with this image hosted in ArcGIS Online.
image_url = "https://geosaurus.maps.arcgis.com/sharing/rest/content/items/6e9ff93df28a4231b4bd6e729ce42898/data"Prompts to the analyze_image() method may be provided as string, list or dictionary inputs. We will see examples for all 3 below.
1. Prompt as str input
prompt_str = "List all the colors present in the image."colors = analyze_image(image = image_url, prompt = prompt_str)
colorsAIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputTokens': 1831, 'outputTokens': 62, 'totalTokens': 1893}, model='gpt-5.4'), results=[AnalyzeResult(success=True, value=['green', 'dark green', 'light green', 'brown', 'dark brown', 'orange', 'yellow', 'red', 'pink', 'purple', 'blue', 'black', 'gray', 'white'], key='unique_string_1')])As we will continue to see through this guide, the results generated from running these tools in the ArcGIS API for Python are of AIUtilsResponse type.
You can verify if the request was successful using the success field of the response as shown below.
colors.successTrue
Since AIUtilsResponse is a pydantic model, you can access the specific results of the response as shown below.
colors.results[0].value['green', 'dark green', 'light green', 'brown', 'dark brown', 'orange', 'yellow', 'red', 'pink', 'purple', 'blue', 'black', 'gray', 'white']
The service has correctly identified the colors in this image.
We will now proceed to see examples involving the other input types through an ArcGIS item. This is a map image representing the hurricane history for Virginia.
virginia_map_url = "https://geosaurus.maps.arcgis.com/sharing/rest/content/items/36c34ac9e2c5402ab94762f97f3349af/data"2. Prompt as list input
prompt_list = [
"Describe the image.",
"Identify any hurricane related text in the image."
]result_list = analyze_image(image = virginia_map_url, prompt = prompt_list)
result_listAIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputTokens': 1822, 'outputTokens': 448, 'totalTokens': 2270}, model='gpt-5.4'), results=[AnalyzeResult(success=True, value='A grayscale map of Virginia showing hurricane, tropical storm, and tropical depression tracks across the state and nearby Atlantic coast. The title reads "Virginia Hurricanes, Tropical Storms, and Tropical Depressions by Date 1852-2016, 2017-2021." Many storm paths are drawn in light blue for 1852-2016 and in orange/red shades for 2017-2021. County boundaries, neighboring states, a mileage scale, north arrow, legend, and source information are also shown.', key='unique_string_0'), AnalyzeResult(success=True, value=['Virginia Hurricanes, Tropical Storms, and Tropical Depressions by Date 1852-2016, 2017-2021', 'Hurricane Category 2, 2017-2021', 'Tropical Storm, 2017-2021', 'Tropical Depression 2017-2021', 'Hurricane Category 4, 1852-2016', 'Hurricane Category 3, 1852-2016', 'Hurricane Category 2, 1852-2016', 'Hurricane Category 1, 1852-2016', 'Tropical Storm, 1852-2016', 'Tropical Depression, 1852-2016', 'FLORENCE, 2018', 'IDA, 2021', 'ZETA, 2020', 'BERTHA, 2020', 'MICHAEL, 2018', 'ELSA, 2021', 'ISAIAS, 2020', 'FAY, 2020', 'CLAUDETTE, 2021', 'DORIAN, 2019', 'FRED, 2021'], key='unique_string_1')])We can access results for the individual prompts via the prompt index provided to the results field as demonstrated below.
result_list.results[0].value'A grayscale map of Virginia showing hurricane, tropical storm, and tropical depression tracks across the state and nearby Atlantic coast. The title reads "Virginia Hurricanes, Tropical Storms, and Tropical Depressions by Date 1852-2016, 2017-2021." Many storm paths are drawn in light blue for 1852-2016 and in orange/red shades for 2017-2021. County boundaries, neighboring states, a mileage scale, north arrow, legend, and source information are also shown.'
result_list.results[1].value['Virginia Hurricanes, Tropical Storms, and Tropical Depressions by Date 1852-2016, 2017-2021', 'Hurricane Category 2, 2017-2021', 'Tropical Storm, 2017-2021', 'Tropical Depression 2017-2021', 'Hurricane Category 4, 1852-2016', 'Hurricane Category 3, 1852-2016', 'Hurricane Category 2, 1852-2016', 'Hurricane Category 1, 1852-2016', 'Tropical Storm, 1852-2016', 'Tropical Depression, 1852-2016', 'FLORENCE, 2018', 'IDA, 2021', 'ZETA, 2020', 'BERTHA, 2020', 'MICHAEL, 2018', 'ELSA, 2021', 'ISAIAS, 2020', 'FAY, 2020', 'CLAUDETTE, 2021', 'DORIAN, 2019', 'FRED, 2021']
The response provides a succinct summary of this map and extracts key information regarding hurricanes referenced.
3. Prompt as dict input
In order to provide prompts as a dictionary, it is essential to provide the values for the key and context keys of the dictionary for each prompt that you wish to provide.
prompt_dicts = [
{"key": "images_prompt", "context": "How many sub-boundaries of Virginia are included in the image?"},
{"key": "text_prompt", "context": "Identify the states named in the map."}
]result_dicts = analyze_image(image = virginia_map_url, prompt = prompt_dicts)
result_dictsAIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputTokens': 1826, 'outputTokens': 117, 'totalTokens': 1943}, model='gpt-5.4'), results=[AnalyzeResult(success=True, value=95, key='images_prompt'), AnalyzeResult(success=True, value=['Ohio', 'Pennsylvania', 'Maryland', 'Delaware', 'Kentucky', 'Tennessee', 'North Carolina', 'Virginia'], key='text_prompt')])result_dicts.results[0].value95
dicts_result.results[1].value['Ohio', 'Pennsylvania', 'Maryland', 'Delaware', 'Kentucky', 'Tennessee', 'North Carolina', 'Virginia']
The tool here currently identifies 95 sub-boundaries (counties) for Virginia, and detects all the states mentioned in the map image.
Analyze Text
This tool analyzes the provided text against the given prompt(s) using AI capabilities. Users are allowed to submit text for analysis, which can include tasks such as sentiment analysis, entity recognition, and other natural language processing operations.
Similar to the analyze_image() method, prompts to the analyze_text() method may be provided as string, list or dictionary inputs. We will see examples for all 3 input types below.
1. Prompt as str input
sample_text = "During our recent travels, we explored bustling city centers like London and Prague, \
wandered through the ancient, narrow streets of Rome, and relaxed on the sunny beaches of \
Alanya near the historic Alanya Castle. We visited the majestic Taj Mahal in Agra, toured the \
expansive Amber Palace in Jaipur, and took a relaxing break by the calm waters of the Dim River."
prompt_str = "Detect all locations in this text"locations = analyze_text(text = sample_text, prompt = prompt_str)
locationsAIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputTokens': 750, 'outputTokens': 55, 'totalTokens': 805}, model='gpt-5.4'), results=[AnalyzeResult(success=True, value=['London', 'Prague', 'Rome', 'Alanya', 'Alanya Castle', 'Taj Mahal', 'Agra', 'Amber Palace', 'Jaipur', 'Dim River'], key='key:1')])locations.results[0].value['London', 'Prague', 'Rome', 'Alanya', 'Alanya Castle', 'Taj Mahal', 'Agra', 'Amber Palace', 'Jaipur', 'Dim River']
2. Prompt as list input
prompt_list = [
"Detect the sentiment of the text.",
"Identify countries in the text."
]result_list = analyze_text(text = sample_text, prompt = prompt_list)
result_listAIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputTokens': 768, 'outputTokens': 54, 'totalTokens': 822}, model='gpt-5.4'), results=[AnalyzeResult(success=True, value='positive', key='unique_string_0'), AnalyzeResult(success=True, value=['United Kingdom', 'Czech Republic', 'Italy', 'Turkey', 'India'], key='unique_string_1')])We can access results for the individual prompts via the prompt index provided to the results field as demonstrated below.
result_list.results[0].value'positive'
result_list.results[1].value['United Kingdom', 'Czech Republic', 'Italy', 'Turkey', 'India']
3. Prompt as dict input
We will now look at an example that uses a dictionary of prompt inputs using an ArcGIS item. This is a document about Esri's Water Resources. Let us learn more about this item.
document = gis.content.get('c7b4e9240149481ea69af628513962fe')
documentWe extract the description of this item using the description property of the Item and pass it as the text input to our method.
item_description = document.description
item_description"<div>Esri's Water Resources GIS Platform offers a comprehensive suite of tools and resources designed to modernize water resource management. It emphasizes geospatial solutions for monitoring, analyzing, and modeling water systems, helping decision-makers tackle challenges like drought resilience, flood mitigation, and environmental protection. By leveraging the capabilities of ArcGIS, users can transform raw water data into actionable insights, ensuring more efficient and effective water resource management.</div><div><br /></div><div>A central feature of the platform is Arc Hydro, a specialized data model and toolkit developed for GIS-based water resource analysis. This toolset allows users to integrate, analyze, and visualize water datasets for applications ranging from live stream gauge monitoring to pollution control. Additionally, the platform connects users to the ArcGIS Living Atlas of the World, which offers extensive water-related datasets such as rivers, wetlands, and soils, supporting in-depth analyses of hydrologic conditions. The Hydro Community further enhances collaboration, enabling stakeholders to share expertise, discuss challenges, and build innovative solutions together.</div><div><br /></div><div>Esri’s platform also provides training opportunities and professional services to empower users with technical knowledge and skills. Through instructor-led courses, documentation, and best practices, users gain expertise in using ArcGIS and Arc Hydro for their specific water management needs. The combination of tools, datasets, and community engagement makes Esri's water resources platform a powerful asset for advancing sustainable water management initiatives across public and private sectors.</div><div><br /></div>"
Similar to the analyze_image() method, in order to provide prompts as a dictionary it is essential to provide the values for the key and context keys of the dictionary for each prompt that you wish to provide.
prompt_dicts = [
{"key": "summary_prompt", "context": "Summarize this text"},
{"key": "tools_prompt", "context": "Identify the tools mentioned in the text."}
]result_dicts = analyze_text(text = item_description, prompt = prompt_dicts)
result_dictsAIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputTokens': 997, 'outputTokens': 140, 'totalTokens': 1137}, model='gpt-5.4'), results=[AnalyzeResult(success=True, value="Esri's Water Resources GIS Platform provides geospatial tools, datasets, training, and community resources to improve water resource management. It uses ArcGIS capabilities to monitor, analyze, and model water systems for issues such as drought resilience, flood mitigation, and environmental protection. Key components include Arc Hydro for GIS-based water analysis, the ArcGIS Living Atlas of the World for water-related datasets, and the Hydro Community for collaboration and knowledge sharing.", key='summary_prompt'), AnalyzeResult(success=True, value=['ArcGIS', 'Arc Hydro', 'ArcGIS Living Atlas of the World', 'Hydro Community'], key='tools_prompt')])result_dicts.results[0].value"Esri's Water Resources GIS Platform provides geospatial tools, datasets, training, and community resources to improve water resource management. It uses ArcGIS capabilities to monitor, analyze, and model water systems for issues such as drought resilience, flood mitigation, and environmental protection. Key components include Arc Hydro for GIS-based water analysis, the ArcGIS Living Atlas of the World for water-related datasets, and the Hydro Community for collaboration and knowledge sharing."
result_dicts.results[1].value['ArcGIS', 'Arc Hydro', 'ArcGIS Living Atlas of the World', 'Hydro Community']
The tool summarizes the description of this item well and also identifies the ArcGIS tools mentioned.
Translate Text
The translate() method translates language strings passed to the desired array of output cultures. To identify the languages supported in your ArcGIS Online Organization and the culture of those languages, you may invoke the languages property of the GIS object as shown below.
gis.languages[{'language': 'Arabic', 'culture': 'ar', 'localizedName': 'عربي'},
{'language': 'Bosnian', 'culture': 'bs', 'localizedName': 'Bosanski'},
{'language': 'Bulgarian', 'culture': 'bg', 'localizedName': 'Български'},
{'language': 'Simplified Chinese',
'culture': 'zh-CN',
'localizedName': '简体中文'},
{'language': 'Catalan', 'culture': 'ca', 'localizedName': 'Català'},
{'language': 'Croatian', 'culture': 'hr', 'localizedName': 'Hrvatski'},
{'language': 'Czech', 'culture': 'cs', 'localizedName': 'čeština'},
{'language': 'Danish', 'culture': 'da', 'localizedName': 'Dansk'},
{'language': 'Dutch', 'culture': 'nl', 'localizedName': 'Nederlands'},
{'language': 'English',
'culture': 'en',
'localizedName': 'English',
'cultureFormats': [{'name': 'United States',
'localizedName': 'United States',
'format': 'us'},
{'name': 'Great Britain', 'localizedName': 'Great Britain', 'format': 'gb'},
{'name': 'Australia', 'localizedName': 'Australia', 'format': 'au'},
{'name': 'Canada', 'localizedName': 'Canada', 'format': 'ca'}]},
{'language': 'Estonian', 'culture': 'et', 'localizedName': 'Eesti'},
{'language': 'Finnish', 'culture': 'fi', 'localizedName': 'Suomi'},
{'language': 'French',
'culture': 'fr',
'localizedName': 'Français',
'cultureFormats': [{'name': 'France',
'localizedName': 'France',
'format': 'fr'},
{'name': 'Switzerland', 'localizedName': 'Suisse', 'format': 'ch'},
{'name': 'Canada', 'localizedName': 'Canada', 'format': 'ca'}]},
{'language': 'Greek', 'culture': 'el', 'localizedName': 'Ελλάδα'},
{'language': 'German',
'culture': 'de',
'localizedName': 'Deutsch',
'cultureFormats': [{'name': 'Germany',
'localizedName': 'Deutschland',
'format': 'de'},
{'name': 'Switzerland', 'localizedName': 'Schweiz', 'format': 'ch'},
{'name': 'Austria', 'localizedName': 'Österreich', 'format': 'at'}]},
{'language': 'Hebrew', 'culture': 'he', 'localizedName': 'עברית'},
{'language': 'Hungarian', 'culture': 'hu', 'localizedName': 'Magyar'},
{'language': 'Indonesian',
'culture': 'id',
'localizedName': 'Bahasa Indonesia'},
{'language': 'Italian',
'culture': 'it',
'localizedName': 'Italiano',
'cultureFormats': [{'name': 'Italy',
'localizedName': 'Italia',
'format': 'it'},
{'name': 'Switzerland', 'localizedName': 'Svizzera', 'format': 'ch'}]},
{'language': 'Japanese', 'culture': 'ja', 'localizedName': '日本語'},
{'language': 'Korean', 'culture': 'ko', 'localizedName': '한국어'},
{'language': 'Latvian', 'culture': 'lv', 'localizedName': 'Latviešu'},
{'language': 'Lithuanian', 'culture': 'lt', 'localizedName': 'Lietuvių'},
{'language': 'Norwegian', 'culture': 'nb', 'localizedName': 'Norsk'},
{'language': 'Polish', 'culture': 'pl', 'localizedName': 'Polski'},
{'language': 'Portuguese (Brazil)',
'culture': 'pt-BR',
'localizedName': 'Português (Brasil)'},
{'language': 'Portuguese (Portugal)',
'culture': 'pt-PT',
'localizedName': 'Português (Portugal)'},
{'language': 'Romanian', 'culture': 'ro', 'localizedName': 'Română'},
{'language': 'Russian', 'culture': 'ru', 'localizedName': 'Русский'},
{'language': 'Serbian', 'culture': 'sr', 'localizedName': 'Srpski'},
{'language': 'Slovenian', 'culture': 'sl', 'localizedName': 'Slovenščina'},
{'language': 'Slovak', 'culture': 'sk', 'localizedName': 'Slovenčina'},
{'language': 'Spanish',
'culture': 'es',
'localizedName': 'Español',
'cultureFormats': [{'name': 'Spain',
'localizedName': 'España',
'format': 'es'},
{'name': 'Mexico', 'localizedName': 'México', 'format': 'mx'}]},
{'language': 'Swedish', 'culture': 'sv', 'localizedName': 'Svenska'},
{'language': 'Thai', 'culture': 'th', 'localizedName': 'ไทย'},
{'language': 'Traditional Chinese (Hong Kong)',
'culture': 'zh-HK',
'localizedName': '繁體中文(香港)'},
{'language': 'Traditional Chinese (Taiwan)',
'culture': 'zh-TW',
'localizedName': '繁體中文(台灣)'},
{'language': 'Ukrainian', 'culture': 'uk', 'localizedName': 'Україна'},
{'language': 'Turkish', 'culture': 'tr', 'localizedName': 'Türkiye'},
{'language': 'Vietnamese', 'culture': 'vi', 'localizedName': 'Tiếng Việt'}]We start with an example to translate a few lines in Spanish to English.
text = "El bosque está lleno de vida. Mientras el sol brilla a través de las altas ramas, el viento sopla suavemente. Los \
pájaros cantan en los árboles y las mariposas vuelan sobre las flores. Estar en la naturaleza nos da paz y tranquilidad."Note that the input to the to_language parameter in the translate() method accepts input as a list, to support multiple output languages at once. However, the from_language parameter accepts a str input.
spanish_to_english = translate(text = text, to_language = ['en'], from_language = 'es')
spanish_to_englishAIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputCharacters': 242}, model='azure-ai-translator'), results=[TranslateResult(translations=[{'text': 'The forest is full of life. As the sun shines through the tall branches, the wind blows gently. Birds sing in the trees and butterflies fly over the flowers. Being in nature gives us peace and tranquility.', 'to': 'en'}], key='key:1')])Unlike the analyze_image() and analyze_text() methods, the process to access the response (translated text) is slightly different, as shown below.
spanish_to_english.results[0].translations[0]['text']'The forest is full of life. As the sun shines through the tall branches, the wind blows gently. Birds sing in the trees and butterflies fly over the flowers. Being in nature gives us peace and tranquility.'
Let us conclude with an example using an ArcGIS item from Ontario, Canada in French. We fetch the item and access its snippet which we provide as input to the translate() method.
ontario_canada_item = gis.content.get('9c2e4d1a671f41eb913411f048aaf661')
ontario_canada_itemitem_snippet = ontario_canada_item.snippet
item_snippet'Les données du rapport annuel historique sur les forêts de l’Ontario constituent un ensemble de données consolidées à l’échelle de la province, provenant des rapports annuels sur les opérations forestières depuis 2002. Format des données : ESRI Geodatabase'
french_to_english = translate(text = item_snippet, to_language = ['en'], from_language = 'fr')
french_to_englishAIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputCharacters': 257}, model='azure-ai-translator'), results=[TranslateResult(translations=[{'text': 'The Ontario Historical Annual Forest Report is a province-wide consolidated dataset from annual reports on forest operations since 2002. Data Format: ESRI Geodatabase', 'to': 'en'}], key='key:1')])french_to_english.results[0].translations[0]['text']'The Ontario Historical Annual Forest Report is a province-wide consolidated dataset from annual reports on forest operations since 2002. Data Format: ESRI Geodatabase'
We see that the tool has successfully translated the snippet from French to English.
Conclusion
The arcgis.ai module is useful when your workflows involve unstructured data in the form of field notes, inspection reports, public comments, customer requests, attached photos or scanned documents. Deriving necessary information from these data sources using these services supported within the ArcGIS API for Python can improve productivity.
To delve deeper into the arcgis.ai module with an example that involves real-world data, we encourage you to take a look at this sample.