Accessing and creating content¶
Your GIS can host a variety of maps, web layers, analytical tools, apps and individual files. Using the gis
module, you can search for, access and manage all your folders, contents and automate such maintenance workflows as scripts.
In this page, we are going to observe how the following can be accomplished:
Import libraries¶
import os
import datetime as dt
import pandas
from arcgis.gis import GIS
About search¶
As mentioned in the gis module
guide, the Python API uses Resource Manager classes to manage Web GIS users, groups, datastores, and content. You access the UserManager
, GroupManager
and ContentManager
classes as the users
, groups
, and content
properties of the GIS
, respectively. (See Helper objects
for details.)
Each resource manager implements the ArcGIS REST API search
operation as a method. For example, gis.content.search()
. It's important to understand that the search
mechanism uses many different inputs to find possible matches, ranks them and returns appropriate results. The search
becomes ideal for human interaction, but fuzzy
when looking for specific records programmatically. The search results are non-deterministic. Using search
may not necessarily be the best approach for finding specific items, but more a group of items from which to further filter.
It's also important to know that using search
programmatically, like with the Python API, does not correspond identically to searching with an application written in a different language. The various search
options in the ArcGIS Online or Portal for ArcGIS Enterprise interfaces work differently than the Python API resource managers' search
. Different applications may use different inputs. The relationship between a content search
in one application, like a Portal or ArcGIS Online viewer, is not one-to-one with a content search using the Python API even when logged in as the same user.
Let's look at an example to clarify:
# Log into ArcGIS Online
gis = GIS('home')
# Search for Feature Layers owned by the logged-in user
my_content = gis.content.search(query="owner:" + gis.users.me.username,
item_type="Feature Layer",
max_items=15)
my_content
Using Fiddler to examine the REST API query made to the server, it looks like this:
https://geosaurus.maps.arcgis.com/sharing/rest/search?
f=json
&q=owner:arcgis_python AND (type:"feature service") accountid:xxxxxxxxxxxxxxxx
&start=1
&num=100
&sortFields=avgRating
&sortOrder=desc
&token=<my_web_gis_token>
Logging into the same ArcGIS Online Organization in a web browser as the same user, navigate to My Content
, and then select All My Content
in the Folders
panel. Then navigate to the Filters
panel, choose Layers
and then further filter on Feature Layers
. Using the Developer Tools of the web browser, examine the REST API Query and see it looks like this:
https://geosaurus.maps.arcgis.com/sharing/rest/search?
num=20
&start=1
&sortField=modified
&sortOrder=desc
&q= owner:"arcgis_python" orgid:xxxxxxxxxxxxxxxx
(type:("Feature Collection" OR "Feature Service" OR "Stream Service" OR "WFS")
-typekeywords:"Table") -type:("Code Attachment") -typekeywords:("MapAreaPackage")
-type:("Map Area")
&f=json
&token=<my_arcgis_online_token>
https://geosaurus.maps.arcgis.com/sharing/rest/search?
num=6
&start=0
&sortField=title
&sortOrder=asc
&q=(group:"group_id1" AND
id:("item_id1" OR "item_id2"
OR "item_id3" OR "item_id4"
OR "item_id5" OR "item_id6"))
&f=json
&token=<my_arcgis_online_token>
The REST API queries differ between the applications written in different languages. You can expect different results between seemingly similiar queries to the same server when using different applications to search. The Python API search
may return different results than the ArcGIS Enterprise or ArcGIS Online map viewers and other applications.
Search Tip¶
Python list comprehensions
can augment the resource managers search
functionality to allow for more precise queries.
NOTE: List comprehension processing is performed on the client side.
To return a list of items owned by the logged in user published on a specific day, create a datetime
object to represent midnight as the start of the day, and a second datetime
object representing the end of the day. Convert each to millisecond integers since that is how an ArcGIS portal stores date values.
day_start = dt.datetime(2019, 5, 30, 0, 0, 0, 0)
day_end = dt.datetime(2019, 5, 30, 23, 59, 59, 999999)
start_timestamp = int(day_start.timestamp() * 1000)
end_timestamp = int(day_end.timestamp() * 1000)
Use the timestamps in a list comprehension:
content_published_53019 = [item
for item in gis.content.search(query="* AND \
owner:" + gis.users.me.username,
max_items=50)
if item.created > start_timestamp
and item.created < end_timestamp]
Write a function to return a human-readable date string and print the list
def readable_date(portal_stamp):
return dt.datetime.fromtimestamp(portal_stamp/1000).strftime('%B %d %Y at %I:%M.%S %p')
title = "Item Title"
item_type = "Item Type"
publish = "Published"
print(f"{title:40}{item_type:25}{publish:40}")
for content in content_published_53019:
print(f" {content.title:<40} {content.type:25} {readable_date(content.created):40}")
Searching for content¶
Accessing the content
property of your gis
object returns you an instance of ContentManager
class. You can use the search()
method of the ContentManager
object to search for items on your GIS. You can use the search()
even if you are connected as an anonymous user, in which case you get to search for public content on ArcGIS Online.
The first step is to import the API and connect to your GIS
from arcgis.gis import GIS
gis = GIS(url='https://pythonapi.playground.esri.com/portal', username='arcgis_python', password='amazing_arcgis_123')
Lets see what you get when you access content
property
type(gis.content)
As explained earlier, we don't recommend creating an instance of the ContentManager
directly, but instead recommend accessing it through the content
property of your GIS
object as shown below:
search_result = gis.content.search(query="title:Ports along US West Coast", item_type="Feature Layer")
search_result
The search()
method returned us a list of Item
objects that matched the search criteria and that the logged in user had permissions to access. The query
parameter supports a variety of inputs. Refer to the search reference for more ideas on what is supported. Some typical examples are shown below:
# search and list all feature layers in my contents
search_result = gis.content.search(query="", item_type="Feature Layer")
search_result
Notice how the search result includes items owned by other users that are shared across the organization. You can limit the search to only your content by specifying that in the query string as shown below:
search_my_contents = gis.content.search(query="owner:arcgis_python",
item_type="Feature *")
search_my_contents
Wild card search¶
Search supports usage of wild cards. You can use one asterisk *
for multiple character wild card search and one question mark ?
for single character.
search_result_USA = gis.content.search(query="title:USA*")
search_result_USA
Searching for content outside your portal¶
So far, we searched for content that belonged to the logged in user, as well as items shared within the portal and published by built-in accounts and other named users in the same portal. When connected to ArcGIS Online or connected to ArcGIS Enterprise configured to access ArcGIS Online, you can also search for content published and shared publicly even if the named users are in other portals. You can do so by turning the outside_org
flag to True
.
# search for content that begin with a prefix - say 'USA'
search_result_USA = gis.content.search(query="title:USA*", outside_org=True)
search_result_USA
Notice the Living Atlas
content returned in the results. The portal administrator configured the portal to access Living Atlas centent.
When using the Jupyter notebook environment, you can utilize the rich representation of items for your query results.
from IPython.display import display
for item in search_result_USA:
display(item)
Once you know an item's id (called itemId
), you can access it using the get()
method instead of searching for it.
# lets get the itemid of first item from previous query
first_item = search_result_USA[0]
known_item_id = first_item.id
print(known_item_id)
# lets use the get() to access this item
online_historic_item = gis.content.get(known_item_id)
online_historic_item
Searching for content in ArcGIS Online¶
So far, we searched for content accessible to a user logged into ArcGIS Enterprise. Let's connect directly to an ArcGIS Online Organization and futher explore ways to search for web GIS content.
# connect to ArcGIS Online
gis2 = GIS('home')
Let us search for public web scenes visualizing 3D cities. Note, I want to see content that is most popular by number of views to be returned on the top. I am also concerned that if too many items show up, the query might be running slow, hence restrict the results to 15 items.
public_3d_city_scenes = gis2.content.search(query="3d cities", item_type = "web scene",
sort_field="numViews" ,sort_order="asc",
max_items = 15, outside_org=True)
for item in public_3d_city_scenes:
display(item)